Yahoo d/l limit

[t2000kw]

Hi guys,

So.... what is the current limit people are getting on downloads ?

1,000 , ??  10,000 ??  more ? less ?


Dave

I've got to about 30,000 or so before being cut off. But it seems arbitrary as to when Yahoo decides to cut off the connection. Sometimes it tales less than that.

Just got locked again by Yahoo. Now I'll change my download limit before the wait period to 199 and go from there.

I did get pretty far this last time before getting locked.

I just now set my options to allow me to d/l 25 messages at a time, up to 250 messages, then wait 1 minute and start all over again. I haven't tested this out yet but am right now on a group where I have about 15,000 more messages to d/l.

[t2000kw]

This COULD be possible if you use a different Yahoo account to visit the group with your browser than the one you do with PGO.

> also interesting...when I would be shut out on previous versions, I could not access yahoo at all.  Nowadays, I can access yahoo manually through my browser, even though PGO reports a yahoo lock.

This shouldn't happen. If you get told by PGO that you're locked out then you shouldn;t be able to access Yahoo.

Please check your Workflow option isn't still set. This feature is redundant as PGO now recognises the difference between failed messages & a lock-out situation.

Cheers,

Wilson.

[Wilson Logan]

True!

Cheers,

Wilson.

[mattylad]

reboot your router.

If your on DHCP from your ISP you get a new IP and can continue.

(Upsets the kids though - they get kicked out of their games :D)

[Wilson Logan]

Doesn't work for me. My address is static but if yours is dynamically allocated, it may work.

Cheers,

Wilson.

[thamoskuk9]

I found that in the group that I was DLing there were a fair number of deleted messages. What I think might be going on is this:

When you set PG Offline to download, say, 300 messages, it seems to keep downloading until it actually gets 300 valid messages -- meaning that any invalid/deleted message numbers don't count against the DL counter, and you wind up pounding Yahoo for that many extra times.

For example, I found that by subtracting the highest downloaded message in one "DL block" versus the highest from the previous block, there were actually more DL attempts (including nonexistent messages) than the DL limit said: with the DL limit set to 300, I had 328 and 322 DL attempts (and I was getting locked out), then with the limit reduced to 250, it was 277 and 261 (and I no longer got locked out).

For groups with lots of deleted or unavailable messages, this might be pushing Yahoo to cut us off sooner than it otherwise would? Worth a thought, anyway.

So for the record, my current settings are now 250 messages, with a 65-minute wait. I completed my 19,000 message group with these settings.

Pres

[Wilson Logan]

Hi,

Yes, you're right. It is mentioned in the introductory email which I send every registered user.

2) When you're doing a refresh or initial download of multiple groups & you get a group you can't read e.g because Yahoo has changed the group format or many messages are deleted, PGO tries to get the first of 30 messages. It then assumes that the message has been deleted & then tries to get the next message *in sequence* not the next 30th message. This happens until the end of the group is reached or you get locked out (after about 200 of these errors). If you set the Workflow option so that you stop after receiving 10 invalid messages you can at least avoid being locked out. In future you can set that Group to be skipped during Refreshes. 

Cheers,

Wilson.

[Steeley]

FWIW, I used PG Offline to archive a group that was being closed after 15 years - 96,000+ messages using the default settings ... in one swoop! It took 8 hours, but there were no stalls or lockouts. I don't know if Yahoo changed something or I just caught their limit counter sleeping, but it is what it is. 8)

[Wilson Logan]

The lockout is modified by how active your group is.

So on a busy group you might get 10k before being locked out,  on a very quiet group you might get 120k.

Cheers,

Wilson.

[Steeley]

The lockout is modified by how active your group is.

So on a busy group you might get 10k before being locked out,  on a very quiet group you might get 120k.

Cheers,

Wilson.

Ah, OK..  the group wasn't particularly active (a few messages a day - down from a few hundred - over the past couple years - mostly just from a user spamming it and being abusive and obstinate - the owner inherited it from his brother who passed away and didn't want to moderate or let anyone else and so the volume fell off dramatically - eventually the owner just closed it as I suggested).

Lots of good stuff in there earlier that I was glad PGO was able to retrieve before the lights were turned off.

[Wilson Logan]

It happens a lot.

Yahoo tends to attract umm... mortally challenged individuals (ie old folk) so its small wonder that group founders fall off their perches and don't leave any clear line of succession.

Not assisted in any way by Yahoo.

How hard would it be to have a facility to appoint a successor?  You just list an email address and if you fail to login within a month, he gets control.

Easy, peasy, fully automatic.

 

[Steeley]

Well, in this case the original owner's relatively youthful age masked his mortality, and he did have moderators to carry on (of which I was one), but ownership passed to his older brother who was curator of all his other personal effects as well, a benevolent, but ambivalent sorta guy.

But what ultimately led to the demise of the group was a new member that was toxic and untrainable, that I ultimately banned, which generated a loud, long and persistent angst from a small cadre that dominated the traffic. Finally I just gave moderation to "Obstinate Prime" and bowed out. Turned out none of us (including him) recognized OP's mortal limitations either, and he followed the original owner into that long quiet sleep, but not before he let the guy I banned back in and OP at least came to appreciate why I banned the guy in the first place before suddenly and permanently retiring to HIS dirt bed (the guy wasn't hostile to MY authority as OP surmised, he was hostile to anyone's authority - All non-academic knowledge comes in suppository form).
Metaphysical question: If you're given an atomic wedgie at the moment of death, are you stuck with it for eternity? In this case, I hope so.
Alas, our nuvo-superhero of the new millennium - Toxic Man - was still a member and nobody who cared had access to the eject button. Thusly unfettered he blossomed into his full potential, and the writing, as they say, was written on the outhouse wall.

Rather than get into the middle of the circular firing squad the owner eventually just put it out of everybody's misery. The only one who posted vitriolic objection to the termination notice was Toxic Man. I'm still debating with myself whether to keep his last post as the inscription on the head stone when I republish the archive or scrub the group of evidence of his existence entirely from the final record which, given Yahoo's use of embedded reply chains makes decarboxylation of the metastasizing intellectual cancer anything but a trivial task. Thread and user sorting isn't discriminating enough - using those approaches, either a lot of babies would go out with the dirty bathwater, or a lot of bathwater remains stinking up the babies.
It's 96000 messages, remember? The only good thing I suppose is that only about the last half are potentially tainted.  :-\

Anyway the archive is supposed to display the accumulated and precious subject knowledge provided by the members, not an ugly case study of how to kill a discussion group. My debate is whether the hassle involved is greater than the pleasure derived. (I'll be severely chastised no matter what way I decide and I'm becoming weary of repeatedly falling on the sword of noble intent.)

So, some groups succumb to the owner's age or the fickle finger of fate sans wisely planned succession, some to mismanagement, some to neglect, and some to poisoning. This was a noxious combination of them all - hard to decide which to put down as the primary cause of death.

It happens a lot.

Yahoo tends to attract umm... mortally challenged individuals (ie old folk) so its small wonder that group founders fall off their perches and don't leave any clear line of succession.

Not assisted in any way by Yahoo.

How hard would it be to have a facility to appoint a successor?  You just list an email address and if you fail to login within a month, he gets control.

Easy, peasy, fully automatic.

[Wilson Logan]

Actually... Yahoo is pseudo threaded.


Each message has an individual message number.  Messages are atomic.

What joins them is the subject line.

That is, in & of itself, no guarantee of continuity as if you make a reply but change the subject line, the link is lost.


Truly threaded  forums (like Facebook) have separate Post and Comment types.

Comments cannot exist without their parent Post.

[Techsys]

This may come as a surprise for some people so I'm sharing my experience.

Last evening, I downloaded all 19 years of our group data, which was 91,336 messages with numerous attachments (excluding "winmail.dat"), 330 MB of photos, 36 MB of other files and a handful of links for a total of about 750 MB of data. No hiccups, no lockouts, no invalid anything.

Before I suppose how this could happen under the supposedly-watchful eye of Yahoo staff, there is a backstory that should be known to anyone who hasn't heard.

In a Business Insider report dated January 8, 2019, Verizon CEO, Hans Vestberg, clearly stated that AOL and Yahoo business units must survive on their own merits. It is then curious that, since this report, zero posts in Groups have been indexed whatsoever and are not included in any "Search Conversations" results. Also, no statistics for any group have been updated since January 12 but most groups currently show January 5 as the last activity which, of course, simply isn't true or even possible. Therefore, I am convinced that Yahoo is dying a slow death. This prompted great concern in our group to secure our data.

I can only imagine that I was able to download, uninterrupted, for one of the following reasons: (a) sadly, no one is watching or cares, (b) automated systems are not being maintained and are failing or have completely failed, or (c) non-residential IP addresses aren't flagged for attention.

After hearing all the scary warnings about pulling too many posts at once, I specifically spun up a Windows 10 VM on Azure, thinking I could change the assigned IP address as necessary. So I pulled 6K messages, stopped for dinner (because I wanted to watch it closely) then came back and said "to heck with it" and let her rip. When I went past 16K and then 25K, I was pleasantly surprised. Everything was done about 4 hours later.

Many, many thanks to Mr. Logan for providing this impressive utility. I will absolutely be registering.

The Business Insider report link: https://www.businessinsider.com/verizon-ceo-hans-vestberg-yahoo-aol-business-2019-1

[Wilson Logan]

Hi Jim,

That is some very interesting information.

Just as a caveat to that, I have been able to get 120,000+ posts from a single group in a single session.

The limit seems to be shared across the group. So, if your group is large, you get a large allowance.

The allowance is consumed by traffic to & from the group.

If the group is not heavily trafficked, a small number of people can have access to the total allowance.

TBF that is simply my guess at what is happening but it is based on nearly 20 years experience of downloading Yahoo groups.

Cheers,

Wilson.