Yahoo d/l limit

[JoeRoss]

Okay, I managed to piss my wife off by getting PGOffline installed on her computer, but darnit I need to play with this program! ;D

Anyway, here's what I found by playing with the new feature (Download speed limit).  Starting with a group containing 442 messages, I successfully retrieved 201 at 25 msg/min before I started getting "invalid page" errors. Anyone having similar problems or suggestions?

[Wilson Logan]

Joe,

We're all beginning to think that the limit is somehow related to time vs messages. I've had a few reports now that if people download say 200 messages & then wait a while (say an hour) and then download another 200 messages, all is hunky dory.

I will be implementing the following functionality ASAP.


'Download x messages then wait n minutes and repeat y times'

Where 'x', 'n' & 'y' are all user specified variables.  

Hopefully this should let you do unattended downloads.


Cheers,

Wilson.

[Brett]

I just installed yesterday and have been getting cut off by Yahoo.  I still have not figured out the limits.
* My fist time downloading, I got 842 messages before getting cut off.  This was a LAN connection, moderate speed.
* At home, on my broadband connection, I only got a few hundred messages before the invalid page.  Interesting that if you try to go to groups.yahoo.com with your browser, it will not let you in.  It must lock out your IP address.
* At home, I also have a dial-up connection.  I tried setting the msg/sec limit to 5 messages per minute, but it did not help.  I was still cut off after a few hundred messages.
* There is a nice side-benefit to the dial up connection.  After I get timed out by Yahoo, if I disconnect, and re-connect, I can download more messages.  I assume it is because I'm getting a new IP address.

I would REALLY like to know what the rules are that Yahoo has set up for cutting people off for too much downloading.  I have 11,000 messages in the group I moderate.  I find that the Yahoo archive search only searches the first few thousand - so I need a better search program (like this).

[Brett]

One manual work-around is to use the Group Setting /Download Settings box and specifiy a smaller range of messages.  So on your first download, you set it to 1 to 150 then it will stop.  Then a while later set it to 151 to 300 and start it again, etc.  At least you can prevent it from going over the limit and getting locked out.

The other work-around that I mentioned before works well.  Use a dial up connection and when yahoo times out.  Just disconnect, dial again, and away you go.  I'm getting 320 messages at a time with this method.

[Wilson Logan]

Version 2.0.160 is now available. It has functionality which allows you to do 'download, wait, repeat' to avoid the Yahoo download limit.

http://www.pgoffline.com/groupmanager/pgpatch20160.exe

 The new functionality is in File--> Options--> Download. The initial download is set at (I think) 150 and the wait at 15 mins.  Testing would suggest that a download of 300 & a wait of 60 mins works well.

Cheers,

Wilson.

[prochargedmopar]

I sure am glad I got all my groups before all this bull.  Now I just update every few days and I've never had that problem.
whew.

[Wilson Logan]

Todd,

I agree. I have a group (7X10minilathe) which has over 100,000 messages. Getting that 300 at a time would have been a bit annoying.

However, if you've got 20 buddies its amazing how many hands make light work.
20 guys could download 100,000 messages in just 16 hours even sticking to 300/hour.

Cheers,

Wilson.
 

[AceHarddrive]

You said 300/60?  Is that without doing  anything else (web surfing...)?  What I mean is you set it, activate it, turn off the monitor and let it work overnight, right?

~Tim  :o

[Wilson Logan]

Tim,

Yes, just set it up, switch off the monitor & go to bed.

Cheers,

Wilson.

[monte]

fyi, I was downloading 300, then wait 60 min and I got over 1000 messages from a group I was downloading.  Then I went and started browsing the group online and promptly got cut off again.

Monte

[Wilson Logan]

Monte,

 Then Yahoo must be accumulating the total amount DL'd by any one IP. Its not happened to me but I'm in Europe & their limits seem to be much laxer here.

I guess if you want to surf Yahoo Groups after DL'ing with PGO you need to DL a bit less than 300 as 300 appears to be at the ragged edge of the limit.

Cheers,

Wilson.

[Pres]

I've just begun fiddling with PG Offline, but it's been happily plugging away for me for the last 12 hours by setting it to 300 messages and a 61 minute delay. I don't really know if the "1" is important, but I figure I'd hate to trigger a lockout if it were at something like 59 min and 57 seconds, so I gave it an extra minute.

I tweaked it on one group with 2800 messages,  and when that finished just before going to bed last night, I began a group with 19,000 messages. Got 2100 so far... :)

Oh yeah, I'm in Japan, but accessing US groups. I was getting lockouts similar to the ones reported for people in the US, so perhaps it's only Europe that gets nice lax DL limits.

Pres

[T-Bone]

Well I am able to only d-load about 150 messages in a session.  I am in the United States and have a dial up connection.  Does the 60 minute wait interval seem like a good starting point for most of you guys?

I am aware of that I can change my IP address by redialing  but I am trying to optimize unattended downloads.

Thanks,

-T

[Pres]

I found that in the group that I was DLing there were a fair number of deleted messages. What I think might be going on is this:

When you set PG Offline to download, say, 300 messages, it seems to keep downloading until it actually gets 300 valid messages -- meaning that any invalid/deleted message numbers don't count against the DL counter, and you wind up pounding Yahoo for that many extra times.

For example, I found that by subtracting the highest downloaded message in one "DL block" versus the highest from the previous block, there were actually more DL attempts (including nonexistent messages) than the DL limit said: with the DL limit set to 300, I had 328 and 322 DL attempts (and I was getting locked out), then with the limit reduced to 250, it was 277 and 261 (and I no longer got locked out).

For groups with lots of deleted or unavailable messages, this might be pushing Yahoo to cut us off sooner than it otherwise would? Worth a thought, anyway.

So for the record, my current settings are now 250 messages, with a 65-minute wait. I completed my 19,000 message group with these settings.

Pres

[Wilson Logan]

My thoughts are that this limit is bandwidth based, so failed download attempts don't add (significantly) to the bandwidth used.

What does definitely vary is message length. Lets say that the limit is 2.4 Mb. That means that you could get three hundred 8k messages before being kicked. What if users on your group are a bit more chatty & the average message length is 9k?  I think that you can see that number of messages can only be a guideline when it comes to setting up downloads.

Cheers,

Wilson.