Yahoo d/l limit

[T-Bone]

I would agree that the limit seems to be bandwidth based.  For that reason I get on average between 150-175 messages at a time.  But whats odd is that you folks can get upward of 300 messages.  I looked over my messages and they are definately on the small size with no attachments.


Anybody try the per minute download limitation?  (to emulate as if you were just clicking next on your broswer)


-T

[Wilson Logan]

Per minute download was our first solution but it looks like the limit is not only bandwidth but load based. Downloading messages constantly at exactly the same time interval is the hallmark of an automated system and I think Yahoo is set up to be sensitive to that pattern of downloading.

 Put it this way, if I was them thats what I'd be doing.

Cheers,

 Wilson.

[Brad]

Shure seems like they've changed their parameters.  I'm failing around 200 messages now whereas it was over 300 now.

Brad

[Wilson Logan]

Brad,

 I think we have to expect that. If I were them, I'd keep lowering the limit until I started to get plenty of complaints from ordinary users, then I'd up the limit 10% as a safety margin. I guess they're still experimenting.

Your best defence is to refresh your groups as frequently as possible.

Cheers,

 Wilson.

[Wilson Logan]

Maybe what I need to add is a 'Share' feature.

In Download Settings (next to the Skip properties) there would be a new radiobutton group:

  'Refresh & Share'  or  'Refresh from PGO'

The first option allows you to download from Yahoo & then upload to the PGO server.
The second option allows you to get new messages only from PGO.

E.g.

Bob, Tom & Graham are all members of Hobbicast, Casting & Southbendlathe.

Bob checks the 'Refresh & Share' for Hobbicast and 'Refresh from PGO' for Casting & Southbendlathe.

Tom checks the 'Refresh & Share' for Casting and 'Refresh from PGO' for Hobbicast & Southbendlathe.

Graham checks the 'Refresh & Share' for Southbendlathe and 'Refresh from PGO' for Hobbicast & Casting.

This way each guy only downloads one group but gets the messages for all groups he's interested in.
Its all about teamwork.

Thoughts?

 Wilson.

[Pres]

That'd be cool! If you make it easy to share archives (i.e. allow us to do it within PGO) then I think you'd get a lot of people willing to share. I've currently got about 170MB of messages total from two groups that I'd be happy to share, but if I have to use a CD-R or something like in the tutorials, then it's like, who am I going to mail it to from way over here in Japan?

This would reduce the load on Yahoo, but then put it on PGO. Any given group would probably have limited downloads, but put them all together and that's a lot of disk space/bandwidth...will your server be OK?

[wundrbee]

What about having it setup so that there's an option to download from the PGO server and be able to add additional servers. Eventually PGO's space would run out. By letting us add our own, we'd be able to publish the info on the PGO website and add it to PGO.

Jim, bob, and Joe are on such-and-such list Jim and Bob have there own web space, they put the server info on on PGO's website. Joe doesn't have webspace so PGO let's him upload. They each then add the server info from Jim and Bob to the Groups Properties. Then hit "refresh using all sources" or something like that.

It sounds good - but I'd be concerned that it might leave a nice loophole for unscrupulous folks to dig around in other files on your web server.

Geez, I hope I typed that to where it made sense. I just woke up and haven't made it through my first cup of coffee yet

Tammi

[lorraine]

Shure seems like they've changed their parameters.  I'm failing around 200 messages now whereas it was over 300 now.

I'm actually downloading around 400 messages this morning before getting blocked on a dial-up.

Lorraine

[Brad]

Your message gave me hope that I'd solved the problem.  I had a theory that it was throttling on yahoo's end dependent on system load.  If so, we could avoid downloading on weekends.

So I reset my program to d/l 300 before pausing.  If failed at 152.

Oh well, blows my theory out of the water.

Brad (cable modem downloading at around 60 mes/min.

[Brad]

One thing I'll mention is that when you time out, the program thinks 6 or so messages are unavailable, and will skip them when you resume downloading.  You need to use the "Find Missing Messages" option and  download them later otherwise you will have gaps of valid messages in your database.  I'd wait until I had completed the complete download before letting the program locate and download the missing messages.

Brad

[Wilson Logan]

Pres,

>Any given group would probably have limited downloads, but put them all together and that's a lot of disk space/bandwidth...will your server be OK?

I have 20GB a month. Thats good for about 2.6 million messages. Also, this feature is just for catching the latest messages, not for DL'ing whole groups, thats what the zipped archives are for.

The short answer to your question is... yes!

Cheers,

 Wilson.

[Wilson Logan]

Tammi,

 >What about having it setup so that there's an option to download from the PGO server and be able to add additional servers.

My original idea was to have a Napster style file sharing facility but I'm not happy about the security of such a thing, hence all traffic goes through my server so that no-one is exposed to an unknown 3rd party.

Your idea for adding servers has merit though.

Cheers,

 Wilson.

[ginahoy]

Hi Wilson and all,
I've just started downloading large blocks for the first time tonight. I started with 175 messages. For the particular group I was downloading, this turned out to be 3.48 MB (20kB/msg). I recorded this with "DU Meter" a utility that displays a horizontal-scroll bar graph of upload/download rate and also has a built-in upload/download stopwatch.

To test your theory that the Yahoo limit is bandwidth-based, others should post examples of how many bytes and messages they transfer without pausing. Or better yet, how many bytes had been transferred when they hit the limit and get kicked.

Is anyone aware of another program for monitoring download rate and volume? Unfortunately, DU Meter must be purchased. As and aside, I considered checking the pgo's database (.mdb) file before and after the download, but on second thought,  I have no idea how closely the record size compares to the number of bytes downloaded. Retransmitted bytes, IP overhead, etc, must also be considered. A download monitor seems like the only way to do this with any accuracy (hint, hint!)

David Butler

[wundrbee]

My original idea was to have a Napster style file sharing facility but I'm not happy about the security of such a thing, hence all traffic goes through my server so that no-one is exposed to an unknown 3rd party.

Wilson,
This was my concern also - It was just too early to articulate it very well!

Tammi

[Wilson Logan]

David, Tammi,

 I did ask the developer about adding a download meter for the very reasons you've mentioned but because the bandwidth used isn't just the data downloaded, its simply too big a task given that the feature is a 'nice to have' & not mission critical.

I'm not saying we won't come back to this but not until we've got through our current development queue.
 

 Cheers,

 Wilson.