Version 4.0.800.568 (beta)

[danb]

(Report bugs for this release within this thread; or in redmine for the more advanced users.)

=============================================
Version 4.0.800.568 (beta)
https://dl.dropboxusercontent.com/u/42622/PgOffline/pg-offline-4-0-800-568-setup.msi

Fixed: bug#1878 message content now cleared when message is deleted
=============================================

[txpigeon]

Dan,

No bugs discovered this time (yet? ;>)  I know you're doing this part time and have to set priorities.  I won't speak for others, but the most important thing for me right now would be if the Speed or Repeat worked.  It would be so comforting to start it and wait for the group to download.  The limit seems to be about 2400/hour (40/minute) based on my experience, but it may be slightly less.  I've been getting 2200, stopping, wait an hour, and repeat.  A group with a few thousand messages wouldn't really be a problem.  The one I'm currently working on is almost 150,000 though.  I've managed to lose track and lock myself out several times.  Lock-out has run from 2 to 4 hours.

I surely appreciate all you're doing and have done.  If I can be of any help, please holler at me.

Thanks,
Duane

[danb]

The messaging aspect is what we want to get nailed down first, so the speed and repeat fall within that category.   I'll be looking into this now.

[txpigeon]

Many, many thanks!

Duane

[txpigeon]

I noticed a small glitch yesterday.  I hit the Stop button and it stopped.  The last few posts that it was in the process of getting never showed up in the message window.  When I tried to restart to include those messages, it said it was skipping existing messages.  Not a serious problem since it only missed 1 or 2, but something to look at one day.  Goes back to last received message and works fine when I get locked out and it can't parse the messages.

Duane

[txpigeon]

I found another difficulty.  On one of the groups I'm trying to archive, it seems to have a problem with posts that are over a certain size.  I suspect you have a limit for the message field and these are bigger.  Some folks weren't very good at trimming, so posts can get rather large.  I've found quite a few with the problem in the 3-10KB range.  In at least one case, a single post was 36KB.  It seems to retrieve the message, but spills over into additional records when exported.  One that I checked doesn't show all the message in PGO, but exports most of it before going into the spillover mode.

Duane

[Wilson Logan]

Interesting. Its probably to do with the database schema implemented for SQLite in Version 4.

It should be simple to fix. Just a matter of resizing the field.

Cheers,

Wilson.

[txpigeon]

I just figured out another thing that's confusing the program.  If the Subject line starts with a double quote, ", it doesn't show the subject, but loads the message starting there.  It also rolls over to the next few records and throws things off.  Probably due to someone not realizing they were in Subject instead of Body.

Wilson, Re: field size

That's what I figured, but I'm not sure what the upper limit would need to be.  I know I've seen cases on one group where someone included an entire Daily Digest in their reply, over 130KB.  It's difficult to stay ahead of those that are unfamiliar with computers.  I'm relatively sure that SQL would only use the space needed for the actual content so it wouldn't really take more space for the database, but I'm not all that familiar with it.  Is that how it would work?

Duane

[Wilson Logan]

Hi Duane,

You can have fixed or floating maximum field sizes. A floating (effectively unlimited) field size avoids any possibility of running out of space for a particular field but runs the risk of someone doing something stupid, like entering several megabytes of data.

If this was a company internal program then you could simply forbid employees from doing stupid things but PGO has to interface with the rest of the world and well, you know what that means.

Sensibly, to protect itself PGO needs to have a maximum field size but I'm open to discussion on how big this should be.

Cheers,

Wilson.

[txpigeon]

Wilson (& Dan),

I agree that there needs to be a limit, but it's going to be difficult to figure what it should be.  If everyone always top-posted (or bottom-posted), then the first (or last) 2-3k characters would be enough to maintain the integrity of the messages.  There doesn't seem to be any consistency on the several groups I've looked at, so that makes it harder to set a limit.  While missing one message, or part of one, seems minor to me, it could be very important to others in the group.

As I said, I'm not very familiar with SQL, so this idea may not be possible.  Have the field limited to some length, but insert extra record(s) for a long message containing the additional characters only when needed.  It appears that currently there is a column for the record number and one for the message number.  The record and message numbers are now integers.  Could they be changed to allow insertion of extra records to include "sub-records", such as 192.1 or 192.11 without causing undue risk or extra work?  I'm trying to do some research on SQL as I have time, but it looks like I have a lot to learn!

Thanks,
Duane

[Wilson Logan]

Hi Duane,

A quick check of SQLite informs me that it has only variable length records i.e. if you set a max length of 100 and the field is 20 chars it only uses 20 chars.

This would imply that it is safe to set a very high max record length without hogging massive amounts of space.

I will discuss this with Dan when he gets back.

Cheers,

Wilson.

[txpigeon]

A bit more info on the large post "problem".  I used SQLiteStudio to look at a db3 file directly.  All appears to be fine there.  It's when I export to a csv file that the "overflow" shows up in unexpected cells.  I now believe the information is being stored correctly.

Duane

[lklawrie]

I'm replying here but the beta looks to be 4.0.800.567 (not 568).

I wanted to start a new test database (to see if problem reported in a few sentences was  areal problem), but trying to import my old databases (import from folder, get list, import), gives me an immediate error:

SQLite error (787): abort at 39 in [INSERT INTO discussion group (name, ...)

Is there something else I need to do to start a new DB?  Can I remove the current db it generates in Appdata and it will automatically start a clean one?

In anycase, some of the old databases I have imported seem to be "empty" and would have to get the entire database again -- which would be fine for some and interminable time for others.

Linda

[danb]

A bit more info on the large post "problem".  I used SQLiteStudio to look at a db3 file directly.  All appears to be fine there.  It's when I export to a csv file that the "overflow" shows up in unexpected cells.  I now believe the information is being stored correctly.

Duane

If you attempt to open a CSV with some large cell data then you may have a problem.  Excel will have a low limit on a cell's text length.