Hi All,
As you may know I have been (trying to) learn VC# to take on the maintenance of PGO myself.
Its a big learning curve so I have engaged a new developer, Ihor from the Ukraine.
He has identified the main issue as being the data request format.
In the past Yahoo accepted HTTP requests but now it requires Web Browser control requests.
This is a problem because there any many methods within PGO which use HTTP request.
Our plan is to prove the concept by replacing the HTTP requests for login & message download first & build a test release.
You can expect a new release before the end of this week (30/01/2015).
Cheers,
Wilson.
Ps. is anyone able to login or get messages with any version of PGO now?
I need to know because it will mean this Yahoo change isn't fully rolled out.
I can still get messages from some of my groups. The last time this started I wrote messages about the groups I could not get to. These groups are the same ones that I am having problems with again. Unfortunately I am leaving on a 2 week trip so I will not be able to help much. These groups: BackpackGearSwap, etrex_legend, MCAK, OZI-USERSCE, and tidetool-notification-list are the ones that I can not get into from PGO.
Forgot to include I am using version 4.0.820 and running Windows 7 Pro.
I can still get messages from public groups but not from others. I should note that I can get these messages even if not logged in to Yahoo already. This is true whether I have the skip authorization box checked or not. However, I can not get to restricted or private groups
-Gabe
Bill, Gabe,
Well, that confirms what we thought i.e. login is not working.
I'd have been concerned if it was working.
Ihor has sent me a screenshot of his new version of PGO which has login & message downloading working.
We have an issue with VS2013 being unable to compile our installer project but I think its fixable.
Once we have a release, Ihor will send it to me & I will test it.
Now, if it works, should I release it? Remember, only login & download messages will be working. Files, folders, pictures & attachments will not be working.
Cheers,
Wilson.
Ihor sent me a new release last night (821).
It works, albeit, its as slow as a week in jail.
Interestingly it is now multi threading. I was able to download six groups simultaneously.
I have no idea what the limit on simultaneous threads is. There may not even be one.
Of course, this has played merry hell with the 'Stop' function as it has no idea what to stop.
But... all in all, quite interesting.
I will upload this version here later today.
Cheers,
Wilson.
File is here:
www.personalgroupware.com/groupmanager/pg-offline-4-0-821.zip
BTW expect a warning when you install:
"windows firewall has blocked some features of phantomjs is a headless webkit"
Basically you need to allow access.
This is the new way we are getting logged on to Yahoo.
Thanks to Wilson and Ihor for taking this forward. I've installed the new version but it just seems to sit and then hang when I select "refresh". I'll leave it for a while since you warned us it would be slow, and see what happens ...
Hmmm, 47 minutes now and still sitting there saying "Logging in with account [my username]". That may be starting to be a little long :-)
No, don't use Refresh.
The new multi threading has probably broken it.
Stop the program, restart it and select a group and choose Start.
Choose another group and choose Start. Repeat as necessary.
I'm am currently downloading 8 groups simultaneously.
Admittedly, its taking 2 minutes per page but I'm getting 8 groups at the same time.
Lets see what the Yahoo download limit makes of that...
Cheers,
Wilson.
When I look along the icons, all except Refresh look semi-greyed out. However, clicking on the group name then Start does the same as Refresh - "logging in ..." then just hangs. Sitting for 17 minutes now after using the "Start" button.
Ok... sounds like it might be a firewall issue.
Release 821 needs access through your firewall. Previous versions did not but then previous versions now can't log on to Yahoo.
If you open your Windows Firewall if you look in Inbound there should be at least two exceptions for phantomjs.exe
Please see the attached screenshot of my Windows Firewall.
Cheers,
Wilson.
Thanks, Wilson - here's my screenshot which looks like it has the same settings as yours. I rebooted, opened PGO, selected my group, pressed start, and eight hours later it seems to be still hanging. It still just says "Logging in with account [my username]".
Interesting development just now - 55 minutes after starting a new session, I got a "receiving messages for group" message, followed by the old "Object reference not set to an instance of an object" and then termination. But then a message from Yahoo saying an unexpected log-in attempt had been detected - screenshots below.
Curious to know if others have had this happen too.
Martin
NZ
Just tried a different group, and a multi-group download, with a weird result: the different group started downloading fine in batches of 25 messages; the second one gave me the "Object reference" error msg. I managed to catch both in the one screenshot:
Hi Martin,
That's interesting...
The phantomjs does a login but doesn't identify itself. Maybe Yahoo defaults the location to USA.
Having said that, I'm in the UK and I don't have the issue you're seeing although a user in the US is seeing the same thing as you.
Ihor (the new developer) has let me know about a very useful utility called Fiddler.
Basically it intercepts the traffic between your PC & the outside world.
Looking at its output should tell us at what point PGO is getting hung up.
Have a look at the attached screenshot of Fiddler from my PGO run.
I start Fiddler then I start PGO.
In PGO, I select my group (EDMHomeBuilders) and I press Start.
I switch back to Fiddler and I can see the requests PGO makes & the responses it receives.
If you can do the same we can see where PGO gets into bother.
The Fiddler application can be downloaded here:
http://www.telerik.com/fiddler
Cheers,
Wilson.
Thanks for the new build. Occasionaly I have the start up hang. I sometimes have to close PG Offline with Task Manager when this occurs. OS is Win7 64 bit. When it does connect it downloads slowly for a while. I have loaded a new group and it's MASSIVE with messages back to 2003 Only up to 2011 and message count is over 40,000 already. Twice PG Offline has stopped and an error message saying Yahoo has blocked connections from my ISP. I hope I am not causing some great trauma? ;)
Is there a database size limit?
As an aside, I ran the current build without seeing the do not use Refresh button post. Nothing drastic seems to have happened. I don't fully understand the possible consequences of using Refresh in this build....
Can databases be exported as normal text files for storage on media such as DVD? What's the best way to do this please?
Many thanks to you and your Russian colleague for your perseverance.
Quote from: Wilson Logan on January 24, 2015, 04:03:27 PM
Our plan is to prove the concept by replacing the HTTP requests for login & message download first & build a test release.
Quote from: Wilson Logan on January 27, 2015, 11:18:15 AM
File is here:
www.personalgroupware.com/groupmanager/pg-offline-4-0-821.zip
Thanks for the update Wilson & Ihor.
Back to full functionality for me. Downloads did stop at about 8000 messages on 2 groups I was downloading simultaneously. Restarting the downloads worked well, continuing on from where it stopped then downloaded over 25000 messages without any problems from the larger group.
Success rate with attachments was similar to previously. Less than half the larger pictures download but other file types appear to work much more reliably.
Your efforts are really appreciated.
Quote from: Basil on January 29, 2015, 11:53:48 AM
Can databases be exported as normal text files for storage on media such as DVD? What's the best way to do this please?
Do these threads help?
Export from PG Offline http://www.personalgroupware.com/smf/index.php?topic=991.0
Export from SQLite http://www.personalgroupware.com/smf/index.php?topic=1000.0
Hi Basil,
You WILL get locked out by Yahoo if you download <a certain amount> of messages in <a certain amount of time>
Only Yahoo know what these actual limits are but you will be locked out sooner or later.
You just have to wait <a certain amount of time> and you will get back on.
BTW yes there is a database size limit... 1.7 Tb.
Cheers,
Wilson.
Regarding Exporting data... it was a feature of PGO but is currently under repair.
I will ask Ihor when he looks at it to implement the long requested 'Export to RTF / CSV / WORD', etc.
Cheers,
Wilson.
Thanks Patch, and thanks Wilson, I will try the double selection once PG Offline has finished its chores, don't want to upset it right now, it's trundling along nicely, and Yahoo are being generous with my demands :)
My double selection seems to have got upset with me BUT I was able to open a second instance of PGO and download an entirely different group at the same time as I am downloading from the first PGO instance, so you could do it that way.
If simply starting a new download from within the same PGO instance fails.
Cheers,
Wilson.
I'll have a go at this - thanks. In the meantime, I tried a different approach with my "problem group" after last posting. I have a second, admin login to that particular group so used that with PGO and it accessed and updated the group instantly. No idea why the one valid login should work and the other not!
Quote from: Wilson Logan on January 29, 2015, 11:28:32 AM
Hi Martin,
That's interesting...
The phantomjs does a login but doesn't identify itself. Maybe Yahoo defaults the location to USA.
Having said that, I'm in the UK and I don't have the issue you're seeing although a user in the US is seeing the same thing as you.
Ihor (the new developer) has let me know about a very useful utility called Fiddler.
Basically it intercepts the traffic between your PC & the outside world.
Looking at its output should tell us at what point PGO is getting hung up.
Have a look at the attached screenshot of Fiddler from my PGO run.
I start Fiddler then I start PGO.
In PGO, I select my group (EDMHomeBuilders) and I press Start.
I switch back to Fiddler and I can see the requests PGO makes & the responses it receives.
If you can do the same we can see where PGO gets into bother.
The Fiddler application can be downloaded here:
http://www.telerik.com/fiddler
Cheers,
Wilson.
Hi Martin,
Double check the first login you used.
Also check that the Group Login is what you expect.
The Master Login and the Group Login are not necessarily the same.
Right click on the group name in the group pane on the left and choose Group Settings--> Download Settings to check the Group Logon
Cheers,
Wilson.
They were correct - the same settings would have been needed for the group that was successfully downloaded with the first login. I changed both when I switched to the Siddeley admin login. I've just put them both back and the program has hung again. At least if I stick with the admin login I'm OK for that one group, which is my main concern.
Quote from: Wilson Logan on January 29, 2015, 09:06:29 PM
Hi Martin,
Double check the first login you used.
Also check that the Group Login is what you expect.
The Master Login and the Group Login are not necessarily the same.
Right click on the group name in the group pane on the left and choose Group Settings--> Download Settings to check the Group Logon
Cheers,
Wilson.
Interesting...
I was downloading a different group and I stopped that & I just tried to download Siddeley and I got the 'Object Reference' error.
I stopped PGO, restarted it and now I can download Siddeley.
?
Wilson.
Odd that you're getting intermittent results with the Siddeley group too. I terminated the last "hung" session, uninstalled PGO, reinstalled, started off again with my normal login to download both Siddeley and the tyneside_northumberland group that normally works fine ... and this time it has just sat for six hours reading "logging in" for both groups with no further action.
Since I really only need to use PGO for the Siddeley group I'll just stick with the admin login from here and fingers crossed all will be fine.
Weird....
Just an update - with my admin login, the Siddeley group is happily updating automatically every two hours now. Perfect!
Martin
NZ
I'd still be interested to know why your other login isn't working...
I will contact you by email.
Cheers,
Wilson.