843506 - inconsistent storing of popstate

Status: RESOLVED INVALID

(MailNews Core :: Networking: POP, defect)

Description

[betterme]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101 Firefox/18.0
Build ID: 20130201065344

Steps to reproduce:

I am using my maildatabase alternatingly on my desktop and my notebook.
All local email is stored below the ..\mai\.. path. This has worked for years since seamonkey times.
I am using Thunderbird in the flavor german, portable application.
And I leave the POP3-messages on the server as a backup-option - and to be able to download them in parallel from different places.


Actual results:

When I switch machines, mails that already got downloaded on the other machine get downloaded again. Sometimes just few, sometimes hundreds - as if the other machine has a different "opinion" on what the last popstate was.
Of course the POPSTATE.DAT file is in the mail-folder.
The problem stays, even if I do a POP3-download on the target machine AND the source PC, switch then, but keep the target machines POPSTATE.DAT.

I have experienced this problem for many months now.
I suspect the information about the popstate is no longer stored consistently below the ..\mai\.. folder or in the POPSTATE.DAT, but - at least partial - also in paths of the local user profile or the local installation.
This becomes more obvious in my way of using thunderbird, but may cause the same annoying effects to anyone doing a restore of a backup of her/his mailbox.


Expected results:

The other firefox-"installation" (portable, so installation means only files in another directory on the other machine) should only download newer emails than the ones already POP3-ed - or none, if no new ones have arrived.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to betterme from comment #0)
> Expected results:
> The other firefox-"installation" (portable, so installation means only files
> in another directory on the other machine) should only download newer emails
> than the ones already POP3-ed - or none, if no new ones have arrived.

You set "Product: Thunderbird", and popstate.dat is usually Thunderbird's file, and "POP3" is usually term in mailer.
Which is actual software? Program called Firefox? Program called Thunderbird? Or other?

What is your actual environment and operations?

Official Thunderbird? "Thunderbird Portable" which is modified version of official Thunderbird?
If Official Thunderbird, how do you do "switch machine" with using same popstate.dat?
If "Thuderbird Portable", how do you do "switch machine" with using same popstate.dat?

In ay case, where is "Tb's profile directory what is used at switched machine" located? USB memory?

If USB memory or HDD via USB, when do you pull off device from USB port?

Anyway, format of popstate.dat is pretty simple and is text file - an entry is "status + UIDL + TimestampValue", where status is k(keep), d(delete), p(partial) like one. And, file is closed by Tb after each POP3 access, so "delete popstate.dat while Tb is running" is a very simple/easy way to certainly force re-download of all mails from POP3 server.
UIDL is written as X-UIDL: header in message source if POP3.
And, POP3 data retrieving and UIDL is very easily tracked by getting POP3 log.
( See bug 402793 comment #28 for getting protocol log )
  Tb uses LIST command to know message number and size,
  Tb uses UIDL commad to know UIDL for message number,
  Tb uses RETR command or TOP 0(header only) or TOP nnn(line, by size limitation)
     for message data retrieving
"Merely saying 'inconsistent storing of popstate'" won't help problem analysis by developer.
Show following data to developer, please.
(1) popstate.dat before restart of Tb, with file meta information
(2) following data for each POP3 access
        popstate.dat before POP3 access, with file meta information
      + POP3 log with timestamp for a POP3 access
      + popstate.dat after POP3 access, with file meta information
(3) popstate.dat after termination of Tb, with file meta information
Because mail data itself is not relevant to problem, remove irrelevant data and private data from log file before attaching log file to this bug, please. (never paste long data to bug, please)

> at least partial

Because you looks to use MS Windows, "Process Monitor" of Sysinternals is usable to track file I/O of popstate.dat by Tb.
  http://technet.microsoft.com/en-us/sysinternals/bb545021
  http://technet.microsoft.com/en-us/sysinternals/bb896645
If problem in Tb's "file writing" is suspected, show developer "Process Monitor log data", please.

Comment 2

[:aceman]

Yes, it is also not clear to me the exact setup of the reporter.

Do you have the TB mail storage on a USB stick, that you plug into several machines, but always download new messages to the same USB storage?

Or do you have the mail storage on one machine but run several instances of TB (one local, one e.g. from the USB stick (portable TB)) that both write to the same TB storage (profile)?

Comment 3

[betterme]

I used to use a USB-Stick, but nowadays I close the application on the one machine (e.g. desktop), then copy the complete mail-directory with all subfolders to the other machine (e.g. notebook) and start TB there, because it works faster then, and I can encrypt the drive with truecrypt. I store the TB-mail-data on a FAT32 drive, so file permissions can not be an issue, too.
As the paths are different on the two machines, I do NOT copy the "installation", i.e. the application directory with the settings.

I use this
http://portableapps.com/de/apps/internet/thunderbird_portable
so thunderbird does not need a classical install, and should store all settings locally, not in the windows profile.
As far as I understand, this works like a wrapper around TB. After installing once, I automatically get all update notifications, downloads and installs up to now using the latest 17.0.3.
I do this, because in the past, when I needed to restore my Windows, I had side-effects like need to reconfigure all my mail profiles - very annoying. From my logic, this should be impossible with a portable application and has never again been a problem since then.

I have not changed the location of the POPSTATE.DAT, it resides directly under the mail-directory and get's copied along with that whole directory. But as described, trying with replacing it (after copying) with the one from the local machine - before starting TB of course - does not chance the problem.

I need to check how to deliver logs etc. that you requested.

Comment 4

[betterme]

I had a short check of my POPSTATE.DAT in the light of your explanations.
As I use the option to leave my mails on the server, it get's quite long, roughly 1MB, 28.000 lines (so roughly 28.000 emails on the server I interpret this), but I can upload the versions from different times.
Deleting it to make TB reload all Emails will cause no small traffic, so I would do this when at a highspeed broadband connection. I understand if the POPSTATE.DAT got somehow "messed up", I should have a "clean" one afterwards to check if the problem persists.
I'll keep you updated.

Comment 5

[betterme]

Attachment: POPSTATE.DAT initial

Comment 6

[betterme]

Attachment: After deletion, all ~11.000 mails were redownloaded - as expected, and this file was created.

Comment 7

[betterme]

Attachment: After copying the whole mailfolder, including the newly created POPSTATE.DAT to the other PC and starting TB there, AGAIN all ~11.000 mails were redownloaded and this is the resulting POPSTATE.DAT

Comment 8

[betterme]

I counted roughly 11.000 mails downloaded, I don't know if I counted the lines in my POPSTATE.DAT wrongly or how this translates to the 28.000 lines. But this doesn't change the problem.

Observation:
Since the double redownload of all emails, the multiply downloaded emails when switching machines seem to be roughly the ones since that date. Number growing with each switch.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to betterme from comment #5)
> Created attachment 723437 [details]
> POPSTATE.DAT initial

> *pop.gmx.net 738429

(In reply to betterme from comment #6)
> Created attachment 723439 [details]
> After deletion, all ~11.000 mails were redownloaded - as expected, and this file was created.

> *pop.gmx.net 738429

(In reply to betterme from comment #7)
> Created attachment 723441 [details]
> After copying the whole mailfolder, including the newly created POPSTATE.DAT
> to the other PC and starting TB there, AGAIN all ~11.000 mails were
> redownloaded and this is the resulting POPSTATE.DAT

> *pop.gmx.de 738429

pop.gmx.de != pop.gmx.net, even when same IP address is used for them, and server, account, Mbox is absolutely same one.
So, it's not "redownload". Phenomenon is simply "data in popstate.dat file is data for diffrent POP3 server, then downloaded from scratch".

In account definition, there are two kind of server name, userID.
(1) Internal key to identify the account. For example, used as key of Password Manager data.
  mail.server.serverN.hostname
  mail.server.serverN.userName 
(2) Actual FQDN and userID for login, If it doesn't exist, hostname / userName is used for login. This entry is created when server name field / user name field is touched or altered at Server Settings UI.
  mail.server.serverN.realhostname
  mail.server.serverN.realuserName

I don't know which is used for popstate.dat.
As hostname/userName is used in IMAP logging(confusing when realhostname!=hostname, realuserName!=userName), I guess hostname/userName is used in popstate.dat too.

Is same account definition used at any PC?

Comment 10

[:aceman]

Good job, WADA. I think realHostName is used in popstate.dat, at least the last occurence of the header in the file.

Comment 11

[betterme]

Thanks for the quick response.
I checked the configurations, it is the same OP3-mailbox, but on one machine I use the email-adress as login, on the other I use my user-ID - the server allows both.
Tricky, I did not know that TB would treat this like two separate mailboxes.

I aligned this now and will observe further.

Comment 12

[betterme]

The problem persists with aligned configurations.
I will try to create and upload a new set of POPSTATE.DAT, but due to the volume of the mailbox I can't do this quickly.

Comment 13

[WADA:World Anti-bad-Duping Agency]

Closing as INVALID, per provoded popstat.dat data by bug opener.
Re-open please, if closing as INVALID is wrong, with attachig sufficient evidece data of "cause is Tb's flaw in code" and detailed description about "what is flaw in code", please.

Comment 14

[:aceman]

Which means if you can provide new popstate.dat files that will show the hostnames+usernames are the same and TB still acts badly, we will reopen this.

Comment 15

[betterme]

I still encounter the problem and have archived POPSTATE.DAT files.
1.POPSTATE.DAT - source machine, after POP, before transfer maildatabase
2.POPSTATE.DAT - target machine, after POP, before getting maildatabase transferred
3.POPSTATE.DAT - target machine, after getting maildatabase transferred, then new DL'ed POP3 what thunderbird thought to be new emails
4.POPSTATE.DAT - source machine, after POP, before transfer maildatabase
5.POPSTATE.DAT - target machine, after POP, before getting maildatabase transferred
6.POPSTATE.DAT - target machine, after getting maildatabase transferred, then new DL'ed POP3 what thunderbird thought to be new emails
7.POPSTATE.DAT - source machine, after POP, before transfer maildatabase
8.POPSTATE.DAT - target machine, after POP, before getting maildatabase transferred - target machine, after getting maildatabase transferred, then new DL'ed POP3 what thunderbird thought to be new emails
9.POPSTATE.DAT

Comment 16

[betterme]

Attachment: 1..9.POPSTATE.zip - see comment #15

Comment 17

[Wayne Mery (:wsmwk)]

Comment on attachment 786364 [details]
1..9.POPSTATE.zip - see comment #15

(feedback and review are for patches)

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to betterme from comment #16)
> 1..9.POPSTATE.zip - see comment #15

Why file size\line number of N.popstate.dat increases by each your "transfer to other PC"?

Even if all mail is re-downloaded by your "transfer to other PC", number of lines in N.popstate.dat should be same, and differnce should be timestamp part of each line only.

Following is seen in all N.popstate.dat files.
> *pop.gmx.de 738429

Actually absolutely same POP3 account definition, except server number, in both Tb?
> mail.server.serverN.hostname
> mail.server.serverN.realhostname
> mail.server.serverN.userName
> mail.server.serverN.realuserName

Following is first entry just after "*pop.gmx.de 738429" line.
> 1.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362237815
> 2.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362231223
> 3.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362237815
> 4.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362237815
> 5.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362231223
> 6.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362237815
> 7.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362237815
> 8.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362237815
> 9.popstate.dat : k 0LgMBo-1TKI122oSq-00niGg 1362231223

Is clock of PC correct and accurate at both PC?
Did you cleanly terminate Tb at old PC before your "transfer to other PC" operation which involves "copy of popstate.dat file"?

Comment 19

[betterme]

Answers =>

Why file size\line number of N.popstate.dat increases by each your "transfer to
other PC"?
=> I'm not changing it. I guess, the lines add up as TB downloads old messages, that it thinks are new.
Not all emails are downloaded newly. It is roughly as much, as were downloaded on the target machine since last transfer of the maildatabase in the opposite direction. That is, why I think, that state must be stored somewhere on the specific machine, but outside the mail directory.

Actually absolutely same POP3 account definition, except server number, in both
Tb?
=> Rechecked:
Server type: Same
Server: Same
Port: Same
User name: Same
Security and authentication: Same

mail.server.server1.hostname: Same
mail.server.server1.realhostname: Same
mail.server.server1.userName: Same
mail.server.server1.realuserName does not exist - same


What IS different: 
On the one machine, server settings / extended options is
seperate inbox ("Separater Posteingang")
and on the other machine it is
global inbox ("Globaler Posteingang").
Reason is, I have - at least with the previous versions when I set it up - not been able to choose the desired folder together with global inbox on the one machine.

Is clock of PC correct and accurate at both PC?
=> almost to the second. Automatically set from internet every few days

Did you cleanly terminate Tb at old PC before your "transfer to other PC" operation which involves "copy of popstate.dat file"?
=> Yes, in all repeated cases that I provided the popstate.dat.
BUT, when I have a crash of the PC while TB is running (which I experience with Windows roughly once in 2 weeks .. 2 months), I have a similar effect WITHOUT any second machine or transfer of files involved: When I restart the PC and TB, old mails are downloaded as new ones, often many days backwards. I suspect the reason is the same - inconsistent or at least not proper storing of what mails were already downloaded. But I can't reproduce this - or at least, I would prefer not to power off my PC during operation as a test, so I have not. It is my productive machine, not some testing device that I want to treat that way.

BTW: Thanks for your quick response!

Comment 20

[WADA:World Anti-bad-Duping Agency]

Following is content of sorted 1.popstate.dat and 9.popstate.dat.
(sork key=column1 to line length)

(1.popstate.dat, number of lines = 24171)

# POP3 State File
# This is a generated file!  Do not edit.
*pop.gmx.de 738429
*pop.gmx.net 738429
k 018W48-1QgvsV0qc6-002oEV 1362229395
k 018W48-1QgvsV0qc6-002oEV 1362236755
k 018W48-1R3iaw1Lys-002nK9 1362229592
k 018W48-1R3iaw1Lys-002nK9 1362236874
k 018W48-1R4LTQ2nxC-002nYF 1362229585
k 018W48-1R4LTQ2nxC-002nYF 1362236872
k 018W48-1R5uOW40LK-002sxl 1362229553
k 018W48-1R5uOW40LK-002sxl 1362236843

(last 4 lines of 1.popstate.dat)
k 0MhnsY-1QVmBn2wd9-00MZJF 1362229091
k 0MhnsY-1QVmBn2wd9-00MZJF 1362236602
k 0Mhnte-1S3Rnk0KSx-00Muz4 1362229929
k 0Mhnte-1S3Rnk0KSx-00Muz4 1362237071

(1.popstate.dat, number of lines = 27473)
# POP3 State File
# This is a generated file!  Do not edit.
*pop.gmx.de 738429
*pop.gmx.net 738429
k 018W48-1QgvsV0qc6-002oEV 1362229395
k 018W48-1QgvsV0qc6-002oEV 1362236755
k 018W48-1R3iaw1Lys-002nK9 1362229592
k 018W48-1R3iaw1Lys-002nK9 1362236874
k 018W48-1R4LTQ2nxC-002nYF 1362229585
k 018W48-1R4LTQ2nxC-002nYF 1362236872
k 018W48-1R5uOW40LK-002sxl 1362229553
k 018W48-1R5uOW40LK-002sxl 1362236843

(last 4 lines of 9.popstate.dat)
k 0MhnsY-1QVmBn2wd9-00MZJF 1362229091
k 0MhnsY-1QVmBn2wd9-00MZJF 1362236602
k 0Mhnte-1S3Rnk0KSx-00Muz4 1362229929
k 0Mhnte-1S3Rnk0KSx-00Muz4 1362237071

(Number of lins of N.popstate.dat)
   1.popstate.dat : 24171
   2.popstate.dat : 24355
   3.popstate.dat : 24455
   4.popstate.dat : 24623
   5.popstate.dat : 24702
   6.popstate.dat : 24791
   7.popstate.dat : 26362
   8.popstate.dat : 26871
   9.popstate.dat : 27473
   "Newly added lines per a your transfer to other PC"  is not same.

(A) Grabage of following exists since initial of popstate.dat
  *pop.gmx.de 738429
  *pop.gmx.net 738429
(B) Duplicatde entry exists since initial of popstate.dat

Please check with "clean popstate.dat data", if you are not testing Tb's behavior when multiple "server name + data size" lines are generated by copy of popstate.dat file by user, and when duplicated entry already exists in popstate.dat file which was originally produced by copy of popstate.dat file by user.

Comment 21

[WADA:World Anti-bad-Duping Agency]

Closing as INVALID again.

If you think following is Tb's bug, open sepatate bug, please.
  When popstate.dat file is copied between Tb with hostname=pop.gmx.de
  and Tb with hostname=pop.gmx.net, following entries are generated in
  popstate.dat of Tb.
    *pop.gmx.de 738429
    *pop.gmx.net 738429
  If this popstate.dat is copied between multiple Tbs and is used by
  the multiple Tbs, duplicated entries are generated in popstate.dat.
  "Which of duplicates in popstate.dat is used by Tb" is inconsistent.