1762327 - Thunderbird repeatedly downloading 1000"s of old pop messages. Caused by corrupted popstate.dat?

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Reg Cardin]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101 Firefox/98.0

Steps to reproduce:

Running Thunderbird 99.0b2 beta on Windows 10 Pro, downloading from Cox mail server. Last week, Thunderbird downloaded 100's of old messages from 2014 through 2020. Read online doc, deleted .dat file (doc suggested corruption and to delete) and restarted Thunderbird. Tbird downloaded >20K messages. I deleted these since they were duplicates. Thunderbird, when restarted, continues to download old messages. I also get "unable to write to mailbox" message, but I have plenty of file space available.

Actual results:

See above. Thunderbird is repeatedly downloading 1000's of old emails. As is, it is unusable.

Expected results:

Only new emails should have been downloaded.

Comment 1

[Ping Chen [:rnons] (away)]

Thanks for reporting, popstate.dat contains a list of messages TB has fetched. Each message is represented by a line like this

k 000002155f7aa7cb 1648538196

The 000002155f7aa7cb is a unique identifier for that message. If popstate.dat is deleted, TB has to fetch all messages again. So this part is not a bug.

Can you please try the following:

1. start with a fresh profile (or delete all messages from Inbox, close TB, remove popstate.dat)
2. Get Messages to fetch all
3. Get Messages again (after restarting TB)

If messages are duplicated now, then it's a bug.

Comment 2

[Reg Cardin]

I have already done as you suggested. Shut down TB, deleted .dat, restarted and TB downloaded messages. The problem has not gone away. It keeps downloading the same messages. I also get a "unable to write to mailbox" message from which I cannot recover. I have to end the TB task via task manager.

Comment 3

[Reg Cardin]

Given the above, I decided on a more radical approach. I had a backup .dat file from 3/21 and copied it over to my current setup. Started TB which then proceeded to download only messages new as of 3/22, a very small number. So, I don't know what the problem is but I have a workable solution going forward.

Comment 4

[Ping Chen [:rnons] (away)]

1. start with a fresh profile (or delete all messages from Inbox, close TB, remove popstate.dat)
2. Get Messages to fetch all
3. Get Messages again (after restarting TB)

If you still want to investigate, after step 2, you can have a look at popstate.dat, and compare it with your backup. If if contains all the rows as in your backup, I don't see why TB would fetch those messages again in step3 .

Comment 5

[Wayne Mery (:wsmwk)]

We also just had a user, also cox, in support complain about this very thing. But I don't think they were using beta

Comment 6

[Anje]

remove popstate.da

A new popstate.dat will get created, but now the pop account has no idea what has previously been downloaded and so would download everything still in the server Inbox. So that download would be expected. So, after downloading everything, does it do it again? If no then it would seem to be fixed.

I presume you had Account Settings to leave message on server, but have you aso selected 'Until I delete them' ?
After deleting loads of emails did you compacted folders?

Comment 7

[Reg Cardin]

Yes, after downloading all messages (essentially rebuilding the .dat file), I deleted all of the downloaded/duplicate messages. When restarted, TB again started downloading old messages.

Yes, I had set TB up to leave messages on the server. I also had selected 'Until I delete them'.

I assumed the .dat file and perhaps another control file had been corrupted. So, I restored a ,dat file from a backup taken before the problem arose. Then, TB behaved as expected, only downloading messages arriving after the date when the backup was created. All good so far.

Comment 8

[Reg Cardin]

Running on TB 101.0b1. Once again, TB downloaded 1300+ emails from the Cox server. Apparently, the popstate.dat file has again been corrupted. This problem, coupled with the painfully slow email delete function, makes TB unusable. Neither the popstate.dat corruption nor the slow delete occurred over several years prior to this beta version. I anyone actively working this issue?

Comment 9

[Ping Chen [:rnons] (away)]

As I said in comment 4

If you still want to investigate, after step 2, you can have a look at popstate.dat, and compare it with your backup.

Can you please upload both popstate.dat? Better with some logs, go to Config Editor, set mailnews.pop3.loglevel to All, open DevTools > Console.

Comment 10

[Ping Chen [:rnons] (away)]

Attachment: Bug 1762327 - Update popstate.dat every 20 messages. r=mkmelin

So that even if an error happens, no need to re-download all messages.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/89b5ee92d67a
Update popstate.dat every 20 messages. r=mkmelin

Comment 12

[Wayne Mery (:wsmwk)]

I have read the review comments but it is not clear to me what the problem is in layman's terms. If the patch chanes the interval at which popstate is updated but does not anything else, there is still a problem - no?

Comment 13

[Magnus Melin [:mkmelin]]

Yes, but now only at max 20 messages will have to be re-downloaded, not potentially 1000s.

Comment 14

[Matt]

My reading of the historical support requests and the mitigation is that this will achieve nothing. In these scenarios, where the entire mail store gets downloaded again, as opposed to the most recent mail is duplicated because of antivirus defects. It is because the popstate.dat data is compromised. Exactly how does writing to file after 20 emails recover from the situation which has obviously already occurred. The data in the popstate.dat file is invalid for this account for some reason?

This is a simplistic approach that does not tackle the obvious problem of the Popstate.dat file content no longer being valid. What is needed is a mechanism to determine why the data is not valid. Is the file corrupt? Or missing? Or duplicated? Or have all the ids changed (some server upgrades do just that)

In recent times there has become a cottage industry in placing Thunderbird profiles into cloud drives using various sync tools with the inevitable data loss that the latency implies. Thunderbird needs to become far more proactive in determining the suitability of the storage location and less fault tolerant to files being unavailable. I have seen folk with 90 or more prefs.js files. I guess they have issues saving changes. Others have multiple popstate.dat files. Clearly the data Thunderbird is using is invalid, but it bumbles along and redownloads what it can.

Basically, I don't think this bug is fixed, or really even mitigated. We have just slowed the download with more disk writes along the way.

Comment 15

[Anje]

Restricting the download is more of a hinderance and would create more issues than it solves.

There are plenty of occasions when there is a need to create a new pop account and they want full download. Recent case, computer was fixed by techie and needed new hardrive. Lost all data and had no backup. Install Thunderbird and create pop account again and allow full download. They would not want to keep clicking on 'Get Messages' to download the thousands of emails.

Another case, server has all emails, creating a new pop but only wants to download a years worth of emails, advise is logon to webmail, create a folder and move all older mail from Inbox to new folder. Problem solved as user did not want to lose all older mail nor download them.

Cases of corrupted popstate.dat are likey caused as stated by Matt.
Often, deleting the old popstate.dat and stopping Anti-Virus from scanning files that get opened - making profile exempt from scanning - fixes the problem after the first complete download.

Comment 16

[Anje]

(In reply to Reg Cardin from comment #0)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101 Firefox/98.0

I also get "unable to write to mailbox" message, but I have plenty of file space available.

I recently came across someone in Support Forum with this "unable to write to mailbox" message error.
After investigating - Something very weird had effected the problem folder. When they right clicked on the problem folder and selected 'Properties', the properties window looked like it was a 'saved search' . The icon on the problem folder did Not look like a saved search icon.
Let's pretend problem folder was the 'Sent' :
In the 'profile' folders, the pop mail account had a 'Sent' FOLDER - I'm not talking about a 'Sent.sbd' folder.
The program could not write to that 'Sent' folder.
The Program was expecting to write to a 'Sent' mbox file. Hence error : "unable to write to mailbox" message.
It was necessary to Exit Thunderbird, access 'profile' folder, click on 'Mail' folder, click on pop mail account name folder - delete the 'Sent' FOLDER and the 'Sent.msf' file.

We already have many cases of incorrect Folder Flags (both Inbox and Drafts flags) being set on eg: Inbox - unknown cause, but plenty of reports in recent release updates.
So, please check the folder 'Properties' to see if you have a folder that appears to have been set to work like a saved search, it would be useful to know about it, so we can monitor the cases.

Comment 17

[Dan Pernokis]

See also: Bug 1778037 which I filed last night.
In my case, only some messages are downloaded -- unless perhaps because I've quit & restarted TB and the GET MESSAGES process starts over. (It also seems now that a mailbox will be polled for a long time and then only a few new messages -- if there are any -- come down.)

Comment 18

[Dan Pernokis]

Further to my Comment 17...
My mailserver accounts don't contain thousands of emails to come down repeatedly. One of my TB's deliberately never deletes anything. The other intentionally limits life expectancy of messages in most accounts to a reasonable 30-, 60-, 90-day timeframe, so any given rogue poll can only see a few weeks of messages and not forever.

Comment 19

[Wayne Mery (:wsmwk)]

If you still see a new issue, please file a new bug report