Corrupted MBOX file for GMAIL
Categories
(Thunderbird :: Folder and Message Lists, defect)
Tracking
(Not tracked)
People
(Reporter: harald.norvik, Unassigned)
References
Details
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36
Steps to reproduce:
Several times over the last couple of years, I have noticed that the list of messages in GMAIL Inbox sometimes refer to a corrupted messages.
Example: Quick search (or Find - same result) gives me a list of messages matching a certain subject. Clicking on a single message to display the content shows nothing or sometimes partial HTML code.
Actual results:
Getting the message source (CTRL-U) shows that the message doesn't have to correct header. Checking the email folder (Imapmail\imap.gmail.com) shows multiple nstmp files and a INBOX file that is smaller than expected.
Searching the INBOX (mbox) file shows no trace of the messages, but several nstmp folders contain them.
Expected results:
I suspect this is related to the compact process in Thunderbird (was set at compact when > 200MB). My Gmail inbox has 13000 messages, and the compact process may be interrupted due to arrival of new messages (check for new msg 10 minutes & Immediate server notification - push?).
Turned off compacting for now, and repairing the Inbox folder (Gmail downloads all messages again).
Although some wiki articles says to compact when offline, the compact option is grayed out when offline.
|
||
Comment 1•1 month ago
|
||
|
||
Comment 2•20 days ago
|
||
Harold reports this has been happening for several years. So there might be more than bug 1890230 happening here. This would seem to be confirmed by bug 1890230 comment 117.
I am reopening this bug report, and I suggest you post your future findings here.
Some possibly related issues https://mzl.la/3BTjwtQ
I appreciate that you reopened this Wayne as the other bug had too many comments so the noise level was rather high.
I have re-enabled the Expunge of Inbox at exit, and I notice that I get the extra INBOX-<n> files together with the INBOX.msf.compact-backup.
To be able to check a previous version of the INBOX file, I am doing nightly backup of the folder with a retention time of 4 weeks.
By comparing the extra INBOX-<n> files I am not able to find a corrupt message yet, but I will keep looking. If my suspicion is correct, this is related to the compact process that doesn't finish. To replicate it, I need to delete a few message from the inbox and then shutdown / restart TB.
I am also considering writing some script that will scan the INBOX.msf and check if all 9C (msgOffset) tags refers to a valid message.
(this format looks like some library book indexing system I briefly worked on in the 80s. )
Sorry that I don't have any other news to provide to you yet.
H.
I discovered again today that I had corruption in the message list when opening certain messages that were about 2-3 weeks old. It's not consistent - not all the messages in a specific time interval are corrupt.
My small python program to verify the 9C message offset in the .msf file turned out to be a waste of time. The 9C tag seems to no longer be used in the msf, so the message list must be created from some other file. I suspect this is the source of the corruption where the message index refers to the wrong position in the MBOX file.
I have daily backups of the imap.gmail.com folder from my profile going back until last folder repair (and download of 13K messags).
Just now I created an offline profile and imported the lasted backup folder with Import /Export tools NG, and this confirms that my MBOX file is good. (this add-in is probably rebuilding the index).
My suspicion that this is related to the Expunge (Compact) on exit is stronger as the Thunderbird install on my laptop shows the messages correctly (Expunge is off here).
If you can point me in the direction of the source of the message list to message content, I can further troubleshoot what's breaking. There seems to be several sqlite files involved.
TB Version 128.3.3
Updated to TB 128.4.2 and currently repairing the Gmail inbox. Turned off Expunge for now.
|
||
Comment 6•12 hours ago
|
||
My small python program to verify the 9C message offset in the .msf file turned out to be a waste of time. The 9C tag seems to no longer be used in the msf, so the message list must be created from some other file. I suspect this is the source of the corruption where the message index refers to the wrong position in the MBOX file.
I don't know the hex number but the message offset is now pointed to by "storeToken". You can dump the message records of the .msf file using this program (mork.pl) referenced here: bug 1911477 comment 23.
Updated to TB 128.4.2 and currently repairing the Gmail inbox. Turned off Expunge for now.
As also extensively discussed in bug 1911477, the current repair functionality actually leaves in the "messageOffset" attribute, so you might see both "messageOffset" and "storeToken" after a repair. It is uncertain if this causes a problem having both (probably not if same value). But you will get a cleaner "repair" (with no messageOffset attributes) if you just delete both the mbox file and the corresponding .msf file and let TB just rebuild them both while re-downloading the folder from the server. (You might want to make a backup of both files before deleting them just in case.)
|
||
Comment 7•10 hours ago
|
||
(In reply to gene smith from comment #6)
My small python program to verify the 9C message offset in the .msf file turned out to be a waste of time. The 9C tag seems to no longer be used in the msf, so the message list must be created from some other file. I suspect this is the source of the corruption where the message index refers to the wrong position in the MBOX file.
Mork tokenises both data and field names, and I could easily imagine that 9C value could change.
Mork will just append new changes to the end of the file (the reading will ignore the earlier values in favour of the new ones), and every so often there'll be a full rewrite/garbage collection, and at that point all the tokenisation is likely to be thrown out and done again from scratch.
I don't know the hex number but the message offset is now pointed to by "storeToken". You can dump the message records of the .msf file using this program (mork.pl) referenced here: bug 1911477 comment 23.
https://www.ggbs.de/extensions/mork.pl
As also extensively discussed in bug 1911477, the current repair functionality actually leaves in the "messageOffset" attribute, so you might see both "messageOffset" and "storeToken" after a repair. It is uncertain if this causes a problem having both (probably not if same value). But you will get a cleaner "repair" (with no messageOffset attributes) if you just delete both the mbox file and the corresponding .msf file and let TB just rebuild them both while re-downloading the folder from the server. (You might want to make a backup of both files before deleting them just in case.)
Yes, the change to remove the .messageOffset attribute from the XPCOM nsMsgDBHdr didn't do a migration phase to remove values from the database, but I don't think it's too big a deal. Old DBs will have that value lingering for a while, but nothing will be accessing it - it's just unused (and to be honest, it's probably safer not fiddling with the DBs to strip it out unless there's some specific problem to solve).
Description
•