Choosing "repair folder" often deletes the main mbox file
Categories
(Thunderbird :: Untriaged, defect)
Tracking
(Not tracked)
People
(Reporter: luke.kendall, Unassigned)
Details
User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36
Steps to reproduce:
I right clicked on a local mail folder (underlying file on local system was in mbox format), and opened Properties.
I clicked on Repair folder
Actual results:
All message summaries vanished
The mbox file containing all the messages (about 1.5Gb size) was gone
I copied the mbox file back from backups, otherwise it would have been disastrous.
Expected results:
The mbox file should not have been deleted, and the index files should have been deleted and repaired
(My god it was hard to find the user agent string!)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.24) Gecko/20101027 Thunderbird/2.0.0.24 Mnenhy/0.7.6.666
I expected it to be in the About panel. But that says:
128.14.0esr (64-bit)
Thunderbird for Linux Mintmint-001 - 1.0
This bug happens perhaps one time in three when I click Repair Folder.
I have learned from bitter experience to backup the mail folder (mbox format) beforehand.
|
Reporter | |
Comment 1•5 months ago
|
||
The bug occurs to mail folders regardless of size - big or small.
I haven't been able to find a pattern to reproduce it without fail.
But I have no idea why TB would have any code path that would include deleting the actual mail data file, except for what must be provided via the right-click on folder, "Delete" command.
I think this is an extremely serious bug.
|
||
Comment 2•5 months ago
|
||
Can you reproduce in Help | Troubleshoot mode.
Must be something special with your setup, as I think this is the first report I ever saw of this.
In all cases on Support Forum that I have seen issues like this, the index did not match what was actually in the mbox file.
The emails had already been deleted from mbox file, they were long gone, but the index was not in synch with mbox file. User could not select emails and see them. They Repaired folder and it looked like it has deleted the emails. So when all emails apparently disappeared from Message List they blamed the 'Repair', but in reality the Repair worked. It was no longer displaying stuff that did not exist.
Potential reasons for emails already being deleted:
In Support Forum, in most cases the user was allowing an Anti-Virus to scan profile folders and the AV was allowed to auto fix if it found something it did not like. If that occured then the entire mbox file is likely to be deleted as AV does not realise the mbox file contains more than one email. AV does not touch the index file and so it will appear out of synch. Advise - do not allow AV to scan profile folders.
In some cases, there was a 'Retention Policy' set on a folder (right click on folder and select 'Properties' then select 'Retention Policy' tab).
It was set to delete mail older than X days, so folder looked empty and a 'Repair' did not reveal emails, but the mbox file was still intact in the profile and a repair of the file altering all 'X-Mozilla-Status:' to 0001, save file and delete the equivalent *.msf file recovered all emails when Thunderbird was restarted. However, if a mass deletion had occured and the folder was subsequently compacted - everything would be lost in mbox file, but index would have updated.
In other cases, the user was accessing the same mail account on other devices eg: phone or another computer. They had deleted emails off phone and not realised it had deleted emails off server and so the imap account in Thunderbird then deleted the emails to match the server. Another had set up a Pop account to not leave emails on server - same result. In Thunderbird Imap account it suddenly appeared to delete emails. One user did not realise they had Outlook still operating in the background and it was causing the deletion of mail off server. So there are external factors that can effect what is seen in Thunderbird.
You say:
I right clicked on a local mail folder (underlying file on local system was in mbox format), and opened Properties. I clicked on Repair folder
Before you clicked on Repair Folder button - did you access the profile and check the mbox file?
OR did you check it after the Repair when things did not appear as expected ?
Why did you feel a need to perform a Repair Folder? Did you do it because no emails were displaying ?
Is the effected folder in 'Local Folders' account OR a Pop account or an IMAP account ?
The mbox file may not be seen in Profile mail account, if you have the IMAP 'Synchronise & Storage' - 'Advanced' button settings to not download full copies. Only headers are downloaded, so only index .msf files would be seen.
All of these types of questions and set up queries etc would be checked if you had asked a question in the Thunderbird Support Forum to see if you had something causing a problem. Have you asked a question on this issue in the Support Forum? If yes, then post a link.
|
|
Reporter | |
Comment 4•5 months ago
|
||
"The emails had already been deleted from mbox file, they were long gone, but the index was not in synch with mbox file. User could not select emails and see them."
That isn't the case. I'm running exim and dovecot and the imap server is running locally. I'm accessing the files from Thunderbird from two computers on the LAN. It happens so frequently that I am now in the habit of copying the mbox file before clicking Repair. And when TB deletes the file (instantly) I copy the backup back and try Repair again. And again...
"In Support Forum, in most cases the user was allowing an Anti-Virus to scan profile folders"
I'm running on Linux, and I use no anti-virus software.
"In some cases, there was a 'Retention Policy' set on a folder (right click on folder and select 'Properties' then select 'Retention Policy' tab)."
The retention policy is set to "Use server settings", and the server settings (for localhost, that is), is "Synchronize all messages locally regardless of age" and "Don't delete any messages"
"Before you clicked on Repair Folder button - did you access the profile and check the mbox file?"
Yes.
The mailbox files are stored in a mail directory under my home directory, not inside the profile. (This is for historical reasons: I used to use other mail programs until about 15 years ago, when I switched exclusively to TB).
I also checked afterwards, and the mbox file is just gone. The first few times I'd copy the files back from my NAS nightly backup, so only lost any emails I'd filed since that backup. Nowadays I explicitly copy the mbox file before daring to try Repair.
I only do a Repair when I can't open a message, or if trying to open a message (which is local) takes more than a minute or two. (Normally of course it takes just milliseconds.)
"Is the effected folder in 'Local Folders' account OR a Pop account or an IMAP account ?"
I can reference the affected file (I'm using mbox files, not maildir directories) both via a local IMAP server and via 'Local folders'.
"The mbox file may not be seen in Profile mail account, if you have the IMAP 'Synchronise & Storage' - 'Advanced' button settings to not download full copies. Only headers are downloaded, so only index .msf files would be seen."
Hmm, interesting. For the IMAP server in TB under 'Synchronization & Storage' I do NOT have Message Synchronizing checked, and under the Advanced button none of the folders are checked.
BUT under localhost I DO have Message Synchronizing checked, and all the mbox files checked for offline use (Download).
Do you think that mismatch could be leading to this problem?
"All of these types of questions and set up queries etc would be checked if you had asked a question in the Thunderbird Support Forum to see if you had something causing a problem. Have you asked a question on this issue in the Support Forum? If yes, then post a link."
Sorry, no, I had not thought to ask the question on the forum since it seemed to me that removing a massive user data file in a repair operation felt like a bug - something that should never be allowed to happen except by deliberate user action, with an "Are you sure you want to remove this?" check.
I confirm this error is present same as the OP reported.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:144.0) Gecko/20100101 Thunderbird/144.0.1
Every time I go to repair a folder, the program deletes both the mbox file & the msf file.
Then it proceeds to re-download and recreate both. Confirmed by both "created on date", and if you turn off Internet access, the mbox file just disappears, while msf sits at 0Kb waiting for the connection.
This is fairly major, as some mailboxes can be many Gb in size and re-downloading them is highly undesirable!
Yes, I can reproduce this in Troubleshoot mode - same thing happens.
|
|
Reporter | |
Comment 7•5 months ago
|
||
It's worse than Michael (above) notes, since if you have moved some of the messages into local mailboxes and later deleted the messages from the original server, that local mailbox cannot be recreated via re-downloading. If you don't have a backup of the mbox file, TB has deleted your information forever.
My mail files go back as far as at least 1994. Many of the messages are now inside mbox files from servers that no longer exist, but I can still sensibly (i.e. meaningfully) move some new messages into some those folders.
(I have 26GB of mail spread across a hierarchy of about 350 mbox files.)
I've posted this on the support forum:
This report is confusing:
According to comment 0, this is about local folders, not IMAP folders.
Repairing for local folders leaves the raw data untouched, and it rebuilds the MSF from the raw data.
Repairing and IMAP folder is only available when online: The raw message data is deleted, it is then downloaded again, and in the process a new MSF is built.
The issue seems to be a misconfiguration where the very same folder is at the same time IMAP and local. That will completely break the system:
See comment 4, quote:
I can reference the affected file (I'm using mbox files, not maildir directories) both via a local IMAP server and via 'Local folders'.
|
||
Comment 10•5 months ago
|
||
Hi,
Thank you for the input.
Yes, I can confirm that if I try to repair a local folder, the mbox file doesn't disappear.
Going by your explanation, my install is performing correctly then.
However, I find it a strange arrangement whereby the same function performs completely diffferent for two different situations.
Not to mention that I fail to see why it needs to do that.
Imagine you have a 10-20Gb mailbox and you have to repair it for some reason - it's a huge download that takes hours in best case!
Why can't both IMAP & local folders be done the same - just a rebuilding of the index?
And a followup question - in this case, what would the result of deleting the .msf file manually and then restarting TB? Will it rebuild it from the leftover mbox?
Thanks.
|
||
Comment 11•5 months ago
|
||
Yes, repairing a 10+ GB IMAP mailbox isn't fun. IMAP is different from local folders since when the message database (MSF) is (supposedly) broken, the only true way to repair it is to synchronise with the server. Of course, this could be improved, like: Rebuild the MSF from the raw data of the IMAP folder, then try to "fix" any differences.
Deleting MSF files is not recommended, but in most cases it will just trigger a repair, so also a re-download for IMAP.
|
|
||
Comment 12•5 months ago
|
||
Interesting. Thank you for the insight.
I would argue the second option (rebuild from the raw data and then sort out the differences) would be a much preferred choice!
In any case, perhaps, there should be a warning to users that this same button does completely different things to different folders.
I don't believe I've seen it mentioned anywhere.
Thanks again.
|
||
Comment 13•4 months ago
|
||
Thunderbird is functioning as designed. So I will close this bug as invalid.
Description
•