1935124 - Error when compacting EMPTY folder. "Folder xxxxx could not be compacted because writing to folder failed. Verify that you have enough disk space, and that you have write privileges to the file system, then try again."

Status: RESOLVED FIXED

(MailNews Core :: General, defect, P2)

Description

[Kai Engert [:KaiE:]]

Have an IMAP folder with zero messages.
The folder should have already been compacted, check the profile directory on disk, ImapMail subfolder, the corresponding file should have a size of zero bytes.

In the UI, ask to compact the folder.
We'll get a popup message reporting the folder ...
"could not be compacted because writing to folder failed. Verify that you have enough disk space, and that you have write privileges to the file system, then try again."

Reproduced with 128.5.1

Comment 1

[Kai Engert [:KaiE:]]

Attachment: Bug 1935124 - Change MboxParser to allow empty files without a failure. r=BenC

Comment 2

[Kai Engert [:KaiE:]]

I've attached a naive fix. I don't know if that approach can have side effects and break other expectations of the code.

Ben or Geoff, could you please check what's the right thing to do?

I'll start a try build.

Comment 3

[Kai Engert [:KaiE:]]

https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=fc7e23fcc369d2809175ccdfe41c468816fe8735

Comment 4

[Kai Engert [:KaiE:]]

Alessandro, you mentioned that you also saw this error message when trying to compact a folder. In your case, was that folder empty or non-empty? If it was non-empty, it would be interesting to be able to look at your file.

Comment 5

[Kai Engert [:KaiE:]]

FTR, Ben had mentioned the error is shown here:
https://searchfox.org/comm-central/source/mailnews/base/src/FolderCompactor.cpp#729

Comment 6

[Alessandro Castellani [:aleca]]

Yes, I experienced this error for a specific folder.
The folder is not empty but it only contains one message.
It's not sensitive data so I'm okay with sharing it with you.
What do you need?

Comment 7

[Kai Engert [:KaiE:]]

Could you please zip the local file, both mbox and msf, and email it to me?
Thanks

Comment 8

[Kai Engert [:KaiE:]]

Thanks Alessandro. Because your file isn't empty, I'll comment on your issue in bug 1935331.

Comment 9

[Ben Campbell]

(In reply to Kai Engert [:KaiE:] from comment #0)

Have an IMAP folder with zero messages.
The folder should have already been compacted, check the profile directory on disk, ImapMail subfolder, the corresponding file should have a size of zero bytes.

In the UI, ask to compact the folder.
We'll get a popup message reporting the folder ...
"could not be compacted because writing to folder failed. Verify that you have enough disk space, and that you have write privileges to the file system, then try again."

Reproduced with 128.5.1

I can't reproduce with these steps... I selected all the messages in my Inbox folder, then compacted it.
So I've got a 0-byte INBOX, but right-clicking and choosing "compact" again doesn't bring the popup message (it just displays "compact complete (approx 0 bytes saved)" in the status bar.
I think the folder has a zero expunge count, so the compaction is skipped before any of the core compact code runs.

Am I missing a step?

(BTW, there is already an (icky) special-case check for empty mbox files:
https://searchfox.org/comm-central/source/mailnews/local/src/nsMsgBrkMBoxStore.cpp#128
so if I can reproduce this then I'd want to know why that code no longer works)

Comment 10

[Ben Campbell]

Oop - I only read first half of your patch before writing comment 9. I see you saw the existing empty-mbox check. But my point still stands - why doesn't my old empty-mbox check work now?

Comment 11

[Kai Engert [:KaiE:]]

I can no longer reproduce the issue with a plain empty IMAP mailbox. Sigh.

Comment 12

[Patrick H. Lauke]

Can still reproduce on latest 134.0b4 (64-bit) / Windows

Comment 13

[Kai Engert [:KaiE:]]

I wonder if this is a special case of bug 1935331.

I remember that when I originally reproduced it, I always used the same folder for reproducing.
I might not have tried to reproduce using a completely fresh folder.

Comment 14

[Kai Engert [:KaiE:]]

I can reproduce it, by creating a fresh Imap folder, copying one message to it, quit edit mbox file, edit an additional line at the beginning, then start TB, wait for TB to repair the message, then delete the message, then try to compact.
Then quit TB, then delete the mailbox file from disk.
Then restart TB, which will then recreate the mbox file with a zero size.
Compact, and it still shows the error.

Conclusion: Having a bad .msf file is also sufficient to get this error message shown.

Comment 15

[Kai Engert [:KaiE:]]

Attachment: makebad.zip

This zip file contains a zero size mailbox file and a msf file.

Extract, and copy both files to the "Mail/Local Folders" directory of your profile.

Then start TB and right click compact.

You should get the error message.

Comment 16

[Corey Bryant]

This is also easy to reproduce with the steps described in bug 1936642

Comment 18

[Kai Engert [:KaiE:]]

(In reply to Corey Bryant from comment #16)

This is also easy to reproduce with the steps described in bug 1936642

Those steps don't work for me.
I suspect both you and the reporter of that bug must have similarly corrupted files.

Comment 19

[faith]

It happens to my trash folder. And only of I compact the root of the folders. So right click on IMAP. If I just right click on my trash folder and press compact, then there is no problem.
So it only happens when I conpress all the folders at once.
My trash folder is empty, when I deletecthe MSF file same thing happens. The other Trash file is 0 bytes

Comment 20

[Kai Engert [:KaiE:]]

Can this be reproduced with a freshly configured IMAP account in a fresh Thunderbird profile?
If not, the bug is specific to the data that has accumulated in your files in your existing profile.

Comment 21

[faith]

I do not know but else how do you explain a 0 bytes file that is corrupted?

Comment 22

[faith]

To add to that, if it became corrupted it was due to the latest 127 update, before that I had no problems.
If the file is corrupted why can I compact the folder with no problems, but when I select ALL the folders and press compact, I get this error.

Comment 23

[Kai Engert [:KaiE:]]

I cannot explain it. There are multiple files belonging to your IMAP account. The corruption could be in one of those files.

Comment 24

[faith]

So what do I do now?!
If the corruption is in any of the files where would I start?
It only shows an error with the Trash folder, I can remove it, I can put another one in it, the problem remained.

Comment 25

[faith]

I have multiple mail accounts on this PC, different providers, they all have this same error since the latest update. that would be 6 corrupted profiles after one update, that can't be right?!

Comment 26

[Patrick H. Lauke]

I have multiple mail accounts on this PC, different providers, they all have this same error since the latest update. that would be 6 corrupted profiles after one update, that can't be right?!

same here, i get the same problem on 3 different accounts, 2 on one server, 1 completely separate on another server/host provider altogether.

Comment 27

[faith]

1 of the 6 accounts didn't have this problem. I copied the empty trash can to the other accounts, and thought it was fixed.
Untill I deleted one mail, emptied the trash can, then the bug came back.

Comment 28

[Kai Engert [:KaiE:]]

(In reply to faith from comment #24)

So what do I do now?!
If the corruption is in any of the files where would I start?
It only shows an error with the Trash folder, I can remove it, I can put another one in it, the problem remained.

If your Trash can is empty, you could try the following:
Look at your account settings, server settings, bottom area, there's a setting "Local Directory", which tells you in which directory the mail for that Imap server is cached locally.
Quit TB
Use a file explorer tool to go to that directory.
Detete Trash and Trash.msf files
Start TB, use the Trash folder, and see if the error returns.

(If you still get the error with that account, some other data, separate from that folder, is likely to be responsible.)

Comment 29

[faith]

Thank you, I have tried that for the last couple of days but it doesn't help.
It also doesn't help with my 5 other accounts that have the same problem, isn't it odd that I updated TB, and all the 6 accounts are corrupted?

Comment 30

[Matthew Turnbull [Bluefang]]

Attachment: Another msf file that breaks compacting

Here's another example of a msf file of an empty folder that breaks compacting. This was the shared Trash folder on my Local Folders account.

Attempting to compact the folder when the on-disk size is 0 results in the popup

The folder 'Trash on Sparky' could not be compacted because writing to folder failed. Verify that you have enough disk space, and that you have write privileges to the file system, then try again.

$ export MOZ_LOG="compact:5"
$ thunderbird
[GFX1-]: glxtest: ManageChildProcess failed

[GFX1-]: glxtest: libEGL initialize failed
[GFX1-]: glxtest: X error, error_code=2, request_code=151, minor_code=3
[Parent 29638: Main Thread]: I/compact AsyncCompactFolders() starting compaction of 1 folders
[Parent 29638: Main Thread]: I/compact BeginCompacting() folder='mailbox://nobody@Local%20Folders/Trash'
[Parent 29638: Main Thread]: V/compact OnCompactionBegin()
[Parent 29638: Main Thread]: I/compact OnCompactionComplete(status=0x80550024 oldSize=0 newSize=0)
[Parent 29638: Main Thread]: E/compact Failed to compact folder='mailbox://nobody@Local%20Folders/Trash', status=0x80550024
[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
[Parent 29638: Main Thread]: I/compact AsyncCompactFolders() finished. TotalBytesRecovered=0

(Ignore the glx errors - I'm pretty sure that's from upgrading the nvidia driver modules and I haven't restarted yet)

A couple observations:

• On occasion, when this error is encountered, all folders will appear empty when selected until Thunderbird is restarted. This caused moderate panic the first time it occurred, as I thought my entire profile had been corrupted.
• Using Empty Trash appears to truncate the folder, and a subsequent compact will result in an error.
• Manually deleting the contents of the folder does not immediately truncate the folder, and a subsequent compact succeeds. This appears to fix the folder. This took me a long time to notice, as I almost always do Empty Trash + Compact.
• Running "Repair Folder" doesn't appear to have any effect.
• Exiting Thunderbird and manually deleting the msf file fixes the folder.
• I do not know what first triggered this issue (i.e. if it was a specific version) but it seems to be around the same time I accidentally toggled Thunderbird into Offline Mode.
• None of my other POP/IMAP accounts were affected (as far as I know), just Local Folders -> Trash.

Comment 32

[Richard Leger]

(In reply to Matthew Turnbull [Bluefang] from comment #30)

• Using Empty Trash appears to truncate the folder, and a subsequent compact will result in an error.
• Manually deleting the contents of the folder does not immediately truncate the folder, and a subsequent compact succeeds. This appears to fix the folder. This took me a long time to notice, as I almost always do Empty Trash + Compact.

@faith
Have you tried to manually delete messages in the Trash without using the Empty Trash feature? Matthew seems to suggest that helped fix/prevent the problem at his end...
If not sufficient, can you retry that after deleting Trash and Trash.msf files in your imap local cache directory as suggested by Kai previously?
Does that makes any difference at your end?

@Kai, @Matthew,

[Parent 29638: Main Thread]: I/compact OnCompactionComplete(status=0x80550024 oldSize=0 newSize=0)
[Parent 29638: Main Thread]: E/compact Failed to compact folder='mailbox://nobody@Local%20Folders/Trash', status=0x80550024

status=0x80550024 could be similar to const NS_MSG_ERROR_MBOX_MALFORMED = 0x80550024; as found in mailnews/imap/test/unit/test_downloadOffline.js (line 86) and mailnews/local/test/unit/test_mboxMalformed.js (line 95) according to https://searchfox.org/comm-central/search?q=0x80550024&path=&case=false&regexp=false

What I noticed:

• If you Empty the trash via Thunderbird 128.5.2esr (64-bit), the Trash.msf remain, but the MBox file Trash is entirely deleted from the disk. The absence of the file may be what causes the error above and the user error message "could not be compacted because writing to folder failed. Verify that you have enough disk space, and that you have write privileges to the file system, then try again."

• If you Delete all messages in Trash manually instead, the Trash.msf remain, as well as MBox file Trash. If you Compact manually the Trash folder, the process does not seems to complete and data seems to remain in the cache file MBox file.

• So either the Empty Trash feature shall re-create an empty but initialised Trash MBox file after deleting the existing one

• ...or the Compact feature, when triggered on the folder or on the entire mailbox, shall check if the MBox file exists prior trying to access it. It should not assume wrongly that such file exists for all folders by default... it could initialise it if required.

Also on a side notes, those files (box, msf) seems to never be really empty, while File Explorer may shows 0KB, it does not mean the file is entirely empty it may still contain few bytes of data, as can be seen in text editor.

Comment 33

[paul]

Dear Thunderbird:

I reported bug 1937431 which was marked as a duplicate of this earlier bug so I add my additional info to this bug as it is the original bug for this issue

I don't know if the following will help but I figured I send it along and let you make that decision.

I have my old Centos 7 system which stopped updating awhile back which means I have thunderbird 113.12.1. I tested and my issue (see Wayne's note two before this one concerning burping on "[gmail]Trash" under Rocky 9 128.5.0.esr) does not happen there. To be expected.

What was of interest was my MacBookPro Sonoma 14.7.1 is on thunderbird 28.5.2.esr and the issue does not happen there.

Additionally, after compacting Rocky 128.5.0.esr and then restarting it it immediately says there is 100mb that can be saved by compacting. Both 113.12.1 and 128.5.2.esr have no issues to report. So my issue of failure "[gmail]Trash" may be the location of the burp but there is apparently 100mb of compacting that was prevented by this burp and it looks to be a local file as all three tbirds share the same gmail info.

I hope there is something in this that helps focus where the issue is

Best,
Paul

Comment 35

[paul]

Regarding my comment #33 where I said "my MacBookPro Sonoma 14.7.1 is on thunderbird 128.5.2.esr and the issue does not happen there", it just happened on the MacBook Pro and I have not updated any software. I suppose it is good that it fails on both copies of 128 as it was weird that it worked on one and not on the other.

I double-checked my old Centos 7 with thunderbird 113 and I am still not seeing the error ... which makes sense as it is an older version

Comment 36

[Kai Engert [:KaiE:]]

Still waiting for Ben's review

Comment 37

[Ben Campbell]

Attachment: Bug 1935124 - Add empty-mbox asyncScan test to test_nsIMsgPluggableStore.js. r=#thunderbird-back-end-reviewers

Comment 38

[Ben Campbell]

Just extended the asyncScan() test to cover an empty mbox case.
It fails without Kais patch and passes with it.

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/00a9a43ff6ca
Change MboxParser to allow empty files without a failure. r=BenC
https://hg.mozilla.org/comm-central/rev/245dc2068147
Add empty-mbox asyncScan test to test_nsIMsgPluggableStore.js. r=kaie

Comment 41

[Kai Engert [:KaiE:]]

Comment on attachment 9445814 [details]
Bug 1935124 - Add empty-mbox asyncScan test to test_nsIMsgPluggableStore.js. r=#thunderbird-back-end-reviewers

[Approval Request Comment]
Regression caused by (bug #): related to folder compact work
User impact if declined: unnecessary error message shown to user for valid storage files
Testing completed (on c-c, etc.): yes
Risk to taking this patch (and alternatives if risky): low

Comment 42

[Corey Bryant]

Comment on attachment 9445814 [details]
Bug 1935124 - Add empty-mbox asyncScan test to test_nsIMsgPluggableStore.js. r=#thunderbird-back-end-reviewers

[Triage Comment]
Approved for esr128

Comment 43

[Corey Bryant]

Comment on attachment 9441647 [details]
Bug 1935124 - Change MboxParser to allow empty files without a failure. r=BenC

[Triage Comment]
Approved for esr128

Comment 44

[faith]

(In reply to Richard Leger from comment #32)

@faith
Have you tried to manually delete messages in the Trash without using the Empty Trash feature? Matthew seems to suggest that helped fix/prevent the problem at his end...
If not sufficient, can you retry that after deleting Trash and Trash.msf files in your imap local cache directory as suggested by Kai previously?
Does that makes any difference at your end?

Hi Richard,

Sorry for the long wait, it does indeed help, the message won't show up. It is not the greatest but it helps :-)