1879897 - Profile has grown - compact grows it

Status: RESOLVED DUPLICATE of bug 1878541

(Thunderbird :: General, defect)

Description

[Ryan Lee Sipes]

On Daily, version 124.0a1 (2024-02-11) Thunderbird tried to compact my "All Mail" folder for a Google account and surfaced an error that it couldn't because there wasn't enough space on the disk. Looking into it I found that that profile was taking up 274 GB of space (when I first saw it, just an hour ago, it was taking up 264GB of space). The profile seems to grow immensely with each action taken, the culprit file living in ImapMail/imap.gmail.com/[Gmail].sbd/All Mail - which alone is 102GB. There are other large files in there such as nstmp-5 (17GB) - but nothing as large as that All Mail file.

There is a lot of mail in this account, but I am very skeptical that it is this much, and compacting seems to increase the disk space taken up. One compact action grew that profile 5GB.

This is pretty bad, rendering both Thunderbird and my computer quickly unusable.

Comment 1

[Alessandro Castellani [:aleca]]

Tracking things in bug 1878541, probably the same underlying issue.
We'll tackle it right away as a top priority.

Comment 2

[Magnus Melin [:mkmelin]]

Are you set to use auto-compact, or does it ask before compacting?

Comment 3

[Ben Campbell]

Has anyone come up with some solid steps to replicate this one? That would help a lot.
It seems like there are multiple issues overlaid here:

1. the ever-increasing folder size
2. compact not coping well (and also not cleaning up properly). The code path for folder compaction is significantly different than for local folders (sigh...of course it is...)

Regarding 1:
Does it happen on any folders other than IMAP folders?
If you look at an affected mbox file, are there lots of duplicate messages? Do those messages look intact, or are do they look corrupted/interrupted by other messages? There's definitely an issue with some protocols attempting simultaneous mbox writes. There's a kind-of fix in Bug 1857450, but it'll cause choking. I'm working on a better fix to prevent simultaneous mbox writes more gracefully, but it requires some more... architectural change (which I'd intended anyway - this just moves it up the priority list :- ).

Comment 4

[Andrei Hajdukewycz [:sancus]]

It actually hasn't happened to me even once since I disabled auto compact, so I'm starting to wonder if that's the culprit. I don't have specific steps to reproduce. I will turn auto compact back on and see what happens. You can thank me for the 3TB of SSD write endurance it will cost later ;)

Comment 6

[Eugene Savitsky]

It should be considered as a next beta blocker.

I have just deleted 1.2Tb of those nstemp files...

Comment 7

[Wayne Mery (:wsmwk)]

I discovered a 41 gb folder file that Thunderbird claims should be 144mb, about 10k messages most of which are bugmail.

The affected folder only accepts message using the Archive function, no other methods - not filters or drag+drop.

Further, it's an archive folder that I never directly open, I never move messages out of it, never delete messages from it, only a rare "undo" of archived a message would delete a message, compact threshold has been 500mb with compact prompte enabled => there will have been very few folder compacts in recent months (I'd estimate 5-10).

Comment 8

[Wayne Mery (:wsmwk)]

I found no other abnormally large files, nor large nstmp files.

Comment 9

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello all,
We have tried to reproduce this issue using macOS 14, macOS 13 as well as Windows 11 and 10 with builds such as 124.0, 115.9.0 and 122 (beta 1 through 3) without any success.

We used the following configuration of email providers 2 Gmail accounts-(one personal/one work), Protonmail, Yahoo .

In order to try and trigger the issue we did the following procedures:

• Tried to configure the accounts via IMAP or POP3 (or a combination of both for different accounts)
• Use the manual Compact option while the account was still synching and emails were being downloaded
• Reduce the size for auto-compact space saved to 1MB
• Disabled the "Ask every time before compacting" option
• Manually selecting the "Compact" option from the context menu for each folder (especially All mail)
• Using the "Repair" option from Folder properties following a manual compact
• Sending and receiving multiple rich-text emails using unicode characters, different filetype attachments of various sizes and hyperlinks
• Manually comparing folder sizes displayed in Thunderbird vs what is displayed in ~/Library/Thunderbird/Profiles/
• Restoring a profile backup (or restoring it to a different location) and then manually compacting folders(especially All mail folder) using the steps provided in SUMO.
• All devices with Thunderbird installed and configured went through different states throughout a span of 2 weeks (Sleep, Always on, Shut down-Powered On, Idle with screen on/off).

Comment 10

[Wayne Mery (:wsmwk)]

Consolidating to bug 1878541.
We can reopen if it becomes necessary.