Closed Bug 1519045 Opened 5 years ago Closed 4 years ago

35 copies of each mail file

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: marcin, Unassigned)

References

(Blocks 1 open bug,
URL
)

Details

I use Thunderbird with few accounts. One runs Dovecot, two are on GMail. Half of my desktop SSD is used by ~/.thunderbird/ folder nowadays.

Decided to take a look why.

Went to biggest folder (1.8GB in 57900 files) and then checked how much Thunderbird thinks it takes - 1801 messages, 48.9MB in total. Each message is stored over THIRTY times. Maildir format, GMail account.

Went to other folder. 823MB in 77345 files on disk. Thunderbird says 3107 messages, 30.7MB in total. Maildir format again, same GMail account.

Ok, let's check Dovecot one. Turns out that this one is mbox based. Biggest one was 2.1GB on disk, 526MB according to Thunderbird.

For each of them I went with "repair folder" button which is basically "drop whatever is on disk and fetch again from server".

So today instead of working I will go folder by folder to 'repair' them to regain my storage space. And then add to the calendar "go and repair all Thunderbird folders" monthly ;(

Why was this ticket filed? See https://support.mozilla.org/en-US/kb/compacting-folders

Flags: needinfo?(marcin)

What makes you think that TB stores "35 copies of each mail file"? Please provide clear steps that allow someone else to see that, including account types (IMAP? POP?) and what "other folder" means.

Support page says that compacting works only on mbox folders. Have one such account but it is less problematic one.

What makes me think? Nothing else than TB uses it's directories. And I had (before "repair folder" was used) nearly every mail stored in over 30 copies there.

Accounta are Gmail IMAP. First folder was /ML/internal/project1, other was /ML/external/project2 (names are irreverent).

3rd account (dovecot one) is IMAP in mbox files. Here compacting may help - will check.

Flags: needinfo?(marcin)
17:56 (0s) hrw@puchatek:~$ cd .thunderbird/v32n84sg.default/ImapMail/gmail-mjuszkiewicz-REMOVED.com/ABC/cur/
17:56 (0s) hrw@puchatek:cur$ grep "^Message-ID: <54E31077.6050805@" *
1460987170367580:Message-ID: <54E31077.6050805@REMOVED.com>
1462007329283673:Message-ID: <54E31077.6050805@REMOVED.com>
1462091559156522:Message-ID: <54E31077.6050805@REMOVED.com>
1462193152989923:Message-ID: <54E31077.6050805@REMOVED.com>
1462322396048762:Message-ID: <54E31077.6050805@REMOVED.com>
1463115051195954:Message-ID: <54E31077.6050805@REMOVED.com>
1463649687278885:Message-ID: <54E31077.6050805@REMOVED.com>
1464385366985462:Message-ID: <54E31077.6050805@REMOVED.com>
1464607071153074:Message-ID: <54E31077.6050805@REMOVED.com>
1464857889249939:Message-ID: <54E31077.6050805@REMOVED.com>
1466757141634681:Message-ID: <54E31077.6050805@REMOVED.com>
1467153528443500:Message-ID: <54E31077.6050805@REMOVED.com>
1467456717206308:Message-ID: <54E31077.6050805@REMOVED.com>
1467895266042753:Message-ID: <54E31077.6050805@REMOVED.com>
1468338015485496:Message-ID: <54E31077.6050805@REMOVED.com>
1468601541680437:Message-ID: <54E31077.6050805@REMOVED.com>
1470655659850734:Message-ID: <54E31077.6050805@REMOVED.com>
1470990926434960:Message-ID: <54E31077.6050805@REMOVED.com>
1472747309180978:Message-ID: <54E31077.6050805@REMOVED.com>
1472813110659032:Message-ID: <54E31077.6050805@REMOVED.com>
1473671082261956:Message-ID: <54E31077.6050805@REMOVED.com>
1473931690441731:Message-ID: <54E31077.6050805@REMOVED.com>
1475691921922120:Message-ID: <54E31077.6050805@REMOVED.com>
1476957440887479:Message-ID: <54E31077.6050805@REMOVED.com>
1477862768198305:Message-ID: <54E31077.6050805@REMOVED.com>
1478083181612927:Message-ID: <54E31077.6050805@REMOVED.com>
1478561456626551:Message-ID: <54E31077.6050805@REMOVED.com>
1479166611564918:Message-ID: <54E31077.6050805@REMOVED.com>
1502885670898650:Message-ID: <54E31077.6050805@REMOVED.com>
1505666741143084:Message-ID: <54E31077.6050805@REMOVED.com>
1508312190157241:Message-ID: <54E31077.6050805@REMOVED.com>
1542660268742831:Message-ID: <54E31077.6050805@REMOVED.com>
1547118240890492:Message-ID: <54E31077.6050805@REMOVED.com>
17:56 (1s) hrw@puchatek:cur$ grep "^Message-ID: <54E31077.6050805@" *|wc -l
33
17:56 (0s) hrw@puchatek:cur$

Compacting changed nothing. Decided to not use 'repair folder' option to have one wrong folder.

Of course TB stores multiple copies of superseded messages. If you write an e-mail and take one hour to compose it, with an auto-save interval of one minute you'll get 60 superseded drafts.

Gmail is another bad example, we don't notice that messages are fetched and stored multiple times, since Gmail that that "All Messages" folder.

Also, when moving or deleting message, some superseded messages might be left behind in the original location, but that shouldn't happen for maildir.

All that said, we should see whether we can reproduce the "multiple messages" issue. One of the two Bens ;-)

Blocks: maildirblockers
Flags: needinfo?(benjamin)
Flags: needinfo?(benc)

Those mails above were copies of one mail from mailing list.

In meantime I removed ImapMail subfolders from local storage and forced Thunderbird to fetch all mails again. Now it takes just 16GB instead of 120GB.

But such action should not be required imho.

Type: enhancement → defect

Hmmm. An interesting one. It's possible it's already been fixed by Bug 1259040.
The grep results Marcin posted look like the old maildir system where timestamps were used to uniquely name the individual files. They are now named using a filename derived from the msgid, and I think there were a bunch of other maildir changes around that time too.
That said, I can't think of any reason why the old code would have been duplicating emails like that, or any changes I made which would fix it.

The timestamp names on Marcin's files are widely spread over a couple of years, so it doesn't look like autosaved drafts. The dupes are less than once per day, which implies that the replication is a gradual ongoing thing...

Has anyone else managed to reproduce this one?

Flags: needinfo?(benc)

I haven't been able to reproduce the error; However I don't use maildir so if the issue is caused by a bug there I don't believe I will be able to trigger it currently. If needs be, I can make a new profile using maildir for one or more accounts, but from what I gather triggering the bug might not be obvious.

Marcin,

What version of TB are you using? The latest beta might keep you from having to recover lost space and might be the next logical step.
Here is the link to the download page.
https://thunderbird.net/en-US/channel/

Sense you have more knowledge on what the bug is doing to your file system you're the best person to tell us if the bug is fixed.

Flags: needinfo?(benjamin) → needinfo?(marcin)

Ben, you'd have to setup something with maildir in order to reproduce this.

Marcin's account is disabled, so we're on our own.

Flags: needinfo?(marcin)

Now I have TB 68.3.1 version in use. Do not remember which one was used when I reported bug. But as I use Fedora it was probably latest available or 1-2 versions older than latest.

got my account unlocked.

And whole ~/.thunderbird is now around 17GB so no problem to report as I may have such amount of emails in configured accounts.

Thanks for the update

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WORKSFORME

(In reply to Marcin Juszkiewicz from comment #13)

And whole ~/.thunderbird is now around 17GB so no problem to report as I may have such amount of emails in configured accounts.

Marcin, another issue since version 8, recently fixed and eventually coming to 78.8.0, is Bug 1681487 - IMAP folder size keeps growing because single message is downloaded and appended repeatedly downloading

You need to log in before you can comment on or make changes to this bug.