319142 - "Compact folder" does not decrease the size of local/pop zero-filled mail database files

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Peter]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5

I have two POP3 accounts in my Thunderbird and they are NOT set to use different folders – everything comes to the global inbox. Nevertheless TB creates different folders for each account in the mail storage database, i.e. the folder C:\Documents and Settings\____\Application Data\Thunderbird\Profiles\________.default\Mail\ has the following folders:

1. Local Folders
2. mail.server1.com
3. mail.server2.com

When you choose "Compact folders" on Local Folders or manually choose "Compact this folder" on any of the folders, TB does compact only the files of "Local Folders". The files in the other two folders keep increasing indefinitely (mail.server.com folders), although the files there are filled with 0s.

I created my account with TB 1.0.5, then updated to TB 1.0.7 and now updated to TB 1.5.rc1. This behavior is always reproducible.

If you manually delete the specific account folders (mail.server.com folders), TB creates them again and starts filling them again with 0s.

Reproducible: Always

Steps to Reproduce:

Comment 1

[jwq]

I don't see this with version 1.5rc2 (20051201) on Mac OS X 10.3.9: "Compact folder" is definitely removing deleted emails from the database files. Perhaps it is specific to your OS or install?

This report looks similar to Bug 280382, but that bug was for version 1.0. Can this be reproduced with the 1.5 release? Does this buggy behaviour also appear in a fresh profile?

Comment 2

[Peter]

I might not have described the error properly. I believe this is 1.0 error, which is not valid for 1.5 branch. But after upgrading to 1.5, keeping the old profile created from 1.0, the empty files stay forever. So if you have sent or received 100 Mb e-mails using 1.0, upgrading to 1.5 (and compacting) does not delete these empty files. Personally, I had discovered by accident that I have half a gigabyte e-mail profile, although my mail was almost empty.

On a global scope, this might be an awful waste of bytes, so it might be useful to implement in the 1.5 branch a clean-up routine, which should take care or those old buggy profiles created with 1.0.

Comment 3

[Jasper Knockaert]

This bug seems still to be around (or to have resurfaced) in Thunderbird 2.0.0.4. Today I found out that the inbox file in the pop3 account folder ("mail1.server1.com" in Comment #1) was again over 0,5 gig. To be clear: the account is set up to use the global inbox (which lives in "Local Folders" rather that the pop3 account folder).

The workaround I apply is to switch off the use of the global inbox, compact the account-specific mailbox and next switch on the global inbox again.

To me it is somewhat unclear where the need for account-specific mailfolders arises when the account is set to use the global inbox. But anyway, there should be a more user friendly way to compact them.

Comment 4

[David :Bienvenu]

How are messages getting into the account-specific  inbox file? Do you have a filter that moves messages there? In theory, messages should never get in there, and you shouldn't be able to delete messages from it, since it's not shown in the UI.

Comment 5

[Jasper Knockaert]

I have no idea as of how and which messages are getting in the account-specific inbox. No custom defined filters on any account/mailbox. Only junk filtering, set to the generic junk folder option. And I never deleted any message from it, as it is not shown in the UI as you remark.

It seems something random: the modification datetime of the inbox file (before compacting) was about two weeks ago, and I did frequent mail checks and received a lot of (junk) messages on that account since then.

Upon switching off the use of the global inbox, these account specific mailboxes became accessible in the UI. When selecting the account specific inbox, it was shown to be empty and I was prompted (correctly) if I would like to compact it. So apparently there's nothing inconsistent: the msf file correctly indicated the inbox to be empty. After compacting, the inbox file size was reduced to O bytes again.

Maybe I should check in my backup archive if I can track the build-up history of the affected file.

Comment 6

[Peter]

The messages are actually not getting to the specific account inboxes - the files are padded with zeroes. Also, the modification dates are very strange. I have 2 different user profiles, and here is the actual data from them.

Profile 1:
                Size    Modification date
Local folders    53M    18.08.2007
Account 1         9M    02.04.2007

Profile 2:
                Size    Modification date
Local folders     7M    18.08.2007
Account 1         6M    20.05.2007
Account 2         1M    26.02.2007
Account 3         0     25.03.2007

I have set TB 2 to autocompact folders automatically.

Comment 7

[Jasper Knockaert]

I verified an archived version of the account specific inbox file (i.e. the version of the mailbox file before having it compacted as described in Comment #3). Because of the size of the file (644 MB) I could not get it to open in a text editor. I tried to open it in Firefox, and all I got was an empty window (without errors). I next zipped it (with BOMArchiveHelper), which boiled down to a 644 KB zip file. That means 0,1 % the size of the original file, a strong indication that there's not much information in the original mailbox file.

As for the modification dates I want to make clear that I was speaking about the "inbox" mailbox file. The other mailbox files (which all have a size of O KB) in the account specific folder have modification timestamps longer ago (up to half a year). As for the msf files, they all have a more recent modification date (different points in time Today), although it is again unclear what triggers them being modified.

Comment 8

[Mikael Willberg]

I had a user today whose IMAP INBOX was empty on server, but Thunderbird 2.0.0.6 (Windows) had INBOX file with size over 12 GB (giga bytes).

Compacting the folder did nothing. Deleting INBOX* files and few test messages later the INBOX was 700 KB. After deleting all messages and compacting the folder still did nothing.

Comment 9

[Wayne Mery (:wsmwk)]

can you reproduce with v3.0 or latest v2.0.0.24 due out next week?

Comment 10

[Peter]

Yes, kind of. Currently I am using TB 3.0.3, and I have upgraded to lots of versions since 1.0.5, including many beta versions.

At the moment, I have a 67MB Inbox file in 'Local Folders' folder (last-modified on 12.03.2010), and a 48MB zero-padded Inbox file which is in the 'mail.server1.com' folder (last-modified on 24.02.2009) - more than 1 year ago.

Unfortunately, I was not able to come with an exact scenario for reproducing this bug - the zero-padded file used to have last-modified date from few days to few months before the current date. Judging by the current situation, I would guess that the bug is not valid for Thunderbird 3.0.

If it turns out that many people have the same problem (i.e. upgraded their profiles for several years and now have some zero-padded leftovers, which would stay there forever), it might be useful to implement a clean-up routine during the upgrade process.

On the other hand, a few hundred megabytes mean much less today compared to 5 years ago, when I reported this bug.

Comment 11

[Wayne Mery (:wsmwk)]

so in your estimation you don't see the problem, either because the problem is gone or you can't reproduce the conditions?

Comment 12

[Peter]

The problem seems to be gone with TB3 from one point of view. From the other - the large empty files are not gone - they just do not increase their size anymore.

I recently moved another account folder (TB2 on Windows Vista, later upgraded to TB3) to a new machine, and the situation was similar: ~4GB size of 'Local folders', ~1G empty files in account folder.

I will check next week some other accounts if they have similar situation.

Comment 13

[Peter]

Just to elaborate a little bit further, as I checked another installation (also TB2 on Windows Vista, later upgraded to TB3).

Current situation:
Local Folders/Inbox: 3.4G
mail.somedomain.com/Inbox: 3G

The second file is padded with 0s, as it was on all the other accounts. The file modification date is before the account was upgraded to TB3, so this is the reason to believe that the bug is not present in TB3, i.e. once upgraded, the empty file stops increasing its size. But nevertheless, it would be nice if TB3 could clean-up those empty files when upgraded.

What is common between all accounts that I have checked is that all of them usually send/receive messages which are encoded with non-latin encodings - different types of cyrillic (windows-1251, KOI8-R) or UTF-8.

The bug might be a result of how TB2 handles messages with non-latin encodings. This might be the reason why nobody else is commenting on this bug.

Comment 14

[Wayne Mery (:wsmwk)]

bienvenu, any thoughts?

peter, with version 3.1.7 can you confirm these things still happen:
1. mail is no longer going to the account files, mail is going to global inbox ...
2. but account specific inboxes are not deleted ...
3. and if you compact an account specific inbox, it doesn't reduce in size

(if #1 is gone then we need to change the summary)

Comment 15

[David :Bienvenu]

This bug doesn't have to do with compact - compact shouldn't have to look at accounts that are using the global inbox. The real bug is how somehow '0' bytes were getting written to the deferred inboxes. I have not heard any other reports of that, and if it has stopped happening in 3.x, then this bug would be incomplete or wfm.

Comment 16

[Peter]

Seems that the bug is not present with TB 3 branch - last modification date of pop.gmail.com Inbox file is currently 24.02.2009, which means that zero-padded files are not increasing their size anymore.

But I can confirm the above scenario:
> 1. mail is no longer going to the account files, mail is going to global inbox
- this has always been the case, mail has always been stored in the global inbox (Local Folders/Inbox), but in addition pop.gmail.com/Inbox used to get filled in with zeros. This seems like a bug in TB 2 branch, since I am not observing it with TB 3.

> 2. but account specific inboxes are not deleted ...
- this I can confirm too: pop.gmail.com/Inbox file is still present and currently is a about 48 MB of 0's. It will stay there indefinitely until I manually delete it. This has always been my point - implementing a clean-up procedure when upgrading TB. Previously I have seen accounts with a gigabyte of zero-padded files, if people just move their Profile folder when moving to a new computer, they will also move the empty files.

> 3. and if you compact an account specific inbox, it doesn't reduce in size
- since I have always used the global inbox (Local Folders/Inbox), compacting compacts Local Folders/Inbox file only. The account specific inbox pop.gmail.com/Inbox is not changed after compacting. I don't want to try the workaround provided by Jasper Knockaert (switch off the global inbox, compact, switch on again), since I want to keep the profile's current condition for future tests.

But again, disk space when I filed this bug 5 years ago was much more important than now. If no one else can confirm the bug though, it might be some rare condition with emails with bizarre Cyrillic encoding. The other 2 people which have commented on this bug have not provided any new information since 2007.

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to comment #16)
> Seems that the bug is not present with TB 3 branch - last modification date of
> pop.gmail.com Inbox file is currently 24.02.2009, which means that zero-padded
> files are not increasing their size anymore.

thanks. please file new bugs for other issues.