Open Bug 286888 Opened 16 years ago Updated 3 months ago

Always make compacting folders automatic, with no UI


(MailNews Core :: Backend, enhancement)

Not set


(Not tracked)


(Reporter: mpt, Unassigned)



(Keywords: privacy)

This is the Thunderbird equivalent of bug 229034, in the hope that Thunderbird's
developers are "removing ... chrome we don't need" and aiming for "simplified
preferences UI and menus" <>.

Compacting folders is an arcane operation that I shouldn't have to worry about.
It should happen automatically, in the background, whenever Thunderbird is not
busy doing anything else, and it shouldn't need any menu items or checkboxes.

Doing this would fix, or render obsolete, bug 205756, bug 223525, bug 236922,
bug 245607, bug 255687, bug 260004, bug 260252, bug 262485 (probably), bug
265431, bug 265472, and bug 271988, and it would subsume bug 61960.
(In reply to comment #0)
> This is the Thunderbird equivalent of bug 229034, 

...which met much opposition and was eventually wontfixed.

mpt, perhaps I'm missing something, but would you want the Trash (or Recycle
Bin) to auto-empty without user intervention?

(In reply to comment #1)
> ...which met much opposition and was eventually wontfixed.

four vocal users isn't exactly much opposition.
(In reply to comment #2)
> four vocal users isn't exactly much opposition.

It is much opposition when only four people commented on the bug.


let's not go into vote counting. You need a decent sample of users to do that.
bugzilla doesn't allow that. And never did. 
the question is: does fixing this bug help anything? And it does, given the list
of bugs mpt gave that would be fixed by this. So i say to go for this.
Bottom line: Besides the name of the mail client, I don't see any change between
the reasoning that lead to wontfixing bug 229034 and doing the same with this one.

1. That bug wasn't wontfixed by somebody with the power to do so, but by a
random other person
2. That bug ignored the long list of bugs that would be fixed by making
compacting automatic
3. That bug doesn't have real reasons for wontfix, just "i don't want that" and
"i might want to compact manually"
I said "Thunderbird's developers", not "the sort of people who still use the
Mozilla suite". If you don't like "simplified preferences UI and menus", please
use the Mozilla suite (or Opera), and stop soiling Thunderbird bug reports.

(And yes, I do have Apple Mail set to auto-delete messages in the Trash more
than 7 days old. A Trash folder is to hold stuff I *probably* don't want until I
almost *certainly* don't want it, well-indicated by its age. But that's a
separate RFE.)
There's one very good reason why this bug should be wontfixed: Users who access
their mail both through IMAP and through a web-interface need this feature.
Otherwise, a mail that was marked for deletion in TB would stay crosslined in
the web-interface until the 7 days period has passed. If you make this interval
significantly shorter, users are bound to lose mail they might not have wanted
to delete.

I can see the benefit for those who only use TB, but to assume that all TB users
don't also use web-interface is quite unfounded. Many users choose to use a mail
client from their home and a web-interface in public places. For them, having no
Purge/Expunge/Compact folders options would be counterproductive.

you are confusing cleaning up the trash and compacting folders. cleaning the
trash should happen after a few days, and isn't this bug.
compacting should happen much more often, like every hour or more. So webmail
users won't see crossed out items. the mails are just deleted.
It's not clear to me this feature is IMAP-related at all; IMAP already has a 
feature for "Clean up (expunge) Inbox on Exit."

The feature I wish to see is automated compaction of folders in POP accounts and 
Local Folders, to regain disk space, without having to do it manually.  (There 
is also a desirable compaction function for news accounts, but it doesn't appear 
to be implemented.)

The primary reason not to make this an automatic, background task is because 
compacting a folder can interfere with storing newly arriving mail for that same 
folder -- bug 274330, bug 139215 (same basic problem as bug 275132, bug 168648).
The program's backend needs to be able to queue these actions internally, 
deferring a Get New Mail if a compact is in process and vice versa.  This 
problem occurs even now, with user-initiated compaction, but would be more 
widespread if automatically occurring behind the scenes.
My IMAP account often overflows because of non-compaction, so no, it isn't just
POP/local. My Junk and Trash folders need compacting, so no, it doesn't apply
just to my Inbox. And "Clean up (expunge) Inbox on exit" is a pref, so no, it
doesn't satisfy this bug. And yes, I know compaction interferes with other
tasks; that's why I wrote "whenever Thunderbird is not busy doing anything else".
*** Bug 287037 has been marked as a duplicate of this bug. ***
We must do this for privacy reasons. People freak out when they find their old
deleted mail still in their mboxes; no one has a clue about the unobtrusive
Compact Folders menu item.

Desktop OS's have taught (most? some?) people about the concept of emptying the
trash/recycle bin, so bug 61960 would do in a pinch.
Flags: blocking-aviary1.1?
Keywords: privacy
I HAVE IMAP folders + *MARK AS DELETED* and i have auto-move-junk (in essence
also *mark as deleted*) in order to give me a second chance to undelete
erroneously deleted or marked-as-junk messages. So i'm hoping this "fix" doesn't
compact my IMAP folders before i've had a chance to verify all my deletions.

What makes this bug even worse for me is that marked-as-deleted messages often
*disappear* from view until a "Get Messages" triggers their re-appearance.

Simplifying UI is fine, but breaking useful behavior is not worth it.

I think auto-compacting of Local and POP folders is an excellent idea. I would
suggest that the usefulness of "mark-as-deleted" should prevent this bug from
affecting IMAP, until a better solution taking this into account is found. 
> a second chance to undelete erroneously deleted or marked-as-junk messages
That's why the Trash was invented. (Having "Delete" mean "don't delete, but mark
as 'deleted'" is utter crack, but that's a separate bug.) If you like, file a
separate RFE for a command to restore Trashed messages to their previous folder.
(In reply to comment #15)
> > a second chance to undelete erroneously deleted or marked-as-junk messages
> That's why the Trash was invented.

The trash is much less useful for this scenario because:

1. *Greater distance traveled to retrieve* - To save vertical space, it would
make little sense to have a trash folder in each account, so having one trash
folder in Local folders makes sense. If the user now accidentally deletes
something (or the junk filter moves something) he/she has to move (far) away
from where he/she is (from IMAP/inbox to LocalFolders/Trash), and then come all
the way back, and re-orient him-/herself.

2. It's *Harder to find* the erroneously deleted message among all the other
many messages in trash that are also there from all the other acounts.

> (Having "Delete" mean "don't delete, but mark
> as 'deleted'" is utter crack, but that's a separate bug.)

That's just like saying: Having "Delete" mean "don't delete, but move to trash"
is utter crack. Both have their useful purpose - one for POP & Local Folders,
the other for IMAP + mark-as-deleted.

> If you like, file a
> separate RFE for a command to restore Trashed messages to their previous folder.

No, this bug should be WONTFIXED until you find a solution that doesn't damage
the described usefulness of mark-as-deleted.
(In reply to comment #16)
> this bug should be WONTFIXED until you find a solution that doesn't damage
> the described usefulness of mark-as-deleted.

... or [this bug] limited to POP and Local Folders (but not IMAP + m-a-d).
For a user, there is no difference between imap and pop. Why should there be
when deleting messages? Why is trash useful for pop, but not for imap?

Delete should delete the message from the list. imap delete does that, the
message is just still stored. That is weird. And showing them makes delete not
delete. That's weird to.

Your story depends on one global trash to get a tiny bit of screen space. I
think the space is worth the usability of just making delete mean delete.
just for reference, my last isp offered imap, i used it, mark as deleted was the
only supported mode, the imap server did not allow other folders, and you could
not use the imap server for arbitrary storage. (the backing store was mailspool,
and once you removed a message from the Inbox 'folder', you could not put it back.)
In that case the Trash folder maybe should be a virtual folder showing the
mark-as-deleted objects. Then cleaning the thrash would mean compacting folders.
This could be automatic or manual, but at least is something the user knows about.
If the virtual folder where under the imap account, the user would be confused
to not see those messages when at another PC (home<->work). Your suggestion also
seems much less intuitive than the current "compact" for imap+m-a-d.

BTW: Michiel, have u ever tried "imap + m-a-d" for any length of time or at all?
With a virtual folder, i mean a folder that displays the events that imap thinks
are marked for deletion. It searches deleted messages in other folders. Those
messages are still on the server. So it works at work and at home.

But this isn't the topic of this bug. The question is: is compacting folders a
concept users know about? Do they really need to suffer the pain because a few
power users use some other mechanism?
And why is imap different from pop?
Guessing: IMAP is more powerful than POP? POP dooesn't have m-a-d?

Should we then dumb-down IMAP to be as limited as POP, just to make them "look"
the same? (i.e., pretend that m-a-d doesn't exist)
Locally, e-mail programs can give POP messages whatever statuses they like, such
as deleted-but-not-really, or junk-when-it's-raining, or flagged-in-a-full-moon.
But they don't, because that would be silly. It's silly for IMAP too; possibly
deleted-but-not-really is exposed in IMAP clients only because developers are
thinking too much about the protocol and not enough about the mental model.

Providing a consistent mental model, with the same power to discard messages,
and the same ability to "Edit" > "Undo", is not dumber -- it's smarter.
If we must have a homogenous mental model, then add m-a-d to POP. 

I don't want to have to go hunting around in the trash and junk folders for
accidentally junked or deleted messages - especially with bug 218467, bug 91218
and bug 149054 not really fixed yet.
yes, we want a homogenous model. Homogenous with the mental model the user has.
And in that model, delete means to get rid of a message. Throw it away. Maybe
into a thrash. It doesn't mean to say that maybe one day you might want to get
rid of it.
(and we are going around in circles here. One users want to mark as delete, all
the others that don't even know about bugzille very likely want delete to mean
delete. Lets stop commenting, and try to have the module owners make a decision)
Let's try to refrain from wild unproven claims, OK?(In reply to comment #26)
> One users want to mark as delete, all the others [...] 
> very likely want delete to mean delete

Let's try to refrain from wild unproven claims (that "happen" to support your
preference), OK?

Please note that IMAP is rarely used by novice users, and m-a-d is off by
default. The defaut behavior would (i suppose) be that IMAP+delete would move a
message to the trash folder, where this bug's request would apply.

Maybe IMAP + m-a-d could auto-compact at session-end, thus also satisfying this
bug's request? This (IIRC) unfortunately currently only works for the inbox, but
not any of the other IMAP (sub)folders. That would be a more sensible solution.

PS. I agree that the UI should be simple and consistent. However, the trick to
excellent software is not to completely eliminate everything that you think
might confuse some users, but to provide a clear and unobstructive path to ever
more powerful functionality, without sacrificing the simplicity.

"Lets stop commenting, and try to have the module owners make a decision" Agreed
It may be relevant that "compact" does three slightly separate things:

1. Local Folders & POP:    Permanently removes invisible yet present deleted 

2. Trash:                  Emties the trash. (slightly related to "compact")

3. IMAP + Mark-As-Deleted: Permanently deletes messages the user has "marked" 
                           for deletion.
afaik, compact folder always deletes deleted-but-not-just-yet messages. I'm very
dure it does that for imap.
Thrash is not related to compact. I don't know if that folder auto-compacts at
the moment, but the operation to empty it is called 'Empty Trash', not 'Compact'

This bug is about compact, not about empty trash.
a few thoughts:

Need an algorithm for timing the compaction. Offhand, I'd suggest >50% slack
space as a good trigger, that scales with the file size. There's that
auto-compact pref that uses a kilobyte number, but I'd guesstimate there isn't a
good one-size-fits-all number for that. Compacting all folders on an account
when its Trash is emptied (comment 13, bug 61960), would seem to imply very slow
shutdown for folks with Empty Trash on Exit and largish folders. A fixed
interval alone wouldn't be helpful when doing cleanup.

If a folder is under the mark-as-deleted scheme, show Compact Folder context
menu item on it as currently and don't consider the folder for automatic
compaction. Unless .01% of the .01% (sombrero numbers, obviously) who use
mark-as-deleted in the first place actually want to combine it with automatic
cleanup. I suppose they'll cry pref if it comes to that.
How about just n seconds after the last operation on the mailbox, and when not
very busy doing other things?
(the delay makes sure you are not compacting 10 times for deleting 10 emails)
Consider: a 100M Inbox, auto-move to Junk folder, badly spammed under account,
default biff interval, compact interval in the same ballpark as default biff
interval; since the junk messages make a brief visit to the Inbox, we'd be
writing an average of around ten megabytes a minute for about one kilobyte of
incoming messages per minute, which is a bit much.

That said, 10% slack space with 15min idle timer + on exit if necessary (similar
to junk training data flush timing), sounds reasonable.
*** Bug 301195 has been marked as a duplicate of this bug. ***
Flags: blocking-aviary1.5? → blocking-aviary1.5-
Flags: blocking-aviary2.0?
I'm a user, not a developer. I've read through your lively discussion, 
understanding only half, maybe less. I only vaguely know about IMAP. I stopped 
using Local Folders when I couldn't get Send Later and Send Unsent Messages to
I am POP and only POP. What I want is to choose 0 KB as the criterion for 
Automatic Compact Of All Folders Upon Close Of Application. If you implement it as 
an option, I'll be happy to be the only person in the world to choose it.
*** Bug 318190 has been marked as a duplicate of this bug. ***
Flags: blocking-firefox2? → blocking-thunderbird2?
*** Bug 319032 has been marked as a duplicate of this bug. ***
I agree with the concept that "Compact Folders" is a techo thing users shouldn't have to worry about. But, the mailstore design in Moz/TBird is not really designed for this and will have severe difficulties trying to implement it.

On the whole I think that this is better considered in terms of using a new mailstore which CAN be compacted in the background - something more like a real database.

Remember that some peopple have very large mailboxes. One of mine is 500MB.

Issues to consider:

When compaction happens, the mailbox is totally unusable until it's finished (right?). You can't start and stop, doing it in chunks, because it involves writing an entirely new copy of the mail file, not editing the existing file. I suppose it would be possible to come up with a scheme that accessed both the new and old mailbox as necessary, but that seems rather complex at the least.

On laptops, compacting folders would keep the hard drives and CPU going potentially for hours, running the battery flat. It's not OK just to automatically tie up the hard drive when the user is away. 

I think that auto-compacting is the correct way of dealing with mail folders. Pine ( automatically compacts folders on the fly for years. The process is transparent to the user since the compacting process is done correctly:
* The compacting process DOES NOT CREATE a new copy of the folder, but rather moves blocks of data within the file. 
* The compacting process DOES NOT BLOCK other operations on the folder, such as adding, reading mails or deleting (moving) emails. 
Users usually delete the most recent emails. Therefore the block to be moved contains the recent messages, and is relatively small even if you have 500Mb folders. 

Please consider automatic compacting with the assumption that it will be implemented correctly, not as it is done currently. If pine can do so, I am sure Thunderbird can do so too.
getting late in the cycle to block on this feature.
Flags: blocking-thunderbird2? → blocking-thunderbird2-
QA Contact: general
I m only a user but I had often problems with "slows" thunderbirds. Now I know from where come this problem : the compacting of the folders isn't activated.

If by using "compact", you mean "empty" trash, I don't really if it would be a good idea or not.

If by using "compact", you mean "really delete the mails from the hard drive" (this mainly affect pop3 accounts in my case) I fully agree. The compacting of folders checkbox should be checked by default.This in an option for optimizing performance, for the confort of the user, but it s hidden in a dark option panel... I don't even count any more the number of times I "saved" user by just activate this option.
This goes against or alike bug 75927 which is target tb3?

and btw, that one is "front end" not "general" ..
No, the main use cases are quite different. This is mainly about not having to know anything about compact at all, bug 75927 you want to know exactly that (and do it easily).
Assignee: mscott → nobody
Duplicate of this bug: 439263
Duplicate of this bug: 463333
Duplicate of this bug: 197605

Let's face it, the current compacting mechanism is deeply flawed. The current implementation copies all the messages in the folder file into a new folder file, and replaces the original file. This solution is blocking (major evil), and is very inefficient (lesser evil). A proper implementation of the compacting mechanism -- which does not block other operations, which makes use of in-file movement of messages blocks rather then copying messages between files, which
implement atomic operations -- can be automatic or semi-automatic and will fix most of the real compacting bugs (31, of which 5 are critical and 4 are major), while rendering many other bugs obsolete (at least 30).

Here is an organized list of the relevant compacting bugs.

Bugs which are related to the blocking misbehavior or the file-copy compacting "solution" side effects:
bug 101584, bug 137210, bug 139215, bug 152675, bug 218075, bug 220179, bug 274330, bug 305930, bug 321277, bug 324576, bug 327674, bug 337554, bug 383599, bug 390351, bug 396470, bug 398684, bug 406851, bug 411971, bug 444343, bug 444697, bug 446316, bug 446437, bug 458335, bug 467395, bug 479285, bug 481650.

Incorrect status after compacting:
bug 368768, bug 369643, bug 370409, bug 380598, bug 414106.

Bugs which are related to automatic/manual behavior:
bug  61960, bug 160281, bug 177074, bug 220064, bug=286888 [this bug], bug 328945, bug 361806, bug 398523, bug 437657, bug 439089.

Other bugs related to the compacting mechanism:
bug 224825, bug 273334, bug 280804, bug 302205, bug 315366, bug 319142, bug 331108, bug 370229, bug 398471, bug 406204, bug 428947, bug 463359, bug 467305, bug 468722, bug 470834, bug 477486.

Bugs which are related to the user interface of compacting (some already fixed, but still open):
bug 161752, bug 190285, bug 255687, bug 324085, bug 341206, bug 345795, bug 384092, bug 390355, bug 423763, bug 460648, bug 460819, bug 471937, bug 473642, bug 477419, bug 478380, bug 479064.

Enhancement requests related to the compacting mechanism:
bug 235153, bug 235741, bug 285681, bug 291812, bug 306228, bug 306561, bug 379103, bug 401295, bug 413955, bug 416532, bug 469050.


The correct compacting mechanism is based on moving messages within the folder file, rather than moving massages between files.

You are using two pointers in the messages file -- the first one points to the beginning of the unused space in the file, and the other points to the beginning of the first message after the unused space. For simplicity, let's assume the size of the message is less then the unused space, so copying the message can be interrupted by other actions, for example the user is viewing another message, moving a message to another folder or retrieving new messages from the mail server. All those operations can pause the in-file copying process, which can be resumed as the interrupting operation is completed. Once the message is fully copied, a short atomic operation is needed to update the message offset in the index/db file, and update the pointers of the unused space and the next message.

When there is no next message, the file is truncated at the unused space pointer, and the operation is finished. Another minor note is the usage of a "dirty flag", which flags if the currently copied message was modified (for example, by detaching or deleting its attachments), and therefore the copying should be redone.

If the message is larger then the unused space, there is a simple solution of blocking other actions for a second, or more complex solution of handling non-continuous messages in the file.

This solution (which is used by Pine and other mail clients, as well as by almost all descent text/hex editors which handle large files), guarantees that the compacting operation does not block other operations, as the only blocking operation is the atomic update of the messages offsets.

This mechanism is much faster, as the user usually erases new messages, which are stored near the end of the file. The new mechanism moves around relatively few messages, while the previous one copies all the older messages in the file. The new mechanism also avoids the long operation and the additional needed space for copying large message files.

Since this mechanism does not block other operations, it can be done automatically as part of the regular operation of the mail application, without the the user awareness. All the UI problems and most of the enhancement request will become obsolete.
Blocks: 454074
Blocks: 306228
Blocks: 361806
Duplicate of this bug: 361806
I think this bug is duplicate with this one: (at least, the roots cause is the same, but the solution just ask to change default behaviour and keep the UI).
I support
Zvi, thanks for presenting an organized view of the bugs.  

Bienvenu or someone else might be able to speak to the rest of comment 47, but I would suspect if a more efficient method of compaction was safe in the thunderbird environment that it would have been done already.
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
perhaps this won't be known until we have feedback from thunderbird 5 in the field...

given that 
- Bug 439089 exists for the efficiency of the backend process - Better Faster IMAP: Auto compact on idle time with better threshold (which isn't just about imap)
- bug 437657 is fixed - increase autocompact threshold "mail.purge_threshold" [and pref on by default] - which addresses the automatic part of this bug 

is there anything further for this bug to tweak or do?
Duplicate of this bug: 668655
Duplicate of this bug: 668655
Duplicate of this bug: 672066
Duplicate of this bug: 671295
I can't follow this long thread. It seems you divert into stray subjects and other BUGs.

The initial suggestion here was to make Compaction automatic without asking the user.
There's an option in preferences. If a user wants to make it automatic, one already can.

What is missing in preferences window is an option, a "tick box" to ask the user. It is always ON. You may still turn it OFF in the list of preferences by searching for "purge".

Compaction contains lots of BUGs, both known, and still unknown.
A user must be able to stop it, or to prevent it from starting.
Do not take control away from user.

I suggest to close this suggestion for enhancement.
I really don't know if this was addressed b/c I didn't read this entire thread, but every time a compact is done & I have a reply e-mail open & I'm in the process of me writing it, I can no longer use that e-mail. TB won't save it I get an error. I have to copy & paste the e-mail into a brand new fresh reply.

Lastly, I'm really not sure TB is compacting ALL the folders & I have HUNDREDS of them & sub & sub sub folders. It never tells me which folder it's compacting, how much longer it will take, & sometimes it can want to compact only minutes after I compacted earlier.

I don't think it works b/c I now have to deal with my "sent" folder b/c it's corrupted. I've never seen it compact the "sent" folder, but then again I barely go into the "sent" folder, I just use the "search" function on the "sent" folder.


Mozilla/5.0 (X11; Linux x86_64; rv:9.0a1) Gecko/20110822 Firefox/9.0a1 SeaMonkey/2.6a1 ID:20110822003134

I have compacting set to 1 MB (the current minimum; before I had 512 K IIRC) and "ask me" (which already requires going to about:config AFAICT); and the few seconds it takes me to react when asked are enough, I think, to avoid collision with mail-fetch in most cases (collisions happened before, several versions ago, but I know some bugs have been fixed since). Also the small threshold makes compaction events more frequent but shorter which seems to help keep my mail folders clean, and avoid corruption (and I don't think I have thousands of folders, but I certainly have hundreds, though not more than two or rarely three levels of depth).

In reply to comment #58: I see "Sent" being compacted after I have moved an email out of there (maybe in order to get both sides of a conversation into some other folder). This is normal since the email that has been moved away leaves a "gap" which is not reclaimed until the next compaction. It could also happen legitimately after an automatic "forgetting" of the oldest messages in Sent (if one of the "Delete if..." radio buttons is selected in the "Disk space" preferences for Local Folders).
Duplicate of this bug: 306228
Duplicate of this bug: 961378
Would there be a way to get it to do compacting when the overall CPU usage is at or near idle? Then have it go either in order of folders, or by large to small or small to large, or even better, allow folks to choose the compact order. I know this would need some more development of a user interface, but "one stop shopping" for compacting would not be a bad thing, and people can then argue up and down the merits of different options.
Duplicate of this bug: 1529254

Wow, I was JUST thinking about this yesterday Wayne.

TB isn't telling me that it's compacting or that I need to compact.

That's not good, right?

See Also: → 716412
See Also: → 1678856

There is of course also maildir, which don't need compacting.

You need to log in before you can comment on or make changes to this bug.