411971 - Auto compact folder seriously flawed and can delete messages without warning

Status: RESOLVED DUPLICATE of bug 439089

(Thunderbird :: General, defect)

Description

[Greg Webb]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
Build Identifier: version 2.0.0.9 (20071031)

I only enabled auto compacting of email folders when upgrading to 2.0 because the messages requesting I turn it on were far more annoyingly insistent than before - given a better choice I would have left it off and manually compacted periodically.

From day one, it caused problems. Consistently, auto compact would kick in while I was reading through a folder - frequently when I marked the first missed mail as junk, producing an error dialog the second I clicked on 'junk' stating the folder was in use by another operation and should be left alone, when the operation was only initiated by my requesting the operation it was complaining about.

If I was reading in a child window not the preview pane, no operations would work in that window and the parent folder window would move to the top of the folder - I would have to stop reading and start again to get the mail to work. When reading through a folder, this is a serious annoyance.

Periodically, the compact routine would automatically kick in _during_ an automatic regular download of new messages. This would cause Thunderbird to raise an error message for an operation it had started automatically - surely it would be possible for Thunderbird to check if one operation was running before starting another?

When an automatic compact coincided with an automatic download, this would stop all filters working. Frequently, it would also cause messages already filtered automatically into subfolders - and deleted from those subfolders - to reappear in the inbox, requiring me to go through downloaded messages a second (and sometimes third or even fourth) time.

This evening, auto compact caused a more serious problem - it deleted around half of the mails in my inbox without warning or placing messages in deleted items - they have been wiped completely, automatically, without warning or option for recovery.

This performance is wholly unacceptable. I have used Mozilla Thunderbird and its predecessor products for my e-mail for around 10 years and have recommended them wholeheartedly to friends and colleagues. Thunderbird 2's auto compact has caused me such serious annoyance, even before it deleted so much of my mail, that I cannot honestly recommend it at present and am seriously investigating alternatives.

If auto compact cannot be fixed, and quickly, the responsible software project management answer must surely be to remove it from the product.

Reproducible: Always

Comment 1

[Wayne Mery (:wsmwk)]

(In reply to comment #0)
> 
> I only enabled auto compacting of email folders when upgrading to 2.0 because
> the messages requesting I turn it on 

Greg, You don't cite any message. What is the exact message?  

Is this all folders that aut ocompact, or just one?
 

> Periodically, the compact routine would automatically kick in _during_ an
> automatic regular download of new messages. This would cause Thunderbird to
> raise an error message

what error message?


>This evening, auto compact caused a more serious problem - it deleted around
>half of the mails in my inbox without warning or placing messages in deleted
>items - they have been wiped completely, automatically, without warning or
>option for recovery.

might be Antivirus. Have you checked the knowledge base??  Here is a quote from Andrew

"Assuming [Antivirus] is the problem, you can try and avoid it happening in the future by following the steps on this page:

http://kb.mozillazine.org/Download_each_e-mail_to_a_separate_file_before_adding_to_Inbox

However, the fact that the problem showed up during the compaction phase suggests that the setting may not be at issue; it could be the anti-virus triggered on the compaction process.  In that case, the following page has more advice, with the suggestion to tell the anti-viral software not to scan the Inbox (or perhaps any Thunderbird mail folders) being what I'm mainly thinking of:

http://kb.mozillazine.org/Antivirus_software "

Comment 2

[Tuukka Tolvanen (sp3000)]

(as an aside: you may want to increase the auto-compaction threshold in tools -> options -> advanced -> space to a level where it doesn't trigger annoyingly often. the default 100k is rather low.)

Comment 3

[Greg Webb]

While I appreciate the suggestions, there are still fundamental issues here - 

1. The design of the AutoCompact function means it requires exclusive access to a folder, yet it does not enable it to queue requests for action. If Thunderbird maintained a request queue and simply paused its folder access operation if there was a blocking action on a specific file or other exclusive object, the problem would not have occurred.

2. There is something inherently flawed in AutoCompact being able to trigger as a result of a user action (such as marking a mail as junk) and yet put it into the action queue _before_ that user action has been recorded completed. For an action to be automatically triggered in response to a user request and then block the user request from completing is inherently user-hostile.

3. The design of AutoCompact at present means it is performing an operation that, if fails, is destructive to user data - yet it kicks off without first performing any form of backup of the user data.

4. Many users read mails sequentially in a child window. If an autocompact kicks in while reading a folder, fine - but it surely cannot be an enormous task to note sufficient information about the email being read to return the user to that email when the operation is complete, rather than forcing them to close the window and start again?

I appreciate there are significant challenges in this sort of function and that it's necessary to maintain performance. I appreciate that mbox files aren't exactly ACID-compliant databases. I appreciate that Thunderbird isn't produced by an army of commercial developers - I'd join in myself if I wasn't committed at present elsewhere. Nonetheless, the design of AutoCompact strikes as sloppy, with far too much potential for causing user problems quite apart from the data loss scenario I suffered.

Point 1 would have clear potential for a lengthy job to fix, but point 2 could surely be addressed relatively simply and points 3 and 4 are likely very quick changes (though clearly 3 would have to perform auto-cleanup routines, disk space checking, and could have some problems with larger files). Any one of these being changed would surely result in very significant user benefit?

Comment 4

[Daniel Veditz [:dveditz]]

Confirming. There are several related bugs on the problems caused by compaction, and having compaction triggered at random times tends to make issues come up more frequently and unexpectedly.

Comment 5

[David :Bienvenu]

Compaction makes a clone of the folder with just the non-deleted messages and only when that succeeds does it copy the clone back over the original. It gets a lock on the folder so other activities can't change the folder.

We're planning on changing auto compaction to happen on idle, and not when the user deletes a message, which should get it much more out of the way of the user. And we should be able to halt the compact gracefully if the user stops being idle or some other activity requires the folder.

I'm minusing since the other work should ameliorate this issue.

Comment 6

[Greg Webb]

If compaction makes a usable, workable clone of the folder before starting, that should surely be in such a manner that were a compact to fail then Thunderbird would automatically roll over (or at least provide the user the option to roll over) to the clone? That was certainly not the case in my experience, as explained.

Still, an incremental compact that's capable of being paused should definitely remove the user-annoyance side of this report. If Thunderbird could stop itself from starting an action that blocked your action in response to your request, and stop itself from running two mutually incompatible actions simultaneously, that would be a major benefit.

Comment 7

[David :Bienvenu]

(In reply to comment #6)
> If compaction makes a usable, workable clone of the folder before starting,
> that should surely be in such a manner that were a compact to fail then
> Thunderbird would automatically roll over (or at least provide the user the
> option to roll over) to the clone? That was certainly not the case in my
> experience, as explained.

Sorry, maybe I wasn't clear. The original folder is the workable, good state, and compaction doesn't touch it. Only if the clone succeeds do we copy the clone over the original. Perhaps we failed to detect that there's something wrong with the clone in your case.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> From day one, it caused problems. Consistently, auto compact would kick in
> while I was reading through a folder - frequently when I marked the first
> missed mail as junk, producing an error dialog the second I clicked on 'junk'
> stating the folder was in use by another operation and should be left alone,
> when the operation was only initiated by my requesting the operation it was
> complaining about.

(A) What is your prefs.js setting for "confirmation dialog of auto compact"?
> mail.purge.ask = false or true
> (true is default. if reset, this entry is not displayed in Config Editor)
Once you request "Don't ask any more"("Do this automatically..." or something) at confirmation dialog, mail.purge.ask=false is set and auto-compact always runs silently. There is currently no way to reset this setting except manual resetting via Config Editor.
(AFAIR, UI for resetting is already requested)
(B) What is your mail.purge_threshhold value? (unit = KB, Kilo-bytes)
(AFAIR, increase of default value is already done by Tb trunk)
(C) Do you have mail folder(s) of 2GB<mail_folder_file_size<=4GB?

"compact folder" itself is relatively safe operation. If failing component is "compact", garbage of nstmp or nstmp-N (work file) remains and is displayed as a mail folder after restart. However, if contention with other operation occurs while compacting, and if failing component is other than compaction or other component interferes compaction process, it may cause problem. Bug 392015 is an example of worst problem.

AFAIR, "scheduling of auto compaction while downloading" will be prohibited by Tb 3, but the change is not back ported to Tb 2.

To Greg Webb(bug opener):  
If you still experience the problem frequently, try next.
(1) Change setting
> mail.purge_threshhold = large value (10000 or more. i.e. 10MB or more)
> mail.purge.ask = true
(2) If confirmation dialog appears while downloading, reply "Cancel".
    Never request "Do automatically ..." at dialog.
Further, there are problems when 2GB<mail_folder_file_size<=4GB.
  Bug 450359(fixed1.8.1.18. i.e. fixed by Tb 2.0.0.18). See also Bug 450991.
If you have big mail box, make it smaller.

I have an mail box around 2GB(currently greater than 2GB. It sometimes becomes below 2GB by delete&compact, but keeps around 2GB because I don't delete many mails). And I saw "Error truncating ..."(phenomenon of Bug 321371) several times on Seamonkey 1.1.14, while POP3 downloading & filtering(my message filter moves many mails to the big folder).
Note: Even after the error message, no problem occurred on local mail folders. So it seems for me simply a mistake in return code handling.
By fix of Bug 450359, I think many problems due to 2GB<mail_folder_file_size<=4GB disappered. But big mail box always causes "long compaction time" & "long rebuild index time". These will surely increase probablity of contention of mail folder(s).

Comment 9

[Wayne Mery (:wsmwk)]

questions in comment 8

you might want to test the current beta
 http://www.mozillamessaging.com/en-US/thunderbird/early_releases/
 backup your profile before using

Comment 10

[Wayne Mery (:wsmwk)]

bienvenu, shall we morph this to align to doing your comment 5?
Bug 542708 may be another example.

It would really help to have more detail from greg and others about symptoms, so we can match up other bugs. "flawed" isn't quite descriptive enough to do that.

Comment 11

[David :Bienvenu]

wsmwk - yes, that sounds reasonable. Long term, moving to something like maildir gets rid of the need to compact at all.

Comment 12

[Wayne Mery (:wsmwk)]

it turns out we already have bug 439089

Comment 13

[chrizilla]

@Wayne, something doesn't seem right here:
this bug with dataloss keyword and critical severity duped to a WONTFIXed bug ?

Comment 14

[Wayne Mery (:wsmwk)]

The original reporter is gone, some of the issues reported I think no longer exist or have been reduced, and the issues remaining issues are I think adequately covered in other open bugs.

If anything we might reopen bug 439089, but that was just one possible approach. Hopefully bug 1890448 will help.