274330 - Mail lost when a filter rule applied while compacting a folder is in progress

Status: NEW

(MailNews Core :: Filters, defect)

Description

[Rafal Peisert]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; pl-PL; rv:1.7.5) Gecko/20041108 Firefox/1.0
Build Identifier: Thunderbird v1.0 (20041206) Pl

On startup Thumderbird asks sometimes if the folders should be compacted. If
allowed and new mail arrives and a filter rule is applied an error saying that
the folder is being compacted and the filter rule couldn't be applied pops up.
The problem is that the messages disappear completely.

Reproducible: Always
Steps to Reproduce:
1. Start Thunderbird
2. Allow it to compact the folders
Actual Results:  
New mail wasn't placed in the appropriate folder and completely disappeared.

Expected Results:  
wait for compacting the folders to be done before retrieiving mail OR at least
leave the messages in the Inbox.

Comment 1

[OstGote!]

related to Bug 139215, bug 168648, bug 152675

Comment 2

[WADA:World Anti-bad-Duping Agency]

Main reason for "mail loss" is :
 - Mail in the account's Inbox does not moved to Global Inbox
   when "move to folder" action in message filter fails.
Same situation is reported to Bug 273778, although the reason why "move mail
failure" is different from this bug.

Comment 3

[WADA:World Anti-bad-Duping Agency]

Change Product/Component.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Same situation has been reported to Bug 275467, although the reason why "move mail
failure" is different from this bug.
Bug 275467's trigger is rename of mail folder of "Global Inbox" account.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Same situation is reported to Bug 275132.
Bug 275132's trigger is downloading to move target folder(Bug 168648).

Comment 6

[Andreas Gayde]

*** Bug 276946 has been marked as a duplicate of this bug. ***

Comment 7

[Mike Cowperthwaite]

I was bit by this bug this morning, so I'm confirming it.  I was using 
TB1.0+0309, Win2K.  I was testing the automatic compact-on-startup setting.
I got the prompt, responded OK, then got the password prompts -- which prompts, 
incidentally, cause the compaction process to stop, so you can't even wait for 
the compaction to quit.

I entered the passwords, and shortly thereafter got a notification of two new 
messages; but these were nowhere to be found.  I checked with my sysadmin who 
said I had in fact downloaded two messages, one from bugzilla and one from some 
random yahoo address.  The bugzilla message would have been filtered (and the 
bugzilla target folder probably would have been compacted itself); the other was 
probably ID'd as junk.  Fortunately, neither was important.

However, there is no evidence of the received mail in the Inbox, the Bugzilla 
folder, or the junk folder (per the workaround described by m-wada at 
bug 273778).  This may well be the same filtering issue as those others, but the 
mail loss does make this critical.

In my opinion, the real problem here is the haphazard implementation of 
compaction.  Once compacting starts, that folder should be locked -- there 
should be no activity allowed on it, automatically or by user command.  If this 
is not acceptable, then an internal action queue needs to be created so that 
nothing happens to the folder until the compaction process is complete.  

Also, the UI for compaction just sucks -- a menu item here, a context menu item 
there, a setting in a nearly-hidden settings dialog, and Yet Another Stupid 
Modal Dialog that appears on startup.  But there's bugs already for those 
problems.

Comment 8

[Scott MacGregor]

where's the modal dialog that comes up on first launch? I've never seen that before.

Compation UI is quite clean from my point of view. File / Compact Folders
or Folder / Right click Compact Folder. Seems pretty cut and dry UI wise from
that perspective.

Comment 9

[Mike Cowperthwaite]

(In reply to comment #8)
> where's the modal dialog that comes up on first launch? I've never seen
> that before.

File | Offline | Offline Settings,
  [] Compact folder when it will save...

If that's checked, and you start the program, and your Inbox (maybe other 
folders too) has NN kb worth of compactible space, you get a prompt.  (I may be 
mistaken on the modal-ness of this, but it doesn't seem to advance to prompting 
me for passwords.)  The same prompt pops up if you switch to a folder that also 
has sufficient reclaimable space.


> Compation UI is quite clean from my point of view. File / Compact Folders
> or Folder / Right click Compact Folder. Seems pretty cut and dry UI wise from
> that perspective.

From my perspective, this is housekeeping that the user should never have to 
deal with as an action; ideally, never even to set parameters.  Bug 223525 and 
bug 197605 speak directly to this point, but really, compaction would be best if 
automatic.  How about compacting once the program has been minimized?

See also bug 205756, bug 177074, bug 157991, bug 198936, bug 260252, bug 255687, 
bug 285681.  As I said, plenty of other bugs (these are all RFEs, actually).

Even the basic menu operation is overly minimal: Compact Folders, from the main 
menu, should be able to compact folders for all accounts.  Compact Folders 
should also be on the context menu for an account (bug 93001).  And for IMAP 
accounts, the verb should be changed -- bug 236922.

Comment 10

[OstGote!]

*** Bug 291201 has been marked as a duplicate of this bug. ***

Comment 11

[WADA:World Anti-bad-Duping Agency]

Fix for similar situation is now available(Bug 273778 has been FIXED.)
Is the problem resolved by the patch for Bug 273778?

Comment 12

[Wayne Mery (:wsmwk)]

Mel, can you test whether this still fails?  And/or lap up related (but non-filter) bugs regarding compact?

no one in the bug is responding, and the reporter's address is dead.

Comment 13

[Melchert Fruitema]

Testing with latest Tb version 2.0.0.21 (20090302) on MacOS 10.3.9.

I have tried many variations on compacting folders in an account. They all produced this message:

"Do you wish to compact all local and offline folders to save disk space."

Note: 'local' does not mean the ones in the Local Folders account, only. It includes the folders of the 'non-global' POP accounts!

So far, so good. I checked 'Do this automatically from now on'.

Then I tried the following scenario:-

Positioned on folders in a mail account in the Mail & Newsgroups window, I attempted to compact 'one' folder, while a move of a large (5Mb) file from one to another folder was in progress.

The compact attempt triggered the following message:

  Alert
  -----
  Unable to delete messages in folder one
  because it is in use by some other operation.
  Please wait for that operation to finish and
  then try again. 

The file size of the folder was not reduced; the compact was cancelled, in fact.

I believe, I proved that the original bug is dead. Would you agree?

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #13)
> The compact attempt triggered the following message:
>   Alert
>   -----
>   Unable to delete messages in folder one
>   because it is in use by some other operation.
>   Please wait for that operation to finish and
>   then try again. 

Reproduced with Tb 2.0.0.21 on MS Win-XP.
"Bulk Move" was terminated/interfered by invoking of "compact folder".

> I believe, I proved that the original bug is dead. Would you agree?

I agree with you.
When I execute "Run Filters On Folder"(filter has rule of "move to A_FOLDER") while compaction of A_FOLDER was in progress, same message appeared.
I think current phenomenon is "silent failure of move by message filter upon download" if "compact folder" is invoked during Message Filtering(AFAIR, already reported bug).
I also think your test & test result is very useful for analysis of Bug 310431 & Bug 324777. (Issue of "silent failure of Junk move". One of causes looks to be "compact folder" during Junk filtering.)

> The file size of the folder was not reduced; the compact was cancelled, in fact.

Unable to reproduce with Tb 2.0.0.21 on MS Win-XP.

In above case, failed task is "Bulk Move" as the error message says. "compaction of folder" continued(file size of "ntmp" increased cotinueously).
In my case, confirmation dialog by auto-compact also appeared. And when I replied OK to the dialog, nstmp-1 was created, similar error message of "other operation is in progress" appeared, and nstmp-1 was deleted when I reply OK to the error message(second compact by auto-compact failed).
After it, first "compact folder"(which was manually invoked) continued, and folder file size was reduced(time stamp of "folder_name" & "folder_name.msf" was updated too, and nstmp/nstmp.msf disappeared), because "deleted but not removed yet" mails existed in the mail folder.

To Melchert Fruitema:
"Compact folder" was really canceled in your case?
Because "Bulk Move of N mails" = (A) "Copy N mails to target folder" + (B) "Put DELETED flag to N mails in source folder", "Compact of source folder" won't reduce file size if "Bulk Move" is terminated during phase (A), unless mails already flagged as DELETED exist in the source mail folder.
How many mails were successfully copied to target folder?
(Note: I'm trying to find cause of Bug 482836 and Bug 485348. So, I want to know condition that "compact folder" is interfered or killed or fails.)

Comment 15

[Melchert Fruitema]

(In reply to comment #14)
> "Compact folder" was really canceled in your case?

Yes, I had to select that action, again. 

> Because "Bulk Move of N mails" = (A) "Copy N mails to target folder" + (B) "Put
> DELETED flag to N mails in source folder", "Compact of source folder" won't
> reduce file size if "Bulk Move" is terminated during phase (A), unless mails
> already flagged as DELETED exist in the source mail folder.

Indeed, the file size is only reduced after my successful compact.

Note: the message indicates that the ((f)actual) delete (by compact) is not possible, at the moment.

> How many mails were successfully copied to target folder?

Only one biggy (5 MB).

> (Note: I'm trying to find cause of Bug 482836 and Bug 485348. So, I want to
> know condition that "compact folder" is interfered or killed or fails.)

The compact is not executed, it is not put on hold, for sure. The message confirms that with Phrase "Please wait ... then try again."

Comment 16

[Melchert Fruitema]

Today, circumstances were such that this message popped up:

"This folder is being processed. Please wait until processing is complete to get messages".

I have no idea why this message is given! What was going on? Any way the get messages processing carried on, happily.

The circumstances:

Wakeup from system sleep mode. Instantly followed by a 'get new messages' for all counts; download of 3 messages from a POP account and 4 message headers in an IMAP account and 2 messages (bodies). Filters were used to move the incoming POP and IMAP messages to 'Local Folders'.

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to comment #16)
> Today, circumstances were such that this message popped up:
> 
> "This folder is being processed. Please wait until processing is complete to
> get messages".
>...
> The circumstances:
> 
> Wakeup from system sleep mode. Instantly followed by a 'get new messages' for
> all counts; download of 3 messages from a POP account and 4 message headers in
> an IMAP account and 2 messages (bodies). Filters were used to move the incoming
> POP and IMAP messages to 'Local Folders'.

Mel, rkent has theorized such errors may be more likely to happen at startup (which wake is sort of like).  And I can see that a user might be impatient and do something like "get new messages", or simply be doing other TB tasks whilst this accumulated work is in progress.  Conversely I think most of us working bugs have decent/fast machines and networks making it less likely we would see such things.

Anyway, I am glad you encountered this problem :)

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> > How many mails were successfully copied to target folder?
> Only one biggy (5 MB).
>(snip)
> The compact is not executed, it is not put on hold, for sure.
> The message confirms that with Phrase "Please wait ... then try again."

Aha! I didn't test with big mail. I tested with big folder instead(many small mails, up to 80K, total 2GB).
(Your case)
As "move of a big mail by bulk move" is in progress, start of compact fails by successful locking/serialization.
(My case)
Compact of "source folder of bulk move" is invoked between "move of mail(s)" and next "move of mail(s)". So compact is started. Then next "move of mail(s)" fails by successful locking/serialization("flag as deleted" step probably fails). Thus "bulk move" is terminated at middle of process.

Melchert Fruitema, I absolutely agree with you on "You proved that the original bug is dead".

Comment 19

[toddgleason]

I think this bug may be real.  I lost an 8MB mail (due to 2 attachments) the other day with 3.0b2.  The email was sent at night while my Vista machine was in hibernation.  I tend to wake it, and then hit Get Mail.  The mail was even sent from the same account (source and destination were actually 2 mailboxes on the same account through the same web hosting service).  Sender has receipt of send, so I believe Thunderbird dropped the email, and yes, I have auto-compact enabled.  I have now un-checked "Compact folders when it will save over" (I had 500K set here).

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #19)
> I think this bug may be real.

[e-mail], what is evidence that problem on you is completely same as original problem of this bug - mail move to move-target folder by message silently fails due to silent auto-compact, and deleted from Inbox?
Please note that "same external symptom" doesn't always mean "completely same problem". Please note that '"Compact folders when it will save over" was enabled' only can not be evidence that original problem of this bug occurred on you.

Gmail POP3 account?
If yes, mail is held in "All Mails" folder of Gmail Web Interface, if "move from Inbox to [Gmail]/Trash or [Gmail]/Spam folder via Gmail IMAP" is not executed. Even if "move from Inbox to [Gmail]/Trash or [Gmail]/Spam folder via Gmail IMAP" is executed, mail is held in Spam folder of Gmail Web(==[Gmail]/Spam via Gmail IMAP) or Trash folder of Gmail Web(==[Gmail]/Trash via Gmail IMAP), unless permanent delete operation such as "Empty Trash" on [Gmail]/Trash is executed via Gmail IMAP.
Check "All Mails", "Spam", "Trash" folder content via Gmail Web Interface. 

To [e-mail]:
 
Even when "Compact folders when it will save over" is enabled, if you disable option for "Do auto-compact silently"(which was set by your response to dialog),  "Confirmation dialog before auto-compact" will be issued. (see Bug 486947 Comment #14 and Bug 486947 Comment #16)
> mail.purge.ask=false => mail.purge.ask=true
Check whether "Confirmation dialog before auto-compact" is issued or not. And, if dialog is issued, check difference between "reply OK" and "reply Cancel" to the dialog.

Comment 21

[toddgleason]

There's no guarantee the problem was the same.  But the email was definitely large enough to trigger a compaction (500K was my threshold and the email was 8MB).

Not a Gmail account at all, a typical IMAP account through a web hosting company.  (it's not the gmail userid I use here)

Since the problem has only happened one time as far as I know, I don't know whether it will help for me to change any settings and re-try.

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #21)
> a typical IMAP account through a web hosting company. (it's not the gmail userid I use here)

Auto-compaction of IMAP folder is never controlled by next setting.
>(local mail folder setting referred by Comment #9) 
> File | Offline | Offline Settings,
>   [] Compact folder when it will save...
And meaning/internal action of "compaction" is very different between IMAP folder and local mail folder. Compaction of IMAP consists of;
   (a) request purging of deleted mail to server by EXPUNGE command
   (b) remove already deleted mail data from offline-store(if offline-use=yes)
       Note: (b) is similar to compaction of local mail folder.
Because this bug is for "local mail folder" case, and because comunication with IMAP server is involved in both "compaction" and "move of mail by filter"(move of mail by COPY command etc.), IMAP case should be processed in separated bug.

[e-mail], please open new bug for IMAP case attaching IMAP log to bug, if you can reproduce your problem.

Comment 23

[WADA:World Anti-bad-Duping Agency]

To bug opener, Mike Cowperthwaite(Comment #7 poster):

What is your answer to next question by Melchert Fruitema in Comment #13?
> I believe, I proved that the original bug is dead. Would you agree?

To Rafal Peisert(bug opener):

Can you still see (2) in your Comment #0?
> Actual Results: ( (1)/(2) is added by me ).  
> (1) New mail wasn't placed in the appropriate folder
> (2) and completely disappeared.
(Problems of (1) still exist. See dependency tree for meta bug 498274.) 

To Mike Cowperthwaite(Comment #7 poster):

Did you see both of above (1) and above (2)? Still Problem (2) exists?

Meta bug 498274 is for issues like this bug. Currently listed bugs are for "local mail folder" case only. In dependency tree for meta bug, I can't see bug for problem of "mail loss" except this bug. (Excluding: Bug 276946 which is DUP'ed to this bug, Bug 275132 which is closed as EXPIRED).

If compact folder is running, "Delete of mail" or "delete step of filter move" is not executed at least by Tb 2.0.0.21 and Tb trunk. i.e. Expunged flag in X-Mozilla-Status is not written. So, even after "Rebuild Index" and/or "Compact Folder", mail data won't be lost.
I think "mail loss" won't occur if next case.
  (a) Local mail folder, Non-Global Inbox, Quarantine option=Disabled.
If "mail loss(for user)" when local mail folder still remains, I guess next case only.
  (b-1) Quarantine option=Enabled : Move fails, then mail is kept in hidden file.
  (b-2) Global Inbox : Move to Inbox of "Local Folders" fails,
                       then mail is kept in hidden Inbox.
I can say nothing about IMAP case, because IMAP case is different from "local mail folder" case.

Comment 24

[Mike Cowperthwaite]

Wayne asked me to check in here, but I don't have much to contribute.  I've long since turned off the automatic compaction, and I'm really not inclined to recreate the conditions I had when I encountered the problem.  In fact, there's a good chance that my long-time POP mailbox will be converted to IMAP in the next year, at which point this bug becomes completely moot to me.

Unless/until the core code has implemented a scheme to lock a folders to compaction while downloading and vice versa, the details about my particular symptom (probably difficult to reproduce) aren't so important -- there will continue to be subtle, occasional symptoms like this.

Comment 25

[Wayne Mery (:wsmwk)]

I suggest this may have been Fixed in version 18.0 2012-10-10 via Bug 674742 - Compacting a Mail folder can delete all copies of the folder in certain error conditions.

Does anyone still see this issue? And whether yes or no, when is the LAST time you saw this issue?

For historical reasons, here are some of the compact fixes in the years prior to 2009 that may address dataloss:

• https://bugzilla.mozilla.org/attachment.cgi?id=200360&action=diff - Bug 313210 - Compacting folder might delete content/messages Fixed in 1.8
• Bug 273778 - If target folder for filter is deleted/moved received mail is not visible in INBOX
• Bug 275132 - Can't filter/move messages to Inbox that is busy, results in loss of moved messages when Global Inbox(mail is not moved to it) under TB Ver. 1.0 (20041206)
• Bug 95584 - Mail manged/lost (possibly filter problem)(possibly 'compact' problem)
• Bug 173357 - Compacting folders when getting lock failed
• Bug 101584 - Improve alert for compact folder interruption by biff/auto get msgs. Currently "This folder is being processed. Please wait until processing is complete to get messages."

FWIW According to comment 20, this bug should only be about pop accounts/local folders.

Comment 26

[Wayne Mery (:wsmwk)]

Perhaps this should be retested by QA after more of the compact / corruption bugs land.

Comment 27

[Corey Bryant]

Vlad/Ramona, can you test on 132.0b4 or above and see if this issue still exists?

Comment 28

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello,

Tried to reproduce this issue using 135.0a1(20241210105041) on Windows 11, macOS 14 and Ubuntu 24 and was unable to encounter it anymore.

Comment 29

[Wayne Mery (:wsmwk)]

Same question as other bug, Is this literally still possible given your understanding of the current code?

Comment 30

[Ben Campbell]

Adding messages to a folder should fail in the folder is undergoing compaction , and the calling code should deal with that failure in some sensible way.
I'm reasonably confident that the former occurs, but not very confident of the latter.

I'd be surprised if there weren't some nasty corner cases lurking somewhere that still trigger this bug.