498185 - MailDB(.msf) is corrupted(Rebuild-Index is invoked), if "Compact Folder" is invoked on "copy target folder" while "copy to folder by message filter" is running. "Compact Folder" itself silently fails.

Status: NEW

(MailNews Core :: Database, defect)

Description

[WADA:World Anti-bad-Duping Agency]

MailDB(.msf) is corrupted(Rebuild-Index is invoked), if "Compact Folder" is invoked on "movet target folder" while "copy to folder by message filter" is running. "Compact Folder" itself silently fails.

Phenomenon was observed with Tb 2.0.0.21 and next Tb trunk build.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1pre) Gecko/20090608 Shredder/3.0b3pre

[Steps to reproduce]
(0) Folder-1: Copy source folder. 4MB mail * 25 mails = 100 MB Mbox
              (big mail, to force long time to copy a mail) 
    Folder-2: Copy target folder. Compact will be invoked.
                1 active 64KB mail + Many Shift+Deleted mails.
                Tested with file size = 100MB to 400MB.
    Message filter: If subject contains ..., Copy to Folder-2.
                    All mails in Folder-1 is copied to Folder-2.
    Open both Folder-1 and Folder2, and terminate Tb.
(1) Restart Tb, open 2 Tb windows.
(2) Window-1: Click Folder-1, "Run Filter on" Folder-1
(3) Window-2: Right click Folder-2(not left click, no explict folder open),
              and "Compact Folder", while filter is running.
              => "Done:" immediately at status bar.
                 ("Compact Folder" silently fails) 
(4) Wait for end of filer copy.
(5) Window-2: Click Folder-2
              Mails are displayed. (1 active mail at (0) and copied 1600 mails
              (Compact was not executed normally)
              Note: Reuild-index was observed at this step several times.
(6) Window-2: Click other folder than Folder-2, and wait for .msf file update
              (wait for delayed .msf close after MailDB close)
(7) Window-2: Click Folder-2 again => Rebuild-Index was invoked.

Note:
If "Compact Folder" is requested at step (5), nothing happens. It's similar phenomenon to bug 492344.

Comment 1

[Sean M. Pappalardo]

I think I'm seeing the same issue in TB 60.4.0, with slightly different circumstances:

1. IMAP folder with 40K messages in it (mbox on-disk storage and set to "keep messages for all folders in this account on this computer")
2. Mass-delete a bunch:
   a) multiple-select ~1000
   b) shift-delete (no copy to trash), confirm warning
3. Invoke Compact on the folder
4. See the "operation is already in progress" message, click OK.
5. Folder now appears empty.
6. Close Thunderbird quickly.
7. Re-open Thunderbird
8. Folder is populated again, but TB re-downloads all messages.

I believe the problem is happening at step 4. TB appears to assume the index is now useless so proceeds to rebuild it from scratch, which is a giant PITA on a large folder with download messages set to on. Even more so with Gloda enabled, (which proceeds to consume all of the system RAM if given enough time! But that's another problem.)

Ideally, TB would check to see if there is another operation in progress before attempting to compact and just leave the index alone, letting the other operation finish what it's doing. (So the user would just need to try compacting again later.)

Comment 2

[Sean M. Pappalardo]

Oh, this is on Debian Linux 9 BTW.