492344 - Once folder update(e.g mail copy) occurs without explicit folder open when outdated msf condition exists, "Compaction of local mail folder" is never executed, until next mail delete happens.

Status: NEW

(MailNews Core :: Database, defect)

Description

[WADA:World Anti-bad-Duping Agency]

"Compact Folder(local mail folder)" doesn't remove deleted mail(X-Mozilla-Status: 0008/0009), if "outdated msf" condition exists, and mail is copied is executed without folder open, and "compact folder" is executed without folder open.
This phenomenon was observed during my trial to duplicate phenomenon of Bug 479285. 

[ Tested build ]
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b5pre) Gecko/20090506 Shredder/3.0b3pre

[ Steps to reproduce ]
(1) Folder "Test" has some mails & some deleted mails(X-Mozilla-Status:0008/0009).
    Execute "Rebuid-Index", view some mails in "Test", and terminate Tb.
(2) Add active mail data to (or remove from) file of "Test" by text editor.
    (Change file size & time stamp. Force "outdated msf".)
(3) Restart Tb
(4) Click other folder than "Test", copy a mail to "Test" via "Copy to folder".
    => Mail data is appended to file of "Test".
       Timestamp of ".msf" file was not updated.
      (Note: If Tb 2.0.0.19, "Test.msf" was deleted. If "Copy mail" is executed)
      (      again, "Test.msf" of file size=0 was created.                     )
(5) Right Click of "Test" (folder is not opened), "Compact Folder".
    => Timestamp of ".msf" file was updated.
    => Deleted mail data(X-Mozilla-Status:0008/0009) was not removed from file
       of "Test".
(6) Left Click of "Test"(folder open), "Compact Folder" of "Test" folder.
    => Deleted mail data(X-Mozilla-Status:0008/0009) was removed from
       file of "Test".

Note-1: Above (5)/(6) was observed with Tb 2.0.0.19 too.
Note-2: If folder is left-clicked(folder is opened) at any step, internal
        rebuild-index is invoked, and "outdated .msf" condition disappers.

Comment 1

[Ludovic Hirlimann [:Usul]]

Wada can you attach an outdated .msf ?

Comment 2

[WADA:World Anti-bad-Duping Agency]

> Wada can you attach an outdated .msf ?

Ludovic, "Outdated .msf" of which step?

Accurately speaking, above step (1)/(2) is emulation of next flow.
  1. Mail folder is closed, and cached MailDB data is written in ".msf".
  2. Copy event occurs. (e.g. Sent mail saving, Junk move, Filter move)
  3. Append of mail data to mail folder file.
  4. MailDB is updated in memory, but not written to disk.
  5. Power failure or system crash.
Meaning of "Outdated .msf" at step (3)/(4) is discrepancy between "time stamp/file length in non-corrupted Test.msf file" and "time stamp/file length of Test.msf file" only. To avoid affect by mail.db_timestamp_leeway, emulation is executed with file size change at step (2).
And something wrong is generated in step (4), but it's not written back to disk file. So there is no .msf file which produces phenomenon of step (5).

Comment 3

[WADA:World Anti-bad-Duping Agency]

Correction.
> discrepancy between "time stamp/file length in non-corrupted Test.msf file"
                  and "time stamp/file length of Test file"

Comment 4

[WADA:World Anti-bad-Duping Agency]

I guess flag for "there are deleted mails(compaction needed)" is lost at step (4).
i.e. Compact is not invoked at step (5).

Comment 5

[WADA:World Anti-bad-Duping Agency]

Attachment: Test.msf after shutdown of Tb(mail-1:deleted,mail-2:active)

Comment 6

[WADA:World Anti-bad-Duping Agency]

Attachment: Mail file of Test (mail-1:deleted/not-touched, mail2:active/subject length is increased manually, mail-3:copied without folder open)

Comment 7

[WADA:World Anti-bad-Duping Agency]

Attachment: Test.msf (after restart of Tb, copy a mail wthout fopder open, and compact without folder opne)

Comment 8

[WADA:World Anti-bad-Duping Agency]

Above three attached files:
(0) Test : mail-1: deleted mail, mail-2: active mail
(1) Shutdown Thunderbird.
    => Attached Test.msf #1
(2) Add string to Subject: of mail-2 by text editor(increase file size).
(3) Restart Tb.
(4) Copy a mail(mail-3) to Test, without folder open.
    timestamp of Test.msf is not changed.
    "Edit of Test.msf" says "Test.msf is write-locked".
(5) Compact via context menu, without folder open.
    timestamp of Test.msf is updated.
    "Edit of Test.msf" says "Test.msf is write-locked".
    => Attached Test, Test.msf #2

Comment 9

[WADA:World Anti-bad-Duping Agency]

Test.msf #2 after step (5) has data for mail-3. It indicates "internal rebuild-index due to outdated msf" is normally invoked at step (4) in comment #0.
So I tested next scenario.
(0) Test : mail-1: deleted mail
(1) Shutdown Thunderbird.
(2) Add mail data for active mail-2 to Test by text editor(increase file size).
(3) Restart Tb.
(4) Nop => Test of "Compact folder when outdated msf" case 
(5) Compact via context menu, without folder open.
    timestamp of Test.msf is updated.
    "Edit of Test.msf" says "Test.msf is write-locked".
    => Test.msf has data for mail-2.
    => mail-1(deleted) is not removed from Test.
Above maybe is said as:
  If "outdated msf" condition exists, "Compact folder" silently fails.

Real problem is next?
  If "explicit folder open via UI" is not executed yet,
  "Internal rebuild-index due to outdated msf condition" fails to clear
  "outdated msf condition".
   - Triggers of "Internal rebuild-index due to outdated msf condition"
     without explicit folder open:
     - mail append(sent mail save, move/copy by filter, Junk move, ...)
     - mail purge
     - auto-compact

Comment 10

[WADA:World Anti-bad-Duping Agency]

New observation.
 (A) Once "internal rebuild-index due to outdated msf condition" is invoked
     without explicit folder open,
     any "compact folder"(context menu, File/Compact folder/auto-compact)
     doesn't reduce mail folder file size, until active mails are newly deleted.
 (B) When unread mails are copied without explicit folder open after (A),
     new mail count of folder pane is automatically updated.
     Note: Test.msf file is write locked at step (A).
           It indicates Test folder is opened internally.
 (C) If "internal rebuild-index due to outdated msf condition" is invoked
     with explicit folder open(mail folder click),
     "compact folder" after rebuild-index reduces mail folder file size.

Phenomenon can probably be said:
- "internal rebuild-index due to outdated msf condition" clears flag for
  "compact is needed", if rebuild-index is invoked without explicit folder open.
- If "compact folder" is invoked without explict folder open when "outdated msf"
  condition exists, "internal rebuild-index due to outdated msf condition" is
  invoked without explict folder open, and "compact is needed" is cleared.
  Then "compaction of folder" doesn't occur.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Patch for bug 449768 and/or bug 459680 doesn't work as expected, if explicit folder open is not executed before invoking internal rebuild-index?

Comment 12

[WADA:World Anti-bad-Duping Agency]

After reading explanation by Kent James about "exception by outdated msf condition upon mouse over" in Bug 493429, I tested without "mouse-over" event.
Problem was reproduced by next simplest test.
 0. Tb knew deleted mails in a folder, Terminate Tb
 1. Add mail data to folder file
    (emulation of outdated msf due to power failure)
 2. Restart & File/Compact Folders (emulation of auto-compact)
    => No mouse over, so no exception at this step
    => Timestamp of msf was updated. Deleted mail was not removed.