495862 - Imap Folders grow - compacting does not help (infinite growth of offline-store by rebuild-index : "Compact doesn't reduce file size" part, Tb Trunk, even after fix of Bug 420115, Bug 466730, Bug 465385)

Status: RESOLVED DUPLICATE of bug 499630

(MailNews Core :: Networking: IMAP, defect)

Description

[gene c]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2a1pre) Gecko/20090527 Minefield/3.6a1pre
Build Identifier:  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1pre) Gecko/20090601 Lightning/1.0pre Shredder/3.0b3pre ID:20090601030621

Imap folders such as INBOX never shrink - they keep growing. Mine reached 20 GiB (which was well over 20 - 30 times the size after deleting INBOX and restarting TB.

As example here is what happens to INBOX as of today (less dramatic since I regularly delete the INBOX file now and restart TB - its only 4 times larger than it should be.

Sizes in the left column.


Before
-------
89881346 2009-06-01 15:24 INBOX
  554360 2009-06-01 15:32 INBOX.msf

After Compacting
-------
89881346 2009-06-01 15:24 INBOX
  554610 2009-06-01 15:34 INBOX.msf

Rebuild Index (got even bigger by 20% or so - 1 new mail came in of size < 1 Kb)
-------
117182203 2009-06-01 15:50 INBOX
   479321 2009-06-01 15:50 INBOX.msf

Exit TB - Delete Files and restart TB on same account
(a few small emails came in during refresh - INBOX is 1/4 the size
-------
27330497 2009-06-01 16:49 INBOX
  510399 2009-06-01 16:56 INBOX.msf




Reproducible: Always

Steps to Reproduce:
1.IMAP INBOX
2.use it
3. It never shrinks, compacting and/or rebuilding index do not shrink it - in fact rebuilding index made it larger
4. delete INBOX - restart TB
5. INBOX is much much smaller



This problem is a direct result of using the old mbox format - 1 large file to hold everything.

It would be well worthwhile for this and speed reasons to add a more modern format such as maildir as the local store format. maildir uses a separate file for each message and so cannot have this problem.

Please consider adding maildir format for local storage.

Comment 1

[WADA:World Anti-bad-Duping Agency]

"File size growth upon each Rebuild-Index" part is Bug 487992.
Please focus on "never shrink by compact" in this bug.

Comment 2

[Ludovic Hirlimann [:Usul]]

Andrew what kind of information should we try to collect in order for you guys to fix databases issues like this ?

Comment 3

[WADA:World Anti-bad-Duping Agency]

gene c, Gmail IMAP?
With Gmail IMAP, I could observe phenomenon of this bug on folders except next special folders;
  [Gmail]/All Mail,[Gmail]/Trash (\Deleted flag is kept. Probablly [Gmail]/Spam too)
But I couldn't repoduce the phenomenon with above special folder. (i.e. "Compact Folder" reduced offline-store file size for [Gmail]/All Mail.)

Comment 4

[gene c]

Mine is using a dovecot imap server.

Comment 5

[WADA:World Anti-bad-Duping Agency]

FYI. I've opened Bug 499630 for special Gmail IMAP case.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Bug 466730(in bug summary) is bug for same external symptom(opened on 2008-11-25), and finally fixed around 2009/1/16. According to Bug 466730 Comment #21, it's verified with 2009/1/18 build. And according to Bug 466730 Comment #21 & #21 (answer to my #22), it's verified by owner of Bug 466730 too.

gene c, Can you reproduce your problem with 2009/1/18 build?      
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2009/01/2009-01-18-03-comm-1.9.1/

Comment 7

[WADA:World Anti-bad-Duping Agency]

Note:
According to your Bug 463359 Comment #10, you look to have experienced the problem with 2009/2/06 build and 2009/2/06 build.

Comment 8

[WADA:World Anti-bad-Duping Agency]

gene c, what is you "delete model" choice? (Server Settings, When I delete a message:)

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Before
> -------
> 89881346 2009-06-01 15:24 INBOX
>   554360 2009-06-01 15:32 INBOX.msf
> After Compacting
> -------
> 89881346 2009-06-01 15:24 INBOX
>   554610 2009-06-01 15:34 INBOX.msf

When "Compact Folder" is executed, EXPUNGE command is issued by Tb?
If yes, response to EXPUNGE command from IMAP server contains "*NNN EXPUNGED" response?
(See https://wiki.mozilla.org/MailNews:Logging)

Comment 10

[gene c]

(in reply to Comment #8 https://bugzilla.mozilla.org/show_bug.cgi?id=495862#c8)

I swear i replied to this ... oh well .. trying again:

 I have it set to 'move to trash'. However I always shift delete or move to different folder. My Trash is always empty. So deleted mails are no longer held on imap server.

 May I respectfully request - that devs please please consider maildir as a local store - the advantages are obvious - and all this compacting code just goes away - we ge a simpler code base that is faster, much simpler and more efficient. Please ? 

  Thanks for your continued efforts in getting this resolved.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #10)
> So deleted mails are no longer held on imap server.

In both "Shift+Delete" case and "Move mail" case, mail in folder(or in move source folder) is merely flagged as \Deleted(uid XXX store flag \Deleted).
Do you mean that your IMAP server automatically expunges mail flagged \Deleted as Gmail IMAP does?

Comment 12

[Peter Weilbacher]

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1pre) Gecko/20090623 SeaMonkey/2.0b1pre

Same effect as described by gene with one of my IMAP accounts. INBOX is > 200 MB here, containing 2966 messages while the INBOX on the server only contains 31 mails of total size not more than 5 MB.

I ran with imap:5 logging and saw this when compacting (folding is mine)
1657633040[7f6c504f5ac0]: 50b69000:<IMAP_SERVER>:S-INBOX:ProcessCurrentURL:
   entering
1657633040[7f6c504f5ac0]: 50b69000:<IMAP_SERVER>:
   S-INBOX:ProcessCurrentURL:imap://pmw@<IMAP_SERVER>:993/Expunge%3E.INBOX:  =
   currentUrl
1657633040[7f6c504f5ac0]: 50b69000:<IMAP_SERVER>:S-INBOX:SendData: 21 expunge
1657633040[7f6c504f5ac0]: ReadNextLine [stream=4f83fa10 nb=14 needmore=0]
1657633040[7f6c504f5ac0]: 50b69000:<IMAP_SERVER>:
   S-INBOX:CreateNewLineFromSocket: * 31 EXPUNGE
1657633040[7f6c504f5ac0]: ReadNextLine [stream=4f83fa10 nb=13 needmore=0]
1657633040[7f6c504f5ac0]: 50b69000:<IMAP_SERVER>:
   S-INBOX:CreateNewLineFromSocket: * 32 EXISTS
1657633040[7f6c504f5ac0]: ReadNextLine [stream=4f83fa10 nb=12 needmore=0]
1657633040[7f6c504f5ac0]: 50b69000:<IMAP_SERVER>:
   S-INBOX:CreateNewLineFromSocket: * 1 RECENT
1657633040[7f6c504f5ac0]: ReadNextLine [stream=4f83fa10 nb=25 needmore=0]
1657633040[7f6c504f5ac0]: 50b69000:<IMAP_SERVER>:
   S-INBOX:CreateNewLineFromSocket: 21 OK EXPUNGE completed

so that seems to work just fine. (I can see the messages disappearing from the server if I use Alpine to access the same folder before and after).

The affected account is set up like this:
- SSL/TLS
- just mark it as deleted
- expunge inbox on exit
- empty trash on exit
- server supports sub-folders
- use IDLE if supported

The only config difference to two more IMAP accounts which have local INBOX sizes of 0 bytes is that those have "empty trash on exit" NOT set.

Let me know what other info I can provide.

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #12)
> * 31 EXPUNGE
> - just mark it as deleted

Was the expunged mail deleted by the Tb(marked with read cross at Tb's thread pane) who executed "Compact Folder"? If yes, was it removed from Tb's thread pane?
Or the expunged mail was deleted by other IMAP client(or via Web Interface if Web mail)?

Comment 14

[Peter Weilbacher]

It was first deleted by me (pressed the Del key while in the list; it was then marked with a red cross, yes) and then expunged by me (selecting "Compact This Folder" from SeaMonkey's context menu of the INBOX folder; the deleted message then vanished from the displayed list of mails).

Comment 15

[gene c]

Reply to https://bugzilla.mozilla.org/show_bug.cgi?id=495862#c11:

  I have expunge on exit set as well .. fyi.

 
  These kind of corruption problems cannot exist with maildir ... pretty please lets add that option for local store ? Rebuilding index from scratch will be slower, but everything else will be faster and corruption free. Please ?

Comment 16

[WADA:World Anti-bad-Duping Agency]

For original case(comment #0, compact after multiple rebuild-index, delete/expunge has no relation).
Regression window.
>(Compact reduced offline-store file size in next test)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081112 Shredder/3.0b1pre
>(Compact didn't reduce offline-store file size in next test)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081113 Shredder/3.0b1pre

(1) By Tb 2009/6/25 build. (To avoid unwanted issues of old builds)
    1*10KB mail in a folder XXX. Delete XXX & XXX.msf and restart Tb,
    Open folder XXX => file_size=10KB. Rebuild-Index => file_size=20KB
    Terminate Tb 2009/6/25 build.
    Note: I tested with "Move to Trash model".
          I think delete model is irrelevant to this bug's case.
(2) Start 2008/11/13 build
    Open folder XXX, Compact Folder => file_size was still 20KB.
(3) Start 2008/11/12 build
    Open folder XXX, Compact Folder => file_size was reduced to 10KB
    (Tb 2.0.0.22 reduced offline-file size too.)

To Peter Weilbacher:

Can you execute similar test to Bug 499630 Comment #3 with ordinal IMAP server? 
Same regression window as above even when delete/expunge has relation?
Or same regression window as Bug 499630 Comment #3? (between 2008/11/21 and 2008/11/22, Gmail IMAP case, Gmail IMAP doesn't return *nnn EXPUNGE to EXPUNGE command, except for [Gmail]/Trash & [Gmail]/Spam.)
Or different regression window?

If different regression window from this bug's one, "between 2008/11/12 and 2008/11/13", open separate bug for ease of problem analysis, please.

Comment 17

[WADA:World Anti-bad-Duping Agency]

Confirming, based on test result of Comment #16.

To Peter Weilbacher: See also Bug Comment #7(test result with [Gmail]/Trash).

Comment 18

[WADA:World Anti-bad-Duping Agency]

Some changes in regression window.
> Bug 457751 
> 3c4f45a99041: fix 457751, r=emre, sr=neil, delete imap message, rebuild index, do undo can crash
>   David Bienvenu <[e-mail]> - Thu, 13 Nov 2008 06:51:53 -0800 - rev 1093
>   fix 457751, r=emre, sr=neil, delete imap message, rebuild index, do undo can crash
> Bug 385502
> 9346f93d9620: 385502, configure all imap folders for offline use by default, r=me
>   emre - Thu, 13 Nov 2008 14:34:26 -0800 - rev 1100
>   385502, configure all imap folders for offline use by default, r=me

By change of Bug 457751, compactor is not invoked if no mail deletion?

Note:
As I wrote in Bug 499630 Comment #12, 2008/11/12 to 2008/11/21 build reduced offline-store file size, if Shift+Delete of mail was executed, although restart of Tb was required due to problems of other bugs.

Comment 19

[WADA:World Anti-bad-Duping Agency]

According to Bug 499630 Comment #21, "Compact of Folder Properties" doesn't invoke Compactor of offline-store. File/"Compact Folders" should be used instead.

To gene c(bug opener) and Peter Weilbacher:

Can you see problem with File/"Compact Folders"?

Comment 20

[Peter Weilbacher]

WADA: File -> Compact Folders does work. But it's not clear what that is supposed to (compact all folders in the currently selected account, in all accounts, or where else?) do, so that is really suboptimal. And it's also not clear what "Compact of Folder Properties" is compacting; if with that the visible and the real contents of a folder get out of sync then I consider it to be broken.

Haven't had time yet to try your regression window (comment 16).

P.S.: Could you not add your reference bug numbers in the whiteboard instead of the bug summary? It always destroys threading in my bugmail folders when someone changes the subject...

Comment 21

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #20)
> But it's not clear what that is supposed to (compact all folders in the currently selected account,
> in all accounts, or where else?) do, so that is really suboptimal.

Open separate bug for good documentation, please.

> And it's also not clear what "Compact of Folder Properties" is compacting;

See Bug 499630 Comment #23, please.

> P.S.: (snip)

I see. I'll be carefull with subject change.

Comment 22

[WADA:World Anti-bad-Duping Agency]

This bug is INVALID? Or WONTFIX?
Or should be morphed to documentation bug?
Or should be changed to request of "Get Compact come back"?

Comment 23

[gene c]

In response to https://bugzilla.mozilla.org/show_bug.cgi?id=495862#c19

Before 
109,850,665 2009-07-07 21:25 INBOX

File->Compact Folders
 34,407,684 2009-07-07 21:32 INBOX

Delete INBOX and restart tb: (one new email came in)
 34,524,427

So yes - I confirm that this does seem to compact the folders ... 

However - i still plead to remove the problem by using maildir, instead of adding hacks upon hacks to deal with a weak design.

Maildir will mean no compacting at all - and all that code just goes away .. the menus get simpler too as a bonus!!

gene/

Comment 24

[gene c]

This is clearly a bug - leaving gigantic folders with a fraction of useful stuff in them is just silly. 

Even asking the user to compact is silly - why should the user have to know about compacting some internal storage decision ?

The data should be maintained by thunderbird - and asking the users to press buttons to overcome flaws is a work-around at best for poor choices.

All compact buttons should eventually be removed - and tb should be programmed to store things in a more sensible way (maildir would accompmlish that - there may be other or better ways).

In the meantime at least make the compact button on a single folder actually work - or better make thunderbird do its own housekeeping.

Comment 25

[gene c]

Another reason to stop using a single file storage format is backups - they are far slower on than multiple files - robustness/corruption issues aside.

Comment 26

[WADA:World Anti-bad-Duping Agency]

What had happened around "Compact and offline-store" was analyzed in Bug 499630.
Closing this bug as DUP of Bug 499630.