487992 - File size of offline store of IMAP folder increases upon each "Rebuild Index"

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[WADA:World Anti-bad-Duping Agency]

> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090411 Shredder/3.0b3pre

File size of offline store of IMAP folder increases each "Rebuild Index".
(0) Set an IMAP folder to offline-use=on. (say Folder-1, mail data size=1MB)
    Delete file of Folder-1 & Folder-1.msf
(1) Restart Tb
(2) Open Folder-1 => Automatic rebuild-index, file size of Folder-1=1MB
(3) Folder-1 : Rebuild Index => file size of Folder-1=2MB
(4) Repeats (3) => file size of Folder-1 increases 1MB upon each Rebuild Index

This is probably caused by mismatch among some features.
(A) "Rebuild Index" doesn't care for offline store.
(B) auto-sync/"Download Now" simply appends mail data to offline store.
(C) Because no "mail delete" exists, "Compact" is never invoked.
    Note:
    "Compact folder" somehow never shrinks offline store size in any case.
    It's different issue. It may be relevant to particularity of Gmail IMAP.

"Rebuild Index" should delete offline-store before start of rebuilding of msf.

Comment 1

[Judith Hellerstein]

Bug 49322 is different.  Bug 49322 occurs when Thunderbird running under IMAP senses its Index file for the Inbox-Inbox.msf file is corrupt it will rebuild and recreate it.  Are you saying your Inbox.msf file is never recreated unless it is manually deleted. In that case this bug, 487992 is a totally different bug. Bug 49322 occurs only when Thunderbird running under IMAP thinks its current index is corrupt and rebuilds it-rebuilds the inbox.msf file. No manual rebuild is ever done to generate this action.

The bug in 49322 is that the Inbox file is not rebuilt/recreated each time the
Inbox.msf is recreated/regenerated. The file, Inbox, grows almost exponentially
so 76.7 mb becomes 159 and 159 becomes 236.5 mb. Until it gets too large.  I
now delete it every week and start over. Creation date of Inbox is does not
change with each rebuild/recreation of Inbox.msf. There are no duplicate
emails, file just gets doubled. 

2) 49322 is different since not looking to compact file. So reference to compact and mail delete button do not apply to bug 49322

3)auto-sync/"Download Now" works correctly, except when the size of the Inbox file is larger than 2-3 gb. As long as inbox.msf is not rebuilt/recreated Download now or download mail offline will work correctly, ie, download the new mail and only incrementally increase the size of the inbox by the size of the new mail that is downloaded.

Comment 2

[Judith Hellerstein]

Meant also to say that bug occurs with the MAC OS platform as well

Comment 3

[Judith Hellerstein]

Sorry Left off the extra 2 in bug number it is 493222

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #1)
> Bug 493222 is different.

As I wrote Bug 493222 Comment #5, this bug is for next problem which you reported in Bug 493222.
> The bug here is that the Inbox file is not rebuilt/recreated each time the
> Inbox.msf is recreated/regenerated. The file, Inbox, grows almost exponentially
> so 76.7 mb becomes 159 and 159 becomes 236.5 mb. Until it gets too large.
Test case in comment #0 is to simpify bug reporting. But your case is really occurred never-negligible problem. If your bug is closed as DUP of this bug, I believe this bug is mis-evaluated as "very rare edge case which doesn't occur in real environment". So I keep your bug open and keep dependency setting. If you think it's incorrect action, change them, please.

Comment 5

[Wayne Mery (:wsmwk)]

Wada, do you still see this?  If so, suggest setting blocking-thunderbird3 to "?"

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #5)
> Wada, do you still see this?

Does it mean you can't reproduce or see phenomenon of this bug any more with latest trunk?

Comment 7

[WADA:World Anti-bad-Duping Agency]

Oh, you are trying to triage many bugs(and trying to close {(many)**N, where N>>1}
incomplete or unclear or invalid bugs) :-)

This bug's phenomenon still occurs, but it's never critical issue because "File/Compact Folders" executes "compact of offline store" as designed and reduces offline-store file size as expected.
This bug is for better behaviour of Tb, in order to reduce user's confusion and to avoid unwanted bug reports at B.M.O due to Tb's not-so-well behaviour.

Comment 8

[Judith Hellerstein]

I can confirm this bug still occurs.  I do not download the latest trunk preferring to wait for the complete version to come out.  I have to trash the inbox and inbox.msf files each week as they get too large because of the bug. If there is another way to fix it, please do tell

Comment 9

[WADA:World Anti-bad-Duping Agency]

FYI.
As I wrote in Bug 499630 Comment #32, "Compact" of folder context menu has started to execute "compaction of offline-store file" again. (by Bug 482476).

Comment 10

[Judith Hellerstein]

I guess I do not understand why you need to choose compact folder at all.  Compact folder is designed to compact a folder that is true to its size, that the size is based on the size of each email that is contained in it. In this case, the inbox size in not calculated correctly. If it was then each time the inbox was deleted and downloaded again it should be the same size each time. In this case, this is not correct. The inbox significantly shrinks in size to the actual size of the files within it and not to a fictitious size. It seems you are using compact to shrink the inbox from its phantom size to a smaller size. A better way would be to fix the bug so that this situation does not occur at all, then you would not need to compact the file.

Comment 11

[Magnus Melin [:mkmelin]]

When you delete a mail, since we use the mbox storage format, the mail is only "marked for deletion" and hidden from you. Therefore compact is needed to remove the deleted messages. Removing it directly would often be veeery slow - for large folders it could be many seconds, minutes even, for each deletion!

Comment 13

[David :Bienvenu]

Attachment: proposed fix

Comment 14

[Magnus Melin [:mkmelin]]

When i hit "Rebuild index" i get this in the native console

2009-10-15 22:13:03     autosyncActivities      ERROR   onFolderAddedIntoQ: [Exception... "Component returned failure code: 0x80004002 (NS_NOINTERFACE) [nsISupports.QueryInterface]"  nsresult: "0x80004002 (NS_NOINTERFACE)"  location: "JS frame :: file:///opt/moz-objdir/mail/mozilla/dist/bin/modules/activity/autosync.js :: anonymous :: line 196"  data: no]
2009-10-15 22:13:03     autosyncActivities      ERROR   onDownloadStarted: TypeError: syncItem is undefined
2009-10-15 22:13:03     autosyncActivities      ERROR   onDownloadCompleted: TypeError: this._syncInfoPerFolder[folder.URI] is undefined
2009-10-15 22:13:03     autosyncActivities      ERROR   onFolderRemovedFromQ: TypeError: syncItem is undefined

Comment 15

[Magnus Melin [:mkmelin]]

Comment on attachment 406358 [details] [diff] [review]
proposed fix

Though that seems to happen without the patch too. Looks good to me! r=mkmelin

Comment 16

[David :Bienvenu]

fix pushed.

Comment 17

[Andrew Sutherland [:asuth] (he/him)]

This seems like back-end-ish logic that should not be living in the UI code for the folder pane.  Perhaps it could be migrated to a new JS module?

Comment 18

[David :Bienvenu]

You're right that this is backend-ish. It could actually have been done in the c++ backend code SetSummaryValid, when the summary is set invalid for an imap or news database. I'm rather hoping that we can fix our offline store story so that this isn't required, or that the decision is postponed until it appears that the offline store is invalid in some way.