File size of offline store of IMAP folder increases upon each "Rebuild Index"

RESOLVED FIXED in Thunderbird 3.0rc1

Status

MailNews Core
Networking: IMAP
RESOLVED FIXED
8 years ago
8 years ago

People

(Reporter: World, Assigned: Bienvenu)

Tracking

({fixed-seamonkey2.0.1, perf})

Trunk
Thunderbird 3.0rc1
fixed-seamonkey2.0.1, perf
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

8 years ago
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090411 Shredder/3.0b3pre

File size of offline store of IMAP folder increases each "Rebuild Index".
(0) Set an IMAP folder to offline-use=on. (say Folder-1, mail data size=1MB)
    Delete file of Folder-1 & Folder-1.msf
(1) Restart Tb
(2) Open Folder-1 => Automatic rebuild-index, file size of Folder-1=1MB
(3) Folder-1 : Rebuild Index => file size of Folder-1=2MB
(4) Repeats (3) => file size of Folder-1 increases 1MB upon each Rebuild Index

This is probably caused by mismatch among some features.
(A) "Rebuild Index" doesn't care for offline store.
(B) auto-sync/"Download Now" simply appends mail data to offline store.
(C) Because no "mail delete" exists, "Compact" is never invoked.
    Note:
    "Compact folder" somehow never shrinks offline store size in any case.
    It's different issue. It may be relevant to particularity of Gmail IMAP.

"Rebuild Index" should delete offline-store before start of rebuilding of msf.
(Reporter)

Updated

8 years ago
Blocks: 493222

Comment 1

8 years ago
Bug 49322 is different.  Bug 49322 occurs when Thunderbird running under IMAP senses its Index file for the Inbox-Inbox.msf file is corrupt it will rebuild and recreate it.  Are you saying your Inbox.msf file is never recreated unless it is manually deleted. In that case this bug, 487992 is a totally different bug. Bug 49322 occurs only when Thunderbird running under IMAP thinks its current index is corrupt and rebuilds it-rebuilds the inbox.msf file. No manual rebuild is ever done to generate this action.

The bug in 49322 is that the Inbox file is not rebuilt/recreated each time the
Inbox.msf is recreated/regenerated. The file, Inbox, grows almost exponentially
so 76.7 mb becomes 159 and 159 becomes 236.5 mb. Until it gets too large.  I
now delete it every week and start over. Creation date of Inbox is does not
change with each rebuild/recreation of Inbox.msf. There are no duplicate
emails, file just gets doubled. 

2) 49322 is different since not looking to compact file. So reference to compact and mail delete button do not apply to bug 49322

3)auto-sync/"Download Now" works correctly, except when the size of the Inbox file is larger than 2-3 gb. As long as inbox.msf is not rebuilt/recreated Download now or download mail offline will work correctly, ie, download the new mail and only incrementally increase the size of the inbox by the size of the new mail that is downloaded.

Comment 2

8 years ago
Meant also to say that bug occurs with the MAC OS platform as well

Comment 3

8 years ago
Sorry Left off the extra 2 in bug number it is 493222
(Reporter)

Comment 4

8 years ago
(In reply to comment #1)
> Bug 493222 is different.

As I wrote Bug 493222 Comment #5, this bug is for next problem which you reported in Bug 493222.
> The bug here is that the Inbox file is not rebuilt/recreated each time the
> Inbox.msf is recreated/regenerated. The file, Inbox, grows almost exponentially
> so 76.7 mb becomes 159 and 159 becomes 236.5 mb. Until it gets too large.
Test case in comment #0 is to simpify bug reporting. But your case is really occurred never-negligible problem. If your bug is closed as DUP of this bug, I believe this bug is mis-evaluated as "very rare edge case which doesn't occur in real environment". So I keep your bug open and keep dependency setting. If you think it's incorrect action, change them, please.
(Reporter)

Updated

8 years ago
Blocks: 495862
(Reporter)

Updated

8 years ago
No longer blocks: 495862
Depends on: 499630, 495862
Wada, do you still see this?  If so, suggest setting blocking-thunderbird3 to "?"
Keywords: perf
(Reporter)

Comment 6

8 years ago
(In reply to comment #5)
> Wada, do you still see this?

Does it mean you can't reproduce or see phenomenon of this bug any more with latest trunk?
(Reporter)

Comment 7

8 years ago
Oh, you are trying to triage many bugs(and trying to close {(many)**N, where N>>1}
incomplete or unclear or invalid bugs) :-)

This bug's phenomenon still occurs, but it's never critical issue because "File/Compact Folders" executes "compact of offline store" as designed and reduces offline-store file size as expected.
This bug is for better behaviour of Tb, in order to reduce user's confusion and to avoid unwanted bug reports at B.M.O due to Tb's not-so-well behaviour.

Comment 8

8 years ago
I can confirm this bug still occurs.  I do not download the latest trunk preferring to wait for the complete version to come out.  I have to trash the inbox and inbox.msf files each week as they get too large because of the bug. If there is another way to fix it, please do tell
(Reporter)

Comment 9

8 years ago
FYI.
As I wrote in Bug 499630 Comment #32, "Compact" of folder context menu has started to execute "compaction of offline-store file" again. (by Bug 482476).

Comment 10

8 years ago
I guess I do not understand why you need to choose compact folder at all.  Compact folder is designed to compact a folder that is true to its size, that the size is based on the size of each email that is contained in it. In this case, the inbox size in not calculated correctly. If it was then each time the inbox was deleted and downloaded again it should be the same size each time. In this case, this is not correct. The inbox significantly shrinks in size to the actual size of the files within it and not to a fictitious size. It seems you are using compact to shrink the inbox from its phantom size to a smaller size. A better way would be to fix the bug so that this situation does not occur at all, then you would not need to compact the file.

Comment 11

8 years ago
When you delete a mail, since we use the mbox storage format, the mail is only "marked for deletion" and hidden from you. Therefore compact is needed to remove the deleted messages. Removing it directly would often be veeery slow - for large folders it could be many seconds, minutes even, for each deletion!
(Reporter)

Updated

8 years ago
Depends on: 521079
(Assignee)

Updated

8 years ago
Assignee: nobody → bienvenu
(Assignee)

Updated

8 years ago
Duplicate of this bug: 521079
(Assignee)

Comment 13

8 years ago
Created attachment 406358 [details] [diff] [review]
proposed fix
Attachment #406358 - Flags: review?(mkmelin+mozilla)

Comment 14

8 years ago
When i hit "Rebuild index" i get this in the native console

2009-10-15 22:13:03     autosyncActivities      ERROR   onFolderAddedIntoQ: [Exception... "Component returned failure code: 0x80004002 (NS_NOINTERFACE) [nsISupports.QueryInterface]"  nsresult: "0x80004002 (NS_NOINTERFACE)"  location: "JS frame :: file:///opt/moz-objdir/mail/mozilla/dist/bin/modules/activity/autosync.js :: anonymous :: line 196"  data: no]
2009-10-15 22:13:03     autosyncActivities      ERROR   onDownloadStarted: TypeError: syncItem is undefined
2009-10-15 22:13:03     autosyncActivities      ERROR   onDownloadCompleted: TypeError: this._syncInfoPerFolder[folder.URI] is undefined
2009-10-15 22:13:03     autosyncActivities      ERROR   onFolderRemovedFromQ: TypeError: syncItem is undefined
OS: Windows XP → All
Hardware: x86 → All
Target Milestone: --- → Thunderbird 3.0rc1

Comment 15

8 years ago
Comment on attachment 406358 [details] [diff] [review]
proposed fix

Though that seems to happen without the patch too. Looks good to me! r=mkmelin
Attachment #406358 - Flags: review?(mkmelin+mozilla) → review+
(Assignee)

Comment 16

8 years ago
fix pushed.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
This seems like back-end-ish logic that should not be living in the UI code for the folder pane.  Perhaps it could be migrated to a new JS module?
(Assignee)

Comment 18

8 years ago
You're right that this is backend-ish. It could actually have been done in the c++ backend code SetSummaryValid, when the summary is set invalid for an imap or news database. I'm rather hoping that we can fix our offline store story so that this isn't required, or that the decision is postponed until it appears that the offline store is invalid in some way.

Updated

8 years ago
Duplicate of this bug: 530766

Updated

8 years ago
Keywords: fixed-seamonkey2.0.1
You need to log in before you can comment on or make changes to this bug.