Closed Bug 327674 Opened 19 years ago Closed 16 years ago

compact imap INBOX makes local copy of inbox larger when compaction does not complete successfully.

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: snoeyink, Unassigned)

Details

User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/417.9 (KHTML, like Gecko) Safari/417.8 Build Identifier: version 1.5 (2051201) The local copy of my Imap INBOX usually does not successfully compact. Instead, it will leave behind nstmp folders of the size that a successfull compaction would leave, and will _grow_ the INBOX by something like that amount. I walk through a long example scenario below in the hope it will aid debugging. I have about 700 folders and subfolders, all of which are subfolders of INBOX on my imap server, imap.unc.edu. I get lots of mail and spam, so compacting the imap folder seems to often be interrupted. If I delete everything, then new local copies are made from the imap server, so it is only the local compaction that is messed up. Reproducible: Always Steps to Reproduce: Rather than steps, let me give a transcript of many things I've tried this morning while watching folder sizes. I've tried many combinations of on-line/offline, etc. Restarting and compacting first thing seems to help the compaction complete. Had problem with all versions up to and including 1.5. Actual Results: Thunderbird 1.5 (20051201) on a Win XP (Tablet edition) SP2 machine. IMAP INBOX on imap.unc.edu (checked "expunge" on exit) probably 500+ folders and subfolders of INBOX. personal namespace "INBOX." public "" (checked allow server to override) checked Make the messages in my Inbox available when I am working offline. Initially C:\Doc...\Application Data\Thunderbird\Profiles\...\ImapMail\imap.unc.edu contains no nstmp files and INBOX is 11MB, INBOX.msf is 157KB. I move/delete some mail from the INBOX and right click to select Compact the Folder. (Checking with Mail on my Mac shows that the online INBOX has been expunged.) ->nstmp file of 9MB created; INBOX unchanged. Compact sent mail folder, just to see that it can compact other folders ->succeeded: Sent Mail now 200KB, but nstmp-1 of 15KB remains. Deleted and moved a few other mail messages. ->I now see an nstmp-2 of 9MB that is time-stamped between Sent Items , and INBOX is up to 15MB. Compact INBOX ->nstmp-3 created 5.8MB. INBOX grows to 16MB. I wait 5 min before clicking in Thunderbird, since that is the difference between the timestamps of the two 9MB nstmp files. (Checking the mac again while this happens, shows the online INBOX is again clean.) ->No change in local files, but thunderbird is still showing the message "Compacting folder Inbox..." I know that Tbird does not always clear messages, but I wait to give it a chance. No CPU or memory activity on Thunderbird in Task Manager, so I go offline to make sure new mail doesn't interrupt compaction. ->Compact folder is grayed out for all imap folders, but the message "compacting folder Inbox..." remains. I compact my local folders to see that that works, and clear the message. ->No change to imap local files. I go back online, and get two new messages ->INBOX now at 16.1MB, Some .msf files are updated, a few others get a -1.msf copy (strange, that includes folders I haven't touched in months, and some .msf folders are neither updated nor copied.) I delete the new new messages and compact INBOX. ->nstmp-4 of 5,791KB created, INBOX grows to 21,256KB, adding a little over 5MB (again.) I compact INBOX again. ->nstmp-5 of 5,789KB created, INBOX grows to 25,413, again adding about 5.15MB. I exit Tbird -> INBOX now 26,321KB. Restart Tbird -> INBOX.msf & two other .msf updated; all else unchanged. Compact INBOX ->wow! it actually compacted INBOX to 5,792KB, first time I've ever seen that happen. I guess I hit a long enough window with no incoming mail/spam to actually have it complete. I delete three more messages totaling 10KB, and compact again. ->Further compacts INBOX to 2,401KB. Strange. New message arrives, I delete it and compact ->Successfully compacts to 5,789Kb. Ok, I actually have seen it complete, now, but I've wasted at least a 45 min, and still have to clean up the nstmp files. I'd rather not have to do manual maintenance this way... Expected Results: Ideally, compaction would successfully complete (and do subfolders, too, which it doesn't seem to do.)
So far my only solution for this situation has been to delete the local copy of my inbox every couple days, and download it again from the IMAP server.
Jack in comment #0: > User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) > AppleWebKit/417.9 (KHTML, like Gecko) Safari/417.8 > Build Identifier: version 1.5 (2051201) Jack, this is a rather odd version string for a windows user :) do you see the problem with thunderbird v2?
Version: unspecified → 1.5
I have not seen this problem in recent months with current versions of 1.5 or 2.0.
My local cache of INBOX messages from my IMAP server still grows on compaction (by roughly the size of the compacted file) before shrinking when compaction completes. Since compaction completes more reliably than it used to, this is less of a problem than it was when each piece of incoming spam would cause it to stop compacting.
Assignee: mscott → nobody
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Version: 1.5 → unspecified
I have found today that the local INBOX of an IMAP account here is more than 200 MB in size and contains 2960 messages (counted using grep "^From:" INBOX | wc -l in my shell on Linux), while the actual IMAP folder only contains 31 messages of approx. 5 MB total size. This is with relatively current SM from trunk: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1pre) Gecko/20090623 SeaMonkey/2.0b1pre (This is the bug that is closest to the effect I'm seeing. Should I open a new one?)
(In reply to comment #5) > Gecko/20090623 > (This is the bug that is closest to the effect I'm seeing. Should I open a new one?) Phenomenon you explained is very normal, if "compact folder" is not executed yet. Peter Weilbacher, if phenomenon even after "Compact Folder", and if other than Gmail IMAP, see bug 495862 for issue even with newest trunk, and provide detailed/sufficient data to analyze problem at bug 495862, please. If phenomenon even after "Compact Folder" and if problem with Gmail IMAP, see Bug 499630.
Yes, thanks for the hint. bug 495862 seems to be the right one. I'll add my info there.
To Jack Snoeyink(bug opener): Does your problem still occur with newest Tb 2.0.0.22? If yes, completely same phenomenon as comment #0? Or phenomenon has varied?
It looks like thins have changed. Compacting completes more reliably, so I had to spend some time trying to edit folders that were being compacted or quit Tbird while compacting to see if I could reproduce the original effect. I have not seen it grow the folders when compaction is interrupted; although it does seem to only grow and not shrink the associated .msf file. E.g., I deleted all my imap mail folders from ...Tbirdprofile\ImapMail\imap.unc.edu, then started Tbird and did Download now on several folders including Sent Items (Retention for that on is "on"). I moved a couple of months of email into an archive; both folder and .msf file increased. (510K->578K) I compacted; the folder size decreased, but the .msf remained the same. Quit Tbird, restarted, .msf touched but unchanged. Compacted folder (this time without viewing headers); still no change. Quit Tbird, delete .msf, restarted, and visited to download headers: 278K Click download now and .msf is 329K. So, if you did something to fix the folder compaction, please see if the same thing will apply to .msf file compaction and we should be golden...
(In reply to comment #9) > It looks like thins have changed. >(snip) > I have not seen it grow the folders when compaction is interrupted; Does it mean this bug is already WORKSFORME for you? > although it does seem to only grow and not shrink the associated .msf file. >(snip) > So, if you did something to fix the folder compaction, please see if the same > thing will apply to .msf file compaction and we should be golden... It's not surprising. Imagine next case. 0. Assume XX mails(UID=1 to XX) in a mail folder, total mail data = YY MB. .msf has data for XX mails(UID=1 to XX) only. File size of offline-store = YY MB. 1. Rebuild-Index, Download Now => .msf still has data for XX mails(UID=1 to XX) only => offline-store holds data of 2*YY MB. (See Bug 487992) 2. Rebuild-Index, Download Now => .msf still has data for XX mails(UID=1 to XX) only => offline-store holds data of 3*YY MB. (See Bug 487992) 3. Compact Folder => .msf still has data for XX mails(UID=1 to XX) only => offline-store now holds data of 1*YY MB only. (If Tb 2. See Bug 499630 for Tb trunk issue in this case) As seen in above, file_size of .msf and file_size of offline-store is independent. If you see (a) infinite .msf file size increase or (b) too large .msf file size compared to number of (active + deleted but not_expunged yet) mails, open separate bug, please. Anyway, closing as WORKSFORME based on your answer. Re-open if original problem occurs again with recent or newer Tb 2, please. If you'll experience same problem as original(comment #0) with Tb trunk(next Tb 3), open separate bug for it, please.
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
Thanks. Will do as you suggest.
FYI. Remaining garbages of nstmp,nstmp-1,... is easily observed using Tb 2 by; 0. Delete Model: Just mark as deleted 1. Shift+Delete of some mails in a folder (say XXX) => marked with read cross 2. Compact => Shift+Delete'd mails disapper from thread pane. (expunge is issued) 3. Open file of XXX by text editor who opens file with "Write Mode". 4. Compact => "Delete of XXX" and "rename of nstmp(-N) to XXX" fails. => garbage of nstmp(-N) remains. 5. Repeat 1 - 4 => nstmp-2,nstmp-3,...(I guess max=9999. What happens after it?) If .msf is corrupted, rebuild-index occurs, and it'll force re-download of all mails. I guess this is the cause of Comment #0. Fortunately, Tb 2.0.0.22 didn't corrupt .msf in above case. Because auto-sync is defaulted by Tb 3, and because large IMAP folder is very popular recently, the garbages can become severe issue for users who use auto-backup. (See Bug 494706 and Bug 498814). If you think the garbage of nstmp,nstmp-1,...,nstmp-N is also bug of Tb, open separate bug for it, please.
You need to log in before you can comment on or make changes to this bug.