Closed Bug 463359 Opened 17 years ago Closed 16 years ago

Compacting Inbox does not reduce the size of Inbox - inbox continues to grow (IMAP, offline folder, Tb 2)

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: smcwilde, Unassigned)

References

Details

(Whiteboard: [confirmed on trunk])

Attachments

(1 file)

Screen captures of TB account settings: server and offline/disk space
112.87 KB, application/zip
Details
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3 Build Identifier: version 2.0.0.17 (20080914) I use Thunderbird configured with an IMAP server - the latest version of Thunderbird was upgraded on my PC in early Sept.... About 3-4 weeks ago - my PC ran out of disk space (down to <10% free space). Found the problem was with my Inbox - it had grown to 22.5GB (more than 1/2 my disk!!) I searched www.mozilla.org/support and googled for problems with compacting inbox and found instructions on how to re-build my inbox. I moved the e-mail in my inbox to a temp folder. Closed TB - Removed INBOX and INBOX.msf. Re-started TB then moved the e-mail messages back to my inbox. A new INBOX got created and the size dropped to about 90,000KB - great news - thought I had the problem solved. Wrong! I have been watching my INBOX coninually grow - it never get smaller. I'm on a project where I get a lot of attachments (.docs mainly) - I clean/delete the messages several times a day and as part of the clean up I routinely compact my INBOX. I have TB configured to mark the messages as deleted (strike through) and when I compact the folder - the messages are removed/disappear.... so the "compact" action seems to be working visually - but my INBOX does not shrink. When I check the file after a compact - there's no change in the timestamp nor the file size. My INBOX is now up to 550,000 KB ... seems messages get added but not removed. The strange thing - is that this is an issue with my INBOX but not with the folders I have on th IMAP server. Compacting individual folders on the IMAP server removed the messages and the files are smaller. I can't find any info in mozilla support that can help. Is it possible something was corrupted when TB upgraded to the newest version. I looked for the nstmp files - searched my PC - but nothing showed up. Any suggestions - help would be greatly appreciated! Reproducible: Always Steps to Reproduce: 1. delete some messages in my INBOX 2. compact my INBOX 3. INBOX file size does not decrease - timestamp does not change Actual Results: INBOX file size does not decrease - timestamp does not change Expected Results: Iwould expect to see a decrease in the size of INBOX.
I see the same problem - using Thunderbird version 2.0.0.17 (20080914) on Windows XP (all current service packs). Regardless of what I delete from my imap account inbox, or move to another folder, the local IMAP inbox folder: C:\Documents and Settings\<user>\Application Data\Thunderbird\Profiles\xxxxx.default\ImapMail\mailbox.server.xx\INBOX never decreases in size. However, the moved messages are no longer visible in or from the inbox (online or offline). I have tried rebuilding the index, and explicitly compacting - neither have any effect. I do not see this with the other predefined imap folders (Sent, Trash, Drafts, junk-mail...) they all seem to happily shrink in size when mail is deleted from them or moved elsewhere ...... This is very annoying for doing backups - I really don't want to (and indeed cannot) backup a 2.5GB file that should only have 23MB of data.... The current behavior simply does not make sense.... This is potentially related to but 428947
Status: UNCONFIRMED → NEW
Ever confirmed: true
Ian - Glad to know I'm not the only one to see this problem. The first time I tried to fix the problem - I moved all the messages in my INBOX to a new folder on my IMAP server. closed TB - Then renamed (for safety) my inbox to INBOX_old and removed my inbox.msf Restarted TB - TB recreated a new INBOX - moved all my messages from the temp folder back in to INBOX. My INBOX shrank dramatically. The messages moved fine but I lost all my tags :-( This week -I found myself needing to re-build my INBOX again b/c it was too large - so this time I tried to see if I could save the tags - I moved all the messages to a local folder (instead of the IMAP server). The messages moved fine - tags were saved. INBOX size before the move was almost 600,000 KB - after the move - same # of messages - INBOX size was 110,000 KB!!! I got curious - your e-mail confirmed that you didn't have a problem with the folders on server - only your INBOX which is local on the PC. I began wondering if the problem was with the local folders.... and it looks like it is!! I use a local "draft" folder - to store message that I haven't finished but don't want to loose (in case my system re-boots or hangs). I looked at the local draft folder - it was 116,000 KB - I thought pretty big for a draft folder. I deleted several old drafts - and the folder size did not shrink. So... I moved all the messages to a new folder. Applied the same drill as I did with INBOX - re-named the draft folder. Went from 116,000 KB down to 148 KB - BIG DIFFERENCE!! Interesting thing - is there is not explicit command to "compact" the draft folder. I haven't tried this on my "Sent" folder - but I suspect I will see the same problem. My guess - there is a bug somewhere in the code that re-builds the folder when messages are deleted - and the disk space is not released/freed-up.
Sounds like a bug in how TB maintains the local imap INBOX file(and potentially also the secondary database, INBOX.msf - although maybe its the latter that makes the former usable :-O ). I did three simple tests to diagnose a bit more. But before discussing these let me note that, using a webmail client I can verify that my remote imap INBOX contains 535 messages (258 unread) and around 50 MB of data. The TB client confirms the same message counts for my imap INBOX. Experiment 1. Use od (octal dump) on the local INBOX file. Od shows the file contains data - it isn't full of zeros or sparse data. Experiment 2. Use zip on the file. Zip compresses the file from 1.5GB --> 850MB, reconfirming that the file is not mostly full of zeros.... Experiment 3. Copy the INBOX, move the copy to the Local Folder area, and read it as a Thunderbird local folder to see what's really in this file. (this assumes INBOX is in mbox format.) Steps to do this were: a. Closed Thunderbird b. Created a copy of the imap INBOX (INBOXCopy) c. Move INBOXCopy to my xxx.default\Mail\Local Folders\ folder d. Re-Started Thunderbird e. Selected to view the "INBOXCopy" local folder. After several minutes of processing, Thunderbird displays thousands of messages in INBOXCopy, many of which I know I'd previously deleted or moved to local folders, and many of which I've never seen before (typically spam that, I suspect, the imap server automatically filters and moves to a junk-mail folder). Also, some of the listed entries are corrupt in some way (e.g. no messge title/ sender/ hearder displayed, or message content window displays raw headers..) So something wondrously weird is going on, clearly something to do with how TB is handling this folder. On the plus side, we never have to worry about losing mail... it is forever buried in the crazy local INBOX file :(
Interesting experiments... even more interesting comment about the corrupted entries. That's what prompted me to re-build my INBOX the 2nd time (~600,000 => 110,000 KB) I was stumped on what was going on - a co-worker suggested I log into the UNIX mail server and use a a UNIX mail command to open my "inbox". I did and discovered one of the messages was corrupted. Hence the re-build from my comments on 11-11. Thought if I removed the corrupted message and re-built my INBOX - the problem would go away. As far as I can tell - it didn't help. Well - yesterday - my INBOX on my PC hit 584,000 KB again - It started at about 380,000 KB - I logged back into my UNIX mail server - the mail file on UNIX was only 117,000 KB. Using a UNIX command to open my mail - found 2 messages corrupted this time. I quit out of the command line. Restarted TB on my PC - and a strange thing happened. TB started to download all my INBOX messages - even though I thought they were already loaded. While it was downloading - I watched in explorer - the inbox grow from 380,000 KB => 584,000 KB. Any idea what's happening??! I have not had time to re-build my INBOX on my PC - but I suspect when I do - it will drop back down to ~117,000 KB. Not sure what's causing the corrupted entries - no error messages or any indications from TB that the messages are corrupted. Also wondering if the corrupted entries are causing the "inbox growing" problem.
Version: unspecified → 2.0
Offline discussions with Sue Wilde indicate that this problem occurs when we have TB configured as follows: In the "Account Settings" --> Offline & Disk Space" dialog option, select (check) the Offline option: [x] Make the messages in my inbox accessible when I am offline This creates the local INBOX file, which then is never purged of 'downloaded for offline use' messages...
I have a very similar problem but it is with the trash box. I can't empty or compact it when it gets full. When I first started using Thunderbird the trash box held at least 1600 deleted messages maybe more. I would go in and delete them and compact the Trash bin. But the capacity started to drop, first under 1000 then closer to 500 and now somewhere between 300 and 500 message I get a trash bin full can not delete any more messages message. I have to go into the directly and manually delete the TRASH directory and start up again. It does all crazy things like showing me unread messages in the trash bin where there are none. Really a serious bug but it is not being addressed. Al (In reply to comment #0) > User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) > Gecko/2008092417 Firefox/3.0.3 > Build Identifier: version 2.0.0.17 (20080914) > > I use Thunderbird configured with an IMAP server - the latest version of > Thunderbird was upgraded on my PC in early Sept.... > About 3-4 weeks ago - my PC ran out of disk space (down to <10% free space). > Found the problem was with my Inbox - it had grown to 22.5GB (more than 1/2 my > disk!!) > I searched www.mozilla.org/support and googled for problems with compacting > inbox and found instructions on how to re-build my inbox. > I moved the e-mail in my inbox to a temp folder. > Closed TB - > Removed INBOX and INBOX.msf. > Re-started TB then moved the e-mail messages back to my inbox. > > A new INBOX got created and the size dropped to about 90,000KB - > great news - thought I had the problem solved. Wrong! > > I have been watching my INBOX coninually grow - it never get smaller. > I'm on a project where I get a lot of attachments (.docs mainly) - > I clean/delete the messages several times a day and as part of the clean up > I routinely compact my INBOX. I have TB configured to mark the messages as > deleted (strike through) and when I compact the folder - the messages are > removed/disappear.... so the "compact" action seems to be working visually - > but my INBOX does not shrink. When I check the file after a compact - > there's no change in the timestamp nor the file size. My INBOX is now up to > 550,000 KB ... seems messages get added but not removed. > > The strange thing - is that this is an issue with my INBOX > but not with the folders I have on th IMAP server. Compacting > individual folders on the IMAP server removed the messages and the > files are smaller. > > I can't find any info in mozilla support that can help. > Is it possible something was corrupted when TB upgraded to the newest version. > I looked for the nstmp files - searched my PC - but nothing showed up. > > Any suggestions - help would be greatly appreciated! > > > > > > Reproducible: Always > > Steps to Reproduce: > 1. delete some messages in my INBOX > 2. compact my INBOX > 3. INBOX file size does not decrease - timestamp does not change > Actual Results: > INBOX file size does not decrease - timestamp does not change > > Expected Results: > Iwould expect to see a decrease in the size of INBOX.
Changed platform and OS settings to all/all as this behavior appears core to TB's behavior, and not something that's OS-specific. (Was originally reported on x86 /XP - I don't have other OS's to test on). 2. Al - I assume you are connecting to an imap mail server,and have the other settings as defined in comment #5?
OS: Windows XP → All
Hardware: x86 → All
(In reply to comment #7) > Changed platform and OS settings to all/all as this behavior appears core to > TB's behavior, and not something that's OS-specific. (Was originally reported > on x86 /XP - I don't have other OS's to test on). > > 2. Al - I assume you are connecting to an imap mail server,and have the other > settings as defined in comment #5? I'm not quite sure what you mean by an IMAP mail server. And no I don't work off line. My mail comes directly from my ISP Cox and it is always a live feed. I have no need to work off line as I am on cable.
Adding IMAP/offline folder in bug summary, for ease of search, to avoid irrelevant comment posting. Problem of "file size growth" was very easily reproduced, but "Compacting Inbox does not reduce the size" was not re-produced with Gmail IMAP folder(both Inbox and non-Inbox folder of offline-use=yes), with Tb 2.0.0.19 on MS Win-XP SP3. (1) Set "offline-use=on" (2) At folder property, Rebuild Index, Download Now => file size=50KB Download Now again => file size is uncahnged(50KB) (3) At folder property, Rebuild Index, Download Now => file size=100KB Download Now again => file size is uncahnged(100KB) (4) At folder property, Rebuild Index, Download Now => file size=150KB Download Now again => file size is uncahnged(150KB) (a) Status of "downloaded or not" is held in ".msf" (b) Status of "downloaded or not" is lost upon Rebuild Index (c) "Download Now" simply adds(appends) downloaded mail data to file for offline use. So it can exceed file size limit of local mail folder file. Sile restriction of 4GB(Win)/2GB(Linux/Mac) is not applicable to the file writing itself. But internal pointer(offset in file) has still 4GB/2GB limit. It may cause some problems upon .msf update, working offline, local search, following compaction etc. (d) X-Mozilla-Status:/X-Mozilla-Status2: header is added to mail data in offline folder file. (5) Compact folder => file size was reduced to 50KB from 150KB. Problem was not receated. (Gmail IMAP treats "flag \Deleted" as "\flag \Deleted + expunge".) (It may be the reason why I can't reproduce problem.) (6) Move all mails from the mail folder to another mail folder. (e) "Flag of deleted mail" in X-Mozilla-Status: header is not set. This is reason why all mails in the file is seen when the folder file is accessed as local mail folder of POP3 or "Local Folders" account. Tb has another problem : Expunge is not issued in some situations. It may be relivant to this bug. (Q1) Can enabling following option be a workaround? - Clean up("Expunge") Inbox on Exit Note: Tb trunk also has problem of "Compacting of folder with offline-use=yes does not reduce the size" (Bug 466730). But trunk's problem is different issue from Tb 2's this bug in many cases, even if completely same symptom. Please be careful in testing with Tb trunk for this bug.
Summary: Compacting Inbox does not reduce the size of Inbox - inbox continues to grow → Compacting Inbox does not reduce the size of Inbox - inbox continues to grow (IMAP, offline folder)
Summary: Compacting Inbox does not reduce the size of Inbox - inbox continues to grow (IMAP, offline folder) → Compacting Inbox does not reduce the size of Inbox - inbox continues to grow (IMAP, offline folder, Tb 2)
I def have file size problems with 3.0 - I am using Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090210 Lightning/1.0pre Shredder/3.0b2pre ID:20090210031755 Tho most of my recent tests were with 20090206 build .. Using Imap I have: 1) INBOX was enormous - compacting, rebuilding index nothing reduced it, turning off offline sync didnt change a thing - file is not removed or changed at all. Only way to shrink it was to by hand delete the INBOX file and rebuild it. 2) Similar problems for other than INBOX 3) Deleting the account in thunderbird did not remove the ImapMail/account-name directory or any of its contents ... to recover space I deleted by hand. Whether this is different or not I don;t know - but I have recovered 20 GiB of space so far .. it is very tedious to do this by hand one mail folder at a time. Seems that compacting is inherent to a single file mail store (mbox) and that this would go away and be much much faster using maildir - Are there any plans to add maildir format for local store ? Problem with deleting by hand is that recovering msg filters and virtual folders can be tedious but deleting the account and restarting it is the only reliable way I can find to recover space.
(In reply to comment #10) > I def have file size problems with 3.0 - I am using >(snip) > Lightning/1.0pre Shredder/3.0b2pre ID:20090210031755 > Tho most of my recent tests were with 20090206 build .. > Using Imap (snip) As I wrote commanet #9, bug with same symptom(Bug 466730) exists for Tb trunk, and is already FIXED. Sorry but I don't know whether Bug 466730 is same problem as this bug for Tb 2 or new problem on Tb trunk. And, as I write Bug 466730 Comment #22, something wrong seems to remain in compaction of offline store of IMAP. gene c, open separate/new bug for problem even after fix of Bug 466730, please.
FYI. Bug 463359 is for same symptom of Tb 2.0.0.18, with mail.server.default.autosync_offline_stores=true for IMAP.
That has happened to me and a colleague. We fixed it by exiting TB, deleting the MSF file, restarting TB, and doing a compact on Inbox.
FYI. Bug 467305 Comment #0 exists for same symptom as this bug for Tb 2.0.0.18. After open of this bug & Bug 467305, Bug 420115 & Bug 466730 were fixed. > Bug 420115 : fixed on 2008-11-21, fixed by Tb 3.0b1 > Bug 466730 : fixed on 2009-01-31, fixed by Tb 3.0b2 After that, Bug 467305 was modified to bug for same symptom even after fix of Bug 420115 & Bug 466730(problem on Tb 3.0b2 or later), by Bug 467305 Comment #2 for Tb 3.0b2. If you encounter symptom of "file size of offline store is not reduced by comapct" on Tb 3.0b2 or later, see Bug 467305, please.
if you see this problem, can you please try beta 2 and report whether the problem is gone (as we think it might be). http://www.mozillamessaging.com/en-US/thunderbird/early_releases/ backup your profile before using the beta
Whiteboard: [fixed in TB3?]
I have been using beta 3 nightly build. Problem was definitely still there as of 1 to 2 weeks ago - as I had to to once again, delete the file and the .msf file. Is this fixed in 3.0b3 nightly as well? If so as of when ? May I please beg and plead for maildir storage format - compacting is no longer needed and these problems just 'go away' .. please please add maildir format. mbox (and outlook pst) have so many more problems. Thanks.
(In reply to comment #16) > I have been using beta 3 nightly build. Problem was definitely still there as > of 1 to 2 weeks ago - as I had to to once again, delete the file and the .msf > file. > > Is this fixed in 3.0b3 nightly as well? If so as of when ? I forget which one. you'd want to run down the bugs already mentioned by wada, and the bugs mentioned within them. please ref also wada's comment about *your* issues - comment 11. > May I please beg and plead for maildir storage format ain't going to happen in TB3 and there are no clear plans for it even beyond TB3. Your comment is best directed to bugs that deal with storage format.
(In reply to comment #16) > I have been using beta 3 nightly build. Problem was definitely still there as > of 1 to 2 weeks ago gene c, same asking as my Bug 467305 Comment #10. Can you open separate bug for "Tb trunk/Tb 3 beta never decreases size by compact folder"? (attaching of IMAP log is preferable)
(In reply to comment #17) > (In reply to comment #16) > > I have been using beta 3 nightly build. Problem was definitely still there as > > of 1 to 2 weeks ago - as I had to to once again, delete the file and the .msf > > file. > > > > Is this fixed in 3.0b3 nightly as well? If so as of when ? > > I forget which one. you'd want to run down the bugs already mentioned by wada, > and the bugs mentioned within them. > > please ref also wada's comment about *your* issues - comment 11. gene, thanks very much for testing. not sure what I was thinking in comment 17 (beta 2?). anyways, afaict no major fixes to compact since about 2009-01-14 (bug 466730 compact all offline stores). So I'm going to say we see this on trunk. Wada, is this the same as bug 487992? Gene, can you reproduce using the steps in that bug? Would be good to get this fixed for Thunderbird 3.
Severity: normal → critical
Component: General → Database
Product: Thunderbird → MailNews Core
QA Contact: general → database
Whiteboard: [fixed in TB3?] → [confirmed on trunk]
Version: 2.0 → Trunk
To Sue Wilde(bug opener, for Tb 2.0.0.17) and Ian Graham(Comment #3 poster): Can you produce problem with latest Tb 2.0.0.21?
The problem is still present in TB 2.0.0.21. The criteria to make this happen are (I believe): * IMAP server * Offline & Disk Space settings: [*] Make the messages in my Inbox available when I am working offline.
(In reply to comment #22) > The problem is still present in TB 2.0.0.21. What delete model do you use? (Server Settings/When I delete a message:) When you request "Compact Folder" after delete of mails, is EXPUNGE command issued by Tb? Do you enable autosync? > mail.server.default.autosync_offline_stores=true As seen in Bug 466730 Comment #21, a case of same external symptom(offline-store doesn't shrink by Compact Folder) was fixed by 2009/1/18 build. (Sorry but I don't know bug 466730 is for what case/what condition.) Can you check with 2009/1/18 build? > http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2009/01/2009-01-18-03-comm-1.9.1/ > You can test by UNZIP only, if you use win32.zip build.
Ian Graham, you referred to Bug 428947 in Comment #1. Does it mean you tested with Gmail IMAP?
I (Ian) have not tested with gmail IMAP. Please note that this does not appear to be an IMAP server problem. I have separate webmail access to the IMAP server, and have used that to verify the files are deleted from the server: they are simply retained in TB's local INBOX file after I select to delete them. I will attached a ZIP file (TB settings.zip) containing screen captures showing my configuration settings for of all the (I think relevant) TB server configuration panels. I have checked Options --> Config Editor and there is no preference called mail.server.default.autosync_offline_stores Where would I look to find that?
ZIP file containing screen captures of the my relevant TB configuration settings.
My delete model is to delete files manually. I first select files I want to get rid of and move them to the Trash folder. When I am done I select the Trash folder and use the right mouse menu to "Empty Trash."
(In reply to comment #25) > Please note that this does not appear to be an IMAP server problem. I never say IMAP server problem. I merely say "If Gmail IMAP, specal care is required." (See 499630. See also Dependency tree for Bug 402793) > I have checked Options --> Config Editor and there is no preference called > mail.server.default.autosync_offline_stores > Where would I look to find that? I asked you about it simply because bug opener of Bug 467305(for Tb 2.0.0.18) says "he sets mail.server.default.autosync_offline_stores=true". I don't know about it of Tb 2, but if you can't see the entry in prefs.js, it means you don't set it true. No need to worry about it. Note: I can see next with Tb trunk(Tb 3). > mail.server.default.autosync_offline_stores = true (default value) > mailnews.offline_sync_mail = false (default value) But I can see next only with Tb 2.0.0.22. > mailnews.offline_sync_mail = false (default value)
(In reply to comment #26) > Screen captures of TB account settings: server and offline/disk space I saw next screen shot, but I couldn't find any IMAP folder for which offline-use option is enabled in the screen shot... > imap files for offline use settings.png
Ian Graham, what do you set in next prefs.js entry? > mail.server.serverX.autosync_offline_stores = true/false > use_status_for_biff = true/false > ( default of both is probably false. ) > ( See http://kb.mozillazine.org/Offline_folders#AutoSync ) How about next? (It's possibly newly used by Tb 3.) > mail.server.serverX.offline_download = true/false > (default=true, if Tb 3. I don't know for Tb 2.) (Correction of comment #29) Oh, this bug was for Tb 2. Ignore my comment #29, please.
My prefs.js contains no lines / settings for either *.autosync_offline_stores or use_status_for_biff. I do have the preference setting: mail.server.server5.offline_download and yes, server5 is my email IMAP server.
OOps, and yes the setting is 'true' user_pref("mail.server.server5.offline_download", true);
(In reply to comment #25) > I have checked Options --> Config Editor and there is no preference called > mail.server.default.autosync_offline_stores (In reply to comment #31) > My prefs.js contains no lines / settings for either > *.autosync_offline_stores or use_status_for_biff. Comment #29 has come back... AFAIK, When auto-sync=off(default), "Compact Folder" does do nothing on offline-store file if offline-use=on is not set in Folder Properties. Ian Graham, what will happen when you do next with Tb 2? 1. Enable offline-use option of an IMAP mail folder(say XXX). Total mail data size = M1 MB. File size of XXX at this step = S1 MB. (garbage) 2. "Download Now" => File size of XXX == (S1 + M1) MB 3. "Compact Folder" => File size of XXX becomes M1 MB?
No compacting options decreased the size of hte folder. According to my webmail account, the size of the messages on the IMAP server folder is 2.1 MB Initial TB folder size (Inbox): 11.5 MB Attempt 1: Select the Inbox folder and use right menu to select "compact". Size afterwards: 11.5MB (no change) Attempt 2: go to Tools -> Account Settings -> Offline & Disk Space -> Select folders for offline use and select the Inbox folder. Next shut down and restart browser, and download new files. Select to compact the folder (as per attempt 1) Size afterwards: 11.5M (no change)
(In reply to comment #34) > No compacting options decreased the size of hte folder. Don't confuse "issuing EXPUNGE command" part of Compact and "remove data in offline-store" part of Compact, please. > Attempt 2: go to Tools -> Account Settings -> Offline & Disk Space -> Select > folders for offline use and select the Inbox folder. > Next shut down and restart browser, and download new files. > Select to compact the folder (as per attempt 1) > Size afterwards: 11.5M (no change) Did you do "view some mails or all mails" or "Download Now" after enabling offline-use for Inbox? Because auto-sync=off in your case, no mail data is downloaded automatically, then there is no data about "mail data in offline-store file" in .msf. So "no increase of offline-store file size in your test" and "no decrease of offline-store file size by Compact" is not surprizsing.
I selected "Get all new messages." then deleted some large messages, and then selected compact. The mailbox size increased, and did not decrease as it should've.
(In reply to comment #36) > I selected "Get all new messages." Because auto-sync=off in your case, I think "Get all new message" downloads standard mail header only. > then deleted some large messages, and then selected compact. Don't step into complex case(delete of mails has relation to phenomenon). Check next case which I requested in Comment #33 first, please. 0. Disable offline-use option of an IMAP mail folder(say XXX), and shutdown Tb 2, restart Tb 2. 1. Enable offline-use option of an IMAP mail folder(say XXX). Total mail data size at IMAP server = M1. File size of XXX at this step = X1. (garbage for Tb) 2. "Download Now" => File size of XXX == (X1 + M1)? 3. "Compact Folder" => File size of XXX becomes M1? > The mailbox size increased, and did not decrease as it should've. Please report offline-store file size at each step. I saw funny phenomenon like next with Tb 2 with Gmail IMAP several times. Total mail data at IMAP server(Gmail IMAP) : M1 Initial offline-store file size : X1 Compact : X1 -> X2 (X1>X2) Rebuild-Index, Download Now : X2+M1 Compact : M1 Value of X1 was different from my expectation. Delta of (X0-X1) looks garbage due to deleted mails which were auto expunged by Gmail IMAP. It's may be Gmail IMAP particular issue, but I'm not sure. (See Bug 499630 for phenomenon with Tb trunk and Gmail IMAP) It's reason why I request you to check simplest case(offlin-use=NO => offline-use=Yes, Download Now, Compact). In this case, "delete of mail after offline-use=Yes" is not involved, and "Rebuild-Index" is not executed yet.
one thing about compacting offline stores is that we fix "corrupted" messages in the offline store, so that if an offline store is corrupt, we will refetch the messages from the imap server and copy them to the compacted offline store. In some cases, this could actually grow the offline store, I suppose, if the corruption was such that we didn't have all the message. But this should be a one time thing, assuming the corruption doesn't keep happening.
I tried the test recommended on comment 37. 1. I disabled offline use. Shut down and restarted Restarted TB. Inbox size: 29.7 MB 2. Enabled offline use for Inbox. Inbox size: 29.7 MB 3. Get all new messages from Server. Inbox size: 29.7 MB 4. Compact Inbox folder Inbox size 29.7 MB I have also viewed the Inbox file with sed or cat (Cygwin. Doing so I can verify that my Inbox file contains messages that were previously deleted (and that were in fact deleted from the IMAP server).
(In reply to comment #39) > 3. Get all new messages from Server. Inbox size: 29.7 MB Are you testing to analyze phenomenon of "offline-store file size doesn't increase by Get All new messages"? Do "Folder Properties"/Offline/"Download Now" instead of "Get All new messages". Test scenario is emulation of "offline-store file size increase" due to re-download in a "corrupted" message case("we didn't have all the message" case). In the scenario, condition is "we have no data in local offline-store file for any mail on IMAP server". But because mail data is not really "corrupted", re-download is not done automatically until user tries to view the mail. So we emulate re-download of all mails by "Download Now".
I did the test you suggest. Indeed, "download now" does download additional file data. But this is not the concern (I think) related to this bug report. The bug here is simply that the size of the local IMAP INBOX file never decreases, but only increases. There seem to be many issues / probelms related to this basic concern. The ones I see, and which I have verified offline with Sue Wilde, are: a) compacting the Imap Inbox does not reduce the size of the Imap INBOX file (the offline store). b) deleting a message from the Inbox does not remove that message from the local Imap INBOX file, (it is removed from the Imap server). c) if I delete a message from the Inbox the message disappears from the messages listed in the TB message pane - but as per (b) the message content is not purged, but remains in the Imap INBOX file (verified by inspection of the file using cygwin utils like tail, sed, etc. )
(In reply to comment #41) > I did the test you suggest. Indeed, "download now" does download additional file data. After that, "Compact Folder" of Tb 2.0.0.22 reduced offline-store file size? Or "Compact Folder" of Tb 2.0.0.22 didn't reduce offline-store file size? How about second "Compact Folder" of Tb 2.0.0.22 after restart of Tb? When you did "Compact Folder", did Tb 2.0.0.22 issue "EXPUNGE" command? (See https://wiki.mozilla.org/MailNews:Logging) > The bug here is simply that the size of the local IMAP INBOX file never decreases, but only increases. In test case very similar to your case, Tb 2.0.0.22 and Tb trunk 2008/11/12 build reduced offline-store file size. See Bug 495862. In test case of Bug 499630(with Gmail IMAP), Tb 2.0.0.22 and Tb trunk 2008/11/22 build descreased offline-store file size. I can't believe "Tb 2.0.0.22 never decreases offline-store file size in any situation"(you are saying so). Tb 2.0.0.22 doesn't decrease or fails to decrease on what condition?
There are some very odd things happening. Let me run through the scenarios I tried, and perhaps you can understand. I apologize for the length here. It does look, though, as if compacting does work - but the scenarios highlight some other problems that make this hard to see or even use. Note that I am using my webmail client to check if messages are being expunged from IMAP server (not using the TB logging facility). Some other background. My IMAP server has around 700 messages in the Inbox. In the following I track the size of the TB local Imap INBOX file, in MB. a) before starting TB: 19.3 MB b) Launch TB - it downloads 40 messages, including one with a 2MB attachment. After TB is running and has finished doanloading: 21.9 MB c) I then select Inbox -> Properties -> Offline -> Download All Folder size is still 21.9 MB d) I now select to "Compact" the folder. Local INBOX file size starts to grow (22.2 MB, 22.5 MB, 23 MB, 24 MB .... This continues for several minutes, and peaks at an INBOX file size of about 32 MB. It then dropped to 21 MB. So it is slightly smaller. e) Using TB, I then moved the message with the circa 2MB attachment to the trash folder and then selected 'empty trash.' Local IMAP folder size: 21MB (unchanged). f) Used webmail client to check server: the messge with the large attachment is still there: marked as flagged for deletion, but still listed in the inbox. g) shut down TB client (file --> Exit) h) rechecked with Webmail client -- the message with the large attachment is still listed in the inbox, marked as flagged for deletion. i) restarted thunderbird. File size is 21MB (no change). TB does not list the file I deleted (with the large attachment), and the Trash folder is also empty. But the message is still shown when I view the inbox using my webmail client. (as per (f) ) . j) I used my webmail client to delete the large file. k) use 'get all new messages' on TB (for Inbox). No change in INBOX file size. l) Inbox -> Properties -> Offline -> 'download now" . No change in INBOX file size (21MB) m) Inbox --> Compact. As with (d), INBOX folder size starts to grow, and over several minutes reaches circa 30 MB. It then shrunk back to 18.9MB - about right for compacting out / purging the deleted file. Yay! Next I repeated the experiment, starting from scratch but skipping the 'download now' and compact steps. a) Start up TB and webmail client. TB IMAP folder size is 18.8MB b) used TB to select 10 messages and move to trash folder. c) refreshed webmail view of inbox - this no longer displayed the 10 messages I moved to the TB trash folder. This is different from the previous case, where wemail showed them but as 'marked for deleetion' d) emptied trash folder. TB IMAP folder size is 18.8MB e) select to compact TB Inbox. Again, a very slow process during which the INBOX file first grows, then shrinks t0 18.5MB - which is about right given I deleted circa 300 K of messages using my webmail client. So compacting does work it seems! Some thoughts: 1. Somehow the 'download now' or 'compact' commands may be interfering with issuing an EXPUNGE command to the IMAP server. If I start TB and don't use either of these functions, then deleting mail on TB (moving it to the trash, purging trash) seems to automatically remove it from the IMAP server (i.e. EXPUNGE works). 2. It does not make sense (or seem very efficient) that TB takes 4-5 minutes to compact a 20MB INBOX. For users with 400 MB inboxes, this would imply compaction would take an hour ... not really acceptable performance.
I tried compacting my inbox with Tb 2.0.0.22 and I can confirm it works (and before it didn't). I checked the file size before and after the compacting and I noticed a significant reduction in size. I'm on Ubuntu 9.04 and my IMAP-server is Dovecot (with Maildir storage).
(In reply to comment #44) > I tried compacting my inbox with Tb 2.0.0.22 and I can confirm it works (and before it didn't). Marcel van Beurden, do you remember Tb 2's version(s) of "before it didn't"? Do you set true in next settings? > mail.server.default.autosync_offline_stores = true > mail.server.serverX.autosync_offline_stores = true
(In reply to comment #43) Ian Graham, can we interpret your comment #43 as "Your comment #22 was wrong. This bug is WORKSFORME at least with Tb 2.0.0.21"? > Comment #22 From Ian Graham 2009-06-24 19:14:48 PDT > The problem is still present in TB 2.0.0.21. Or you can see phenomenon of "compact or compact folders of Tb 2 never reduces offline-size file size" in some situations? (See Bug 499630 for such issue on Tb trunk with Gmail IMAP in special situation.)
Yes, I agree this bug is WORKSFORME - the bug appears to be fixed. However, since for me it too 4-5 minutes to compact a 20MB INBOX, there will likely still be problems for users with large INBOX files (mine is quite small, in fact). If compacting time is proportional to the size of the INBOX, then for large inboxes compacting may never complete during an active TB session, in which case compacting may never get done .....
When I had the problem I had the version that was current in March (then I subscribed to this bug). I don't have neither settings: > mail.server.default.autosync_offline_stores = true > mail.server.serverX.autosync_offline_stores = true
To all problem reporters for Tb trunk(next Tb 3): Read Bug 499630 Comment #23, and never add comment relates to Tb trunk(next Tb 3) in this bug any more, please.
(In reply to comment #47) > If compacting time is proportional to the size of the INBOX, then for large inboxes compacting > may never complete during an active TB session, in which case compacting may never get done ..... You are right. At least one developer has the issue under consideration. See Bug 499630 #23, please.
(In reply to comment #48) > I don't have neither settings: > > mail.server.default.autosync_offline_stores = true > > mail.server.serverX.autosync_offline_stores = true Your case is also "auto-sync=off" case. According to comments by Ian Graham and Marcel van Beurden, this bug is already WORKSFORME(problem really existed with 2.0.0.17 as you said, but problem won't occur with Tb 2.0.0.21. i.e. The problem was fixed by unknown patch). So closing this bug for Tb 2 as WORKSFORME. To Sue Wilde(bug opener): If my "closing as WORKSFORME" will be found to be wrong, re-open bug, please.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: