Last Comment Bug 725810 - Automatic Compact Folders Not Working
: Automatic Compact Folders Not Working
Status: VERIFIED FIXED
[gs]
: regression
Product: MailNews Core
Classification: Components
Component: Backend (show other bugs)
: 11
: x86 Windows XP
: -- normal (vote)
: Thunderbird 15.0
Assigned To: David :Bienvenu
:
Mentors:
https://getsatisfaction.com/mozilla_m...
Depends on: 752295
Blocks: 402392 710056 755993
  Show dependency treegraph
 
Reported: 2012-02-09 14:03 PST by L.A.R. Grizzly
Modified: 2012-06-24 09:05 PDT (History)
8 users (show)
vseerror: in‑testsuite?
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
fixed
fixed


Attachments
proposed fix (2.57 KB, patch)
2012-05-16 14:45 PDT, David :Bienvenu
neil: review-
Details | Diff | Splinter Review
What I was thinking of (2.11 KB, patch)
2012-05-19 07:39 PDT, neil@parkwaycc.co.uk
mozilla: review+
mozilla: approval‑comm‑aurora+
mozilla: approval‑comm‑beta+
Details | Diff | Splinter Review

Description L.A.R. Grizzly 2012-02-09 14:03:26 PST
User Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0a2) Gecko/20120207 Firefox/12.0a2
Build ID: 20120207042017

Steps to reproduce:

I started to notice a lag in my Inbox. Referenced here: http://forums.mozillazine.org/viewtopic.php?p=11707097#p11707097 Once manually compacted (it was over 10mb), the lag was corrected, but setting the automatic compact to 1mb doesn't seem to work.


Actual results:

Setting my folders to automatically compact at 1mb doesn't work. In fact, I don't think it was working when I had it set to 10mb (because I developed the lag problem mentioned here). My Inbox.* file has recently grown to 2.5mb and it still hasn't compacted (even when the Inbox is empty). I tried a new, clean profile with the same results. I transferred my "mail" directory from my TB profile to my SeaMonkey profile and SM compacts it without a problem.


Expected results:

Folders should automatically compact when going above 1mb.
Comment 1 Wayne Mery (:wsmwk, NI for questions) 2012-02-09 15:50:24 PST
Thunderbird and SM both use the same 
compact related code is backend/mailnews core, and TB and SM share that code. So it would be pretty weird for them to act different.
Comment 2 L.A.R. Grizzly 2012-02-09 16:58:16 PST
(In reply to Wayne Mery (:wsmwk) from comment #1)

That's why I suspect there's something awry in the latest TB build(s).
Comment 3 Wayne Mery (:wsmwk, NI for questions) 2012-02-10 04:32:51 PST
tested on vista with 20120209 build. works fine for me.
Comment 4 L.A.R. Grizzly 2012-02-10 12:00:55 PST
(In reply to Wayne Mery (:wsmwk) from comment #3)
> tested on vista with 20120209 build. works fine for me.

Maybe it's a WinXP thing. I'm now using my test profile and the Inbox.* file is now at 3.7mb. I set mail.purge.ask to "true" so I should get a prompt when TB wants to automatically compact. I've had no prompt yet and the file just keeps growing. I have the automatic compact set to 1mb. No add-ons or anything have been added to this test profile. I'm going to try copying my mail folder into SM again to see if SM compacts it. I'll post back with the results.
Comment 5 L.A.R. Grizzly 2012-02-10 12:10:13 PST
I just copied my mail folder from TB to SM and SM compacted the folders within 1 minute. There's definitely a problem with TB.
Comment 6 L.A.R. Grizzly 2012-02-10 12:13:42 PST
I now copied my mail folder from my my regularly used profile (not the test profile) over to SM and SM compacted the folders immediately upon receiving a message.
Comment 7 L.A.R. Grizzly 2012-02-10 21:27:53 PST
After playing around with this problem, I've found out that my Automatic Compact pref doesn't stick. I set it to 1mb and later I find that it has changed back to 20mb. I can't seem to find out what action is making it revert back. Opening and closing TB doesn't always do it. I checked my prefs.js file and user_pref("mail.purge_threshhold_mb", 1); is set to 1mb, but if I check it through about:config, it's says 20mb (as well as in the UI)! If I reset it to 1mb in the UI, I find it changed back to 20mb later on in the day.
Comment 8 L.A.R. Grizzly 2012-02-10 21:52:30 PST
I think I found the trigger. If I set it to 1mb, it reverts back to 20mb as soon as I delete any message from my Inbox, Drafts, or Sent folders.
Comment 9 WADA 2012-02-13 01:32:45 PST
(In reply to Tim from comment #8)
> it reverts back to 20mb as soon as I delete any message from my Inbox, Drafts, or Sent folders.

Setting of thresholdMB is done at here(migration from threshold to thresholdMB).
http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#2037
> 2037     prefBranch->GetIntPref(PREF_MAIL_PURGE_THRESHOLD_MB, &thresholdMB);
> 2038     prefBranch->GetBoolPref(PREF_MAIL_PURGE_MIGRATED, &thresholdMigrated);
> 2039     if (!thresholdMigrated)
> 2040     {
http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#112
> 112 #define PREF_MAIL_PURGE_MIGRATED "mail.purge_threshold_migrated

Did you corrupt mail.purge_threshold_migrated in your prefs.js?
If mail.purge_threshold_migrated=false or is not defined, mail.purge_threshhold is also reffered and is used because migration from threshold to threshold_MB is needed. What value is set?

Did you restart Tb after setting change of mail.purge.ask to "true"?
Comment 10 WADA 2012-02-13 01:35:08 PST
FYI.
Following KB article is useful if you are testing "Compact" of Tb.
http://kb.mozillazine.org/Compacting_folders#Customizing_compacting
Comment 11 L.A.R. Grizzly 2012-02-13 11:52:29 PST
(In reply to WADA from comment #9)

I completely uninstalled TB (including my profile folders). I reinstalled TB and set up a new profile from scratch. I didn't import or copy any files form the old profile (except my personal message filter file, it's quite extensive). It may take a while to see if the problem is gone.

My mail.purge_threshold_migrated is set to "true".

I'll post back when I've made a conclusion. Thanks for your help.
Comment 12 L.A.R. Grizzly 2012-02-16 11:12:45 PST
Well, it's been a couple of days and my TB Inbox is now above 3mb. No automatic compacting has occurred. I copied all of my mail files over to SeaMonkey and SM immediately compacted them! I now do truly believe that there is a bug in Thunderbird (current build: 20120214030022)
Comment 13 Wayne Mery (:wsmwk, NI for questions) 2012-02-17 03:35:34 PST
bienvenu might suggest a path ...

iirc this gets triggered off of a timer. i think you'd have to run tbird in dbugger to determine why compact isn't being done
Comment 14 David :Bienvenu 2012-02-17 08:01:19 PST
for local folders, compact usually happens after a delete, when more than XX MB are being wasted, but no more than once an hour. Perhaps something is horked about your TB profile so that nsMsgDBFolder::HandleAutoCompactEvent bails out before it does anything. Or perhaps we're not recording the expunged bytes in your inbox .msf file (don't know why that would be). I suppose you could send me the prefs.js file from your user profile dir (do help, trouble shooting, and click the show folder button to see where your prefs.js file is). I can't think of any extra logging that would help show where this is going wrong.

It doesn't happen on a timer, but it is run by an event, e.g., the delete causes us to post an event to the event queue and processing that event runs the compact - this is just to unwind the stack before doing the compact.
Comment 15 L.A.R. Grizzly 2012-02-17 11:22:57 PST
(In reply to David :Bienvenu from comment #14)

> I suppose you could send me the prefs.js file from your user profile dir (do
> help, trouble shooting, and click the show folder button to see where your
> prefs.js file is). I can't think of any extra logging that would help show
> where this is going wrong.

I sent you a copy of my prefs.js and user.js files to the email link in your name in the above Comment. Post back if you need them sent elsewhere.
Comment 16 David :Bienvenu 2012-02-17 13:15:48 PST
I did not get the prefs.js file - it might have been quarantined, but I won't know that until tomorrow
Comment 17 L.A.R. Grizzly 2012-02-17 13:43:59 PST
(In reply to David :Bienvenu from comment #16)
> I did not get the prefs.js file - it might have been quarantined, but I
> won't know that until tomorrow

I sent them zipped. The file is prefs.zip. I'll resend them unzipped.
Comment 18 Wayne Mery (:wsmwk, NI for questions) 2012-02-25 07:15:02 PST
see bug 558387
Comment 19 L.A.R. Grizzly 2012-02-25 17:23:24 PST
(In reply to Wayne Mery (:wsmwk) from comment #18)
> see bug 558387

That bug is quite old and pertains to Offline Compacting. For me, compacting was working until just before the update to v12.0a2. It hasn't compacted since. My Inbox is now over 32mb. I found this add-on (see below) which works fine. I can just set the timer to compact the folders at a specific time. I'll have to use it until the bug gets fixed.

https://addons.mozilla.org/en-US/thunderbird/addon/xpunge/
Comment 20 Wayne Mery (:wsmwk, NI for questions) 2012-03-09 05:35:14 PST
bienvenu, do you need more than the prefs?


other compact bugs
bug 492344 Once folder update(e.g mail copy) occurs without explicit folder open when outdated msf condition exists, "Compaction of local mail folder" is never executed, until next mail delete happens.
bug 498185 MailDB(.msf) is corrupted(Rebuild-Index is invoked), if "Compact Folder" is invoked on "copy target folder" while "copy to folder by message filter" is running. "Compact Folder" itself silently fails.
bug 137210 Compact local folder doesn't compact if we are rebuilding msf file
Comment 21 David :Bienvenu 2012-03-09 08:48:37 PST
(In reply to Wayne Mery (:wsmwk) from comment #20)
> bienvenu, do you need more than the prefs?
> 
Apparently so - Tim's prefs.js is very vanilla. One thing I noticed is that his check for new mail interval is very small. I suppose it's remotely possible that we're always busy getting new mail, which does prevent compaction from happening. If loading and applying filters takes a long time, again, it's remotely possible.

Tim, if you go offline, and delete a message, do you get the compact prompt? Oh, and I assume you have deleted messages! Do you have a lot of filters that move mail from your inbox to other folders, on arrival? Those messages shouldn't cause the Inbox to grow; if they do, that might be the issue.
> 
> other compact bugs
> bug 492344 Once folder update(e.g mail copy) occurs without explicit folder
> open when outdated msf condition exists, "Compaction of local mail folder"
> is never executed, until next mail delete happens.
> bug 498185 MailDB(.msf) is corrupted(Rebuild-Index is invoked), if "Compact
> Folder" is invoked on "copy target folder" while "copy to folder by message
> filter" is running. "Compact Folder" itself silently fails.
> bug 137210 Compact local folder doesn't compact if we are rebuilding msf file
Comment 22 L.A.R. Grizzly 2012-03-09 11:14:50 PST
(In reply to David :Bienvenu from comment #21)

> Tim, if you go offline, and delete a message, do you get the compact prompt?
> Oh, and I assume you have deleted messages! Do you have a lot of filters
> that move mail from your inbox to other folders, on arrival? Those messages
> shouldn't cause the Inbox to grow; if they do, that might be the issue.

Going offline didn't make a difference. I did delete some messages. I have 54 filters that move messages from my Inbox to my Trash folder on arrival. I just closed TB and deleted my message filter file. That didn't work either. It still doesn't compact and I get no prompt. My Inbox is currently at 10mb.
Comment 23 David :Bienvenu 2012-03-09 11:54:57 PST
Tim, if you look at the inbox file, does it contain messages that were moved by filters, or just messages that arrived in your inbox and you deleted through the UI? If there are deleted messages there, they should have an x-mozilla-status line that ends in 8 or 9.

And are your filters "delete" filters, or did you create move filters, and pick the trash folder as the destination?
Comment 24 L.A.R. Grizzly 2012-03-09 12:27:58 PST
(In reply to David :Bienvenu from comment #23)
> Tim, if you look at the inbox file, does it contain messages that were moved
> by filters, or just messages that arrived in your inbox and you deleted
> through the UI? If there are deleted messages there, they should have an
> x-mozilla-status line that ends in 8 or 9.
> 
> And are your filters "delete" filters, or did you create move filters, and
> pick the trash folder as the destination?

They are a mixture of messages deleted by my filters and ones that I manually deleted. There are a few different status lines:

X-Mozilla-Status: 001b
X-Mozilla-Status: 0009
X-Mozilla-Status: 0000

All of my personal filters move the unwanted messages to my Trash folder. I have my Trash empty when I exit TB. I also use the "training" feature for removing unwanted messages. I only make a personal filter for those messages that the training feature seems to neglect or is unable to "learn."
Comment 25 David :Bienvenu 2012-03-09 13:07:17 PST
(In reply to Tim from comment #24)
> 
> X-Mozilla-Status: 0000
Are messages with this status messages that you think have been deleted? They're not marked as deleted in the mail store.
Comment 26 L.A.R. Grizzly 2012-03-09 13:18:06 PST
(In reply to David :Bienvenu from comment #25)
> (In reply to Tim from comment #24)
> > 
> > X-Mozilla-Status: 0000
> Are messages with this status messages that you think have been deleted?
> They're not marked as deleted in the mail store.

No. Those messages hadn't been deleted yet. I deleted them and then did a search for X-Mozilla-Status: 0000. No instances of X-Mozilla-Status: 0000 were found after deleting the messages. The Inbox remains at 10mb though.
Comment 27 L.A.R. Grizzly 2012-03-09 13:32:21 PST
I don't think any of my mail files are corrupt or in error. Remember, if I copy all of my mail files to SeaMonkey, they are immediately compacted.
Comment 28 WADA 2012-03-09 15:49:00 PST
(In reply to Tim from comment #27)
> Remember, if I copy all of my mail files to SeaMonkey, they are immediately compacted.

How about newly created Tb profile with same mail box file only?

(1) Create new Tb profile, start Tb with new profile, cancel account creation, create dummy News account("Local Folders" is created), enable auto-compact with required threshold value, mail.purge.ask=true, View/Folders/Unified to avoid unwanted error messages, go back to View/Folders/All, and termiate Tb.
(2) Call relevant mail folder "folderX".
Copy file of folderX and folderX.msf in current profile to ...Mail\Local Folders of new profile(path shown at "Local Directory:" of Server Settings of Local Folders of new Tb profile).
(3) Restart Tb with new profile. folderX appears. Open folderX. 

Is auto-compact invoked?

(4) Terminat Tb, delete folderX.msf, restart Tb, open folderX.

Is auto-compact invoked?

(5) Terminate Tb, restart Tb, create a mail, attach big files to create large mail(larger size than threshold value), Send Later, copy created mail in Outbox of "Local Folders" to folderX, delete the copied mail in folderX.

Is auto-compact invoked?

(In reply to Tim from comment #24)
> I have my Trash empty when I exit TB.

Auto-compact may be invoked by "empty Trash on exit" and timestamp of the auto-compact may be held at someplace. If it's true and held data is used after restart, next auto-compact may be invoked after an hour from restart.

Do you keep Tb running for more than one hour always?
Does your problem occur with "empty Trash on exit" disabled?
Comment 29 WADA 2012-03-09 16:06:42 PST
Timestamp of last auto-compact is saved in gtimeOfLastPurgeCheck, and it seems memory data which is initialized at restart.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#187
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#1983
If start of Tb is requested again just after Tb's window close(i.e. before complete end of Tb termination), current Tb still keeps running, because newly invoked thunderbird.exe passes request of "Tb's main window open" to currently running Tb instance via DDE on Win and the new thunderbird.exe himself terminates.

Do you restart Tb just after Tb's window close frequetly?
Comment 30 L.A.R. Grizzly 2012-03-09 17:36:35 PST
(In reply to WADA from comment #28)
 
> How about newly created Tb profile with same mail box file only?

> (1) Create new Tb profile, start Tb with new profile, cancel account
> creation, create dummy News account("Local Folders" is created), enable
> auto-compact with required threshold value, mail.purge.ask=true,
> View/Folders/Unified to avoid unwanted error messages, go back to
> View/Folders/All, and termiate Tb.
> (2) Call relevant mail folder "folderX".
> Copy file of folderX and folderX.msf in current profile to ...Mail\Local
> Folders of new profile(path shown at "Local Directory:" of Server Settings
> of Local Folders of new Tb profile).
> (3) Restart Tb with new profile. folderX appears. Open folderX. 


  I did all of the above verbatim.


> Is auto-compact invoked?  > NO
> 
> (4) Terminate Tb, delete folderX.msf, restart Tb, open folderX.
 
> Is auto-compact invoked?  > NO

> (5) Terminate Tb, restart Tb, create a mail, attach big files to create
> large mail(larger size than threshold value), Send Later, copy created mail
> in Outbox of "Local Folders" to folderX, delete the copied mail in folderX.

> Is auto-compact invoked?  > YES!!!!!

> Do you keep Tb running for more than one hour always?  > YES
> Does your problem occur with "empty Trash on exit" disabled?  > YES (no compacting)
> Do you restart Tb just after Tb's window close frequently?  > NO
Comment 31 L.A.R. Grizzly 2012-03-09 18:16:10 PST
OK, just out of curiosity, I tried step 5 with my regular profile. I created an email with an attachment. I chose Send Later. I copied it from my Local Folders Outbox to my Account Inbox. Deleting it invoked the auto-compact. This only happened once. I can't get it to invoke again by copying the unsent email into my Inbox again. This may have something to do with the 1 hour timer you mentioned.
Comment 32 WADA 2012-03-09 18:33:17 PST
(In reply to Tim from comment #31)
> This may have something to do with the 1 hour timer you mentioned.

See next code for "1 hour".
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#1980
Comment 33 L.A.R. Grizzly 2012-03-09 18:56:32 PST
(In reply to WADA from comment #32)
> (In reply to Tim from comment #31)
> > This may have something to do with the 1 hour timer you mentioned.
> 
> See next code for "1 hour".
> > http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#1980

OK, this is a little over my head. I reviewed the code. Are you saying that I should change something? I'm sort of a novice.
Comment 34 WADA 2012-03-09 19:17:59 PST
(In reply to Tim from comment #33)
> Are you saying that I should change something?

No. It's merely to let you know following.
Auto compact is invoked only when;
     (i)  auto-compact is triggered by threshold value for size of deleted mails
  && (ii) one hour expired from last compact execution
          (may be any last compact, or last auto-compact&compact-folders)
          when auto-compact is triggered

Was wasted space size actually larger than threshold value?
(1) The Inbox remains at 10mb though. Data of deleted mails exist in Inbox file.
(2) Copy file named "Inbox" to file named "folderX". Size of folderX=10MB.
(3) Restart Tb. folderX appears.
(4) manual Compact of folderX.
Will Comact actually be invoked?
(file size of "folderX" is reduced, mail data with X-Mozilla-Satus: ...8 or ...9 etc. is removed from file)
If file size is reduced, what is reduced size? Larger than threshold value?

How about original Inbox?
If file size is not reduced by "manual Compact of Inbox" even though X-Mozilla-Satus: ...8 or ...9 etc. exist, it may be phenomenon like next.
  compact fails, and status like compact-is-needed is lost,
  then next manual compact or auto-compact is not invoked for the folder.
"Silent compact failure" can occur if compact is interfered by someone. It can be observers by, write open folderX file by tool, manual Compact of folderX.
Comment 35 L.A.R. Grizzly 2012-03-09 19:45:53 PST
I can let it go days without seeing any compacting. I leave TB open for over 8 hours daily. It never compacts. I do close TB every day and I do shut down my machine every night. I just ran into this problem at the end of TB 11.0a2. I never had a problem before that. Something's amiss since I don't have any problems with SeaMonkey using the exact same setup and files (including my personal filters and training data file).
Comment 36 L.A.R. Grizzly 2012-03-10 06:49:42 PST
After getting some sleep and thinking this through, I think this is going to be a real booger to troubleshoot! If I make a change to my profile and test, I have to wait an hour to test again? This makes it difficult to determine whether my changes have made a difference. After looking at the code you mentioned, the whole process seems to be too complicated. I suggest a simpler method:

1) Are any of the folders larger than designated by the user? No->Don't Compact or Yes->Compact

or

2) Use the method of the xPunge add-on mentioned in comment 19. "Compact folders X minutes after startup and every X minutes after that."

I have it set to compact 10 minutes after startup and every 60 minutes after that. Seems simple enough to a novice like me.
Comment 37 Wayne Mery (:wsmwk, NI for questions) 2012-03-10 07:10:12 PST
Isn't compact supposed to immediately invoke on startup if threshold was reached prior to shutdown?
Comment 38 David :Bienvenu 2012-03-10 07:22:13 PST
(In reply to Wayne Mery (:wsmwk) from comment #37)
> Isn't compact supposed to immediately invoke on startup if threshold was
> reached prior to shutdown?

No, for the local mail folder case, it's triggered by deleting a message.
Comment 39 Wayne Mery (:wsmwk, NI for questions) 2012-03-10 07:29:41 PST
(In reply to David :Bienvenu from comment #38)
> (In reply to Wayne Mery (:wsmwk) from comment #37)
> > Isn't compact supposed to immediately invoke on startup if threshold was
> > reached prior to shutdown?
> 
> No, for the local mail folder case, it's triggered by deleting a message.

sure, but what if the trigger/compact failed after delete/prior to shutdown?
are you saying compact for local will *only* happen immediately after delete?
Comment 40 L.A.R. Grizzly 2012-03-16 11:22:42 PDT
Tested today. This bug is still present in TB 13.0a2.
Comment 41 Wayne Mery (:wsmwk, NI for questions) 2012-05-02 11:28:37 PDT
/me wonders if http://getsatisfaction.com/mozilla_messaging/tags/tb12compact may be somehow related
Comment 42 L.A.R. Grizzly 2012-05-02 11:42:30 PDT
(In reply to Wayne Mery (:wsmwk) from comment #41)
> /me wonders if http://getsatisfaction.com/mozilla_messaging/tags/tb12compact
> may be somehow related

That's odd. They're complaining about compacting too much. I never get compacted! This bug is still present in TB 14.0a2.
Comment 43 David :Bienvenu 2012-05-16 14:45:37 PDT
Created attachment 624550 [details] [diff] [review]
proposed fix

we were skipping the last server when totaling up the expungedBytes count, and counting the first server twice.
Comment 44 David :Bienvenu 2012-05-16 14:47:07 PDT
Comment on attachment 624550 [details] [diff] [review]
proposed fix

this also resets expunged bytes when iterating over local folders in case there's an error getting the expunged bytes.
Comment 45 Wayne Mery (:wsmwk, NI for questions) 2012-05-16 15:05:53 PDT
https://getsatisfaction.com/mozilla_messaging/tags/bug_725810 is canonical with the most users.  I haven't merged any of the many topics from https://getsatisfaction.com/mozilla_messaging/tags/tb12compact
Comment 46 David :Bienvenu 2012-05-16 15:28:52 PDT
I've requested a try server build with this fix. Results should show up here - http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/bienvenu@nventure.com-ac395aea3008

It's based off of tb 12 (comm-release) so it should be almost exactly like tb 12. I don't know if it will help with the issues where compact is happening too often but the clearing of expungedBytes might help.
Comment 47 Joe Sabash [:JoeS1] 2012-05-16 17:03:03 PDT
(In reply to David :Bienvenu from comment #46)
> I've requested a try server build with this fix. Results should show up here
> -
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/
> bienvenu@nventure.com-ac395aea3008
> 
> It's based off of tb 12 (comm-release) so it should be almost exactly like
> tb 12. I don't know if it will help with the issues where compact is
> happening too often but the clearing of expungedBytes might help.

I can probably test the hourly compact case. bug 750569
(kind of a waste of resources try-builds seem to be creating 64 bit builds, I'll wait and watch for 32 bit builds)
Comment 48 Joe Sabash [:JoeS1] 2012-05-16 19:18:06 PDT
Testing the try-build: 20120516152551
I think the general behavior is better
But I don't seem to get "compact required" notification until I do a restart.
I didn't wait to see if the notification would pop up after the 60 min timeout
Comment 49 David :Bienvenu 2012-05-16 21:14:09 PDT
(In reply to Joe Sabash from comment #48)
> Testing the try-build: 20120516152551
> I think the general behavior is better
> But I don't seem to get "compact required" notification until I do a restart.
> I didn't wait to see if the notification would pop up after the 60 min
> timeout

a restart restarts the 60 minute timer, and my patch wouldn't change anything about that...
Comment 50 Joe Sabash [:JoeS1] 2012-05-17 08:21:54 PDT
Further testing with the try build:
Set the threshold to 1 MB
Deleted about 700 bugmails..no prompt
Sent myself an 11 MB mail and deleted..no prompt
Restarted the build, deleted 1 message..Got the prompt to compact.

So it seems we may not be seeing the wasted space until a restart.
I'm waiting the required hour now to see if I see the cyclic hourly prompt.
Comment 51 David :Bienvenu 2012-05-17 08:28:14 PDT
(In reply to Joe Sabash from comment #50)
> Further testing with the try build:
> Set the threshold to 1 MB
> Deleted about 700 bugmails..no prompt
> Sent myself an 11 MB mail and deleted..no prompt
> Restarted the build, deleted 1 message..Got the prompt to compact.
> 
> So it seems we may not be seeing the wasted space until a restart.
> I'm waiting the required hour now to see if I see the cyclic hourly prompt.

that's all expected behavior, with and w/o this patch. On the first delete, we checked the wasted space and decided it was less than 1MB. We don't check the wasted space again for an other hour, or until restart, because that check can be somewhat expensive.
Comment 52 Joe Sabash [:JoeS1] 2012-05-17 09:20:55 PDT
Waited 1 hour,deleted 1 small message:
Got the prompt for compaction, so it seems no effect on the frequent compaction issue bug 750569
I'll run with this build for the rest of the day and see if that is always the case
(hourly compaction prompt without hitting the threshold)
Comment 53 neil@parkwaycc.co.uk 2012-05-19 07:37:03 PDT
Comment on attachment 624550 [details] [diff] [review]
proposed fix

>       do
>       {
>+        server = do_QueryElementAt(allServers, serverIndex++);
...
>         if (!supportsCompaction)
>           continue;
...
>-      while (++serverIndex < numServers);
>+      while (serverIndex < numServers);
I think this has too much potential to go wrong.
Comment 54 neil@parkwaycc.co.uk 2012-05-19 07:39:19 PDT
Created attachment 625405 [details] [diff] [review]
What I was thinking of
Comment 55 David :Bienvenu 2012-05-19 10:12:08 PDT
Comment on attachment 625405 [details] [diff] [review]
What I was thinking of

sure, that works too, but you lost the change where I initialize expungedBytes to 0 - can you add that back in, or give me r+ to land it separately?
Comment 56 neil@parkwaycc.co.uk 2012-05-20 12:51:54 PDT
Comment on attachment 625405 [details] [diff] [review]
What I was thinking of

Pushed changeset 5e27680293a8 including expungedBytes = 0; to comm-central.
Comment 57 Joe Sabash [:JoeS1] 2012-05-20 13:05:33 PDT
comm-central:https://hg.mozilla.org/comm-central/rev/5e27680293a8
real links are easier to follow :}
Comment 58 David :Bienvenu 2012-05-20 14:38:30 PDT
Comment on attachment 625405 [details] [diff] [review]
What I was thinking of

[Triage Comment]
we're going to want this for aurora and beta, where this is https://hg.mozilla.org/comm-central/rev/5e27680293a8 - I'll try to land on branches tomorrow.
Comment 60 Mark Banner (:standard8) (afk until 26th July) 2012-05-22 01:47:10 PDT
Assuming this is fixed now.
Comment 61 L.A.R. Grizzly 2012-05-22 14:52:39 PDT
(In reply to Mark Banner (:standard8) from comment #60)
> Assuming this is fixed now.

Hooray! I updated to the latest Aurora today and immediately got a prompt to compact my folders (something I haven't seen in months). The problem appears to be fixed. I'll post back if I have any more difficulties. Kudos!
Comment 62 David :Bienvenu 2012-05-22 14:57:28 PDT
(In reply to L.A.R. Grizzly from comment #61)

> Hooray! I updated to the latest Aurora today and immediately got a prompt to
> compact my folders (something I haven't seen in months). The problem appears
> to be fixed. I'll post back if I have any more difficulties. Kudos!

OK, cool, thx for the bug report and feedback.
Comment 63 L.A.R. Grizzly 2012-05-22 15:06:08 PDT
(In reply to David :Bienvenu from comment #62)
> (In reply to L.A.R. Grizzly from comment #61)

> OK, cool, thx for the bug report and feedback.

Just to note: I've been playing around with TB and I get the prompt every time now. I sent myself some big messages and moved some big messages to my Inbox. I get prompted every time TB is restarted (after deleting a message). I'll now delete a big message and wait the required hour to test without the restart.
Comment 64 David :Bienvenu 2012-05-22 15:07:46 PDT
Is your threshold still set to 1MB?
Comment 65 L.A.R. Grizzly 2012-05-22 15:10:08 PDT
(In reply to David :Bienvenu from comment #64)
> Is your threshold still set to 1MB?

Yes. I've never changed it during this bug report. I've even uninstalled Xpunge 0.43 to be sure the tests are clean.
Comment 66 David :Bienvenu 2012-05-22 15:32:00 PDT
If you raise it, do you not get prompted to compact on restart? We had users who had the opposite issue from this one, which is that they get re-prompted even though they don't think there's more than XX MB of space that would be reclaimed.
Comment 67 L.A.R. Grizzly 2012-05-22 16:23:19 PDT
(In reply to David :Bienvenu from comment #66)
> If you raise it, do you not get prompted to compact on restart? We had users
> who had the opposite issue from this one, which is that they get re-prompted
> even though they don't think there's more than XX MB of space that would be
> reclaimed.

First, after the hour wait without restart, compacting is acting as expected. I deleted a message and my folders were automatically compacted.

As you mentioned above, I raised the threshold to 21mb for a test. I tested numerous times with various messages (deleting and not deleting, compacting and not compacting) and I never got a prompt to compact on restart (with an empty Inbox) like the others complained about. Note that I never experienced the false prompts that the others did.

I think you have my bug fixed, you'll have to wait to see if the other false prompts are corrected. I can't get them to appear on my system (although, I never had them).
Comment 68 Joe Sabash [:JoeS1] 2012-05-22 17:54:50 PDT
Yes, I'm pretty sure the frequent compaction thingy is fixed.(I beat it up pretty good for several hours)
One thing though, the message in the status bar when the check for wasted space is being performed on restart seems to be "Loading Message..."
Not very informative of what is actually happening.
Also, no record of the compaction is recorded in the Activity Manager.

I think one or both of these should be improved.
Comment 69 L.A.R. Grizzly 2012-06-02 20:53:52 PDT
After a week and a half of normal use, this bug is definitely fixed on my system. Kudos!
Comment 70 Wayne Mery (:wsmwk, NI for questions) 2012-06-24 08:43:28 PDT
(In reply to L.A.R. Grizzly from comment #35)
> I can let it go days without seeing any compacting. I leave TB open for over
> 8 hours daily. It never compacts. I do close TB every day and I do shut down
> my machine every night. I just ran into this problem at the end of TB
> 11.0a2.

Grizzly, can you clarify please - did you mean to write "at the end of TB12.0a2"?
Comment 71 L.A.R. Grizzly 2012-06-24 09:05:49 PDT
(In reply to Wayne Mery (:wsmwk) from comment #70)
> (In reply to L.A.R. Grizzly from comment #35)
> > I can let it go days without seeing any compacting. I leave TB open for over
> > 8 hours daily. It never compacts. I do close TB every day and I do shut down
> > my machine every night. I just ran into this problem at the end of TB
> > 11.0a2.
> 
> Grizzly, can you clarify please - did you mean to write "at the end of
> TB12.0a2"?

From my recollection, I started noticing loss of compacting in the last builds of 11.0a2. The problem has been resolved on my system. It was fixed during the latter builds of 14.0a2.

Note You need to log in before you can comment on or make changes to this bug.