Closed Bug 750569 Opened 12 years ago Closed 12 years ago

Compact Folders window appearing very frequently starting in TB12

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(thunderbird13-, thunderbird14-)

Status:
VERIFIED DUPLICATE of bug 710056
Tracking Flags:
Tracking Status
thunderbird13 - ---
thunderbird14 - ---

People

(Reporter: larry.ackerman, Unassigned)

References

(
URL
)

Details

(Keywords: regression, Whiteboard: [gs])

Attachments

(2 files, 1 obsolete file)

Extension to show expunged bytes
17.03 KB, application/octet-stream
Details
Add to cache if valid in nsMsgDatabase::SetSummaryValid
1.07 KB, patch
Details | Diff | Splinter Review 
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0
Build ID: 20120312181643

Steps to reproduce:

reading emails--when deleting one message


Actual results:

the Compact Folders window pops up very frequently 


Expected results:

Compact Folders window should not pop up or maybe once every three months--as in the past behaviour
(In reply to larry.ackerman from comment #0)
See http://kb.mozillazine.org/Compacting_folders if it can help you configure it.
URL: https://getsatisfaction.com/mozilla_m...
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Whiteboard: [gs]
If compaction is failing, the count of wasted space won't go down, and this prompt will keep happening. Does it seem to be trying to compact the same folder(s) over and over again?

there's an hour counter that should prevent compaction prompts from happening more than one hour, but restarting clears the counter.
 https://getsatisfaction.com/mozilla_messaging/tags/tb12compact is up to about 10 reports - no time to run them down for the next couple days

xref Bug 725810 - Automatic Compact Folders Not Working
See Also: → 725810
Whiteboard: [gs] → [gs][regression?]
The configuration appears to be correct--I didn't change it. This started after the upgrade to 12.0  It has also happened on my home computer. I assume the prompt for compaction is directed at my InBox--the folder I am working in. I don't know where to look for the folder/file size but even after I moved half of my messages out of the InBox the hourly prompt continues. When I do a "Compact Folders" the process appears to complete properly--no indication otherwise.
several more reports in the mix.

and it occurs to me, that if users who are not being prompted might also be getting "too frequent" compacts and not know it


And a recurrence in https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_is_asking_several_times_daily_to_compact_my_folders who thought for several days that a Mac reboot solved the issue - but apparently not
tracking-thunderbird13: --- → ?
tracking-thunderbird14: --- → ?
Keywords: regression
(In reply to larry.ackerman from comment #0)
> Actual results:
> the Compact Folders window pops up very frequently 

"Compact Folders window pops up" indicates that mail.purge.ask=true is set and it works as expected in your environment.

> Expected results:
> Compact Folders window should not pop up or maybe once every three
> months--as in the past behaviour

As written in comment #2, Tb has "one hour timer" internally for auto-compact.
So, next is very normal.
(1) Delete some mails, and exceeds threshold value
    => Prompt before auto-compact pops up, and auto-compact is executed by OK
(2) Delete some mails, and exceeds threshold value again
    => Prompt before auto-compact is not shown
       if less than one hour after last compact execution.
(3) Delete some mails, and exceeds threshold value,
    and more than one hour expired after last compact execution.
    => Prompt before auto-compact pops up
Please note that "Delete some mails" doesn't mean manual delete. "Delete at an mail folder" occurs in any of manual delete, manual move, filter delete, filter move.

What is your threshold value setting for auto-compact?

Was no auto-compact actually happened for three months in your environmeny even though mail.purge.ask=true(and with mandatory restart of Tb after setting change, if mail.purge.ask=true is set via Config Editor or is set by manual prefs.js editing)?
In http://gsfn.us/t/2uiti the affected user reports that repairing the folder ("Drafts" in that case) resolved the issue. This would confirm the suspicion in comment #2 that folder corruption prevented the count to be reset accurately.

The problem may be to identify the folder in question, if indeed that's the mechanism responsible for those persistent prompts.
An affected user in http://gsfn.us/t/2tqse#reply_8883594 states that repairing the folder index on all folders didn't resolve the issue. Compact prompts are issued after roughly one hour of running time, thus corresponding to the timer setting.
I have the issue since Thunderbird 12 and it seems like all folders are affected (I mostly navigate in Locale Folders, so I am unsure about other accounts).
Is there any specific reason why this hasn't been confirmed yet? There seem to be enough reports in by now to suggest that this is "for real" or is this report kept in an unconfirmed state as a possible duplicate of bug 725810?
If you can reproduce it on your system then surely mark it confirmed.
I would have done so if I could reproduce it myself. So far it didn't happen to me, but then I'm usually compacting the folders manually already well before the 20MB default threshold is hit. Thus, apparently it has to be hit at least once.
I tend not to confirm until it's roughly in a state for a developer to pick it up. Meaning I can repro, or it has steps that (tyipcally) more than one non-developer person has reproduced, and we know it's not an addon - which is a higher standard than >1 person seeing the same behavior.

But, it also depends on the bug :)
(In reply to rsx11m from comment #12)
> I would have done so if I could reproduce it myself. So far it didn't happen
> to me, but then I'm usually compacting the folders manually already well
> before the 20MB default threshold is hit. Thus, apparently it has to be hit
> at least once.

So that is the specific reason why this bug is not in confirmed state. Nobody could find out the exact steps how to reproduce it.
In my humble opinion, a bug should get confirmed if someone is CCed who can reproduce the problem or experiences it recurrently.

Looking at nsresult nsMsgDBFolder::HandleAutoCompactEvent(nsIMsgWindow *aWindow)
http://mxr.mozilla.org/comm-release/source/mailnews/base/util/nsMsgDBFolder.cpp#1813

There are three changes in the timespan for Thunderbird 12:

http://hg.mozilla.org/releases/comm-release/rev/f9d611e3d0a5
Bug 695256 - Switch from PRBool to bool and replace PR_TRUE/PR_FALSE with true/false in comm-central

http://hg.mozilla.org/releases/comm-release/rev/bed472157fd2
Bug 714733 - Fix crash @nsMsgDBFolder::HandleAutoCompactEven

http://hg.mozilla.org/releases/comm-release/rev/097bc36f180d
fix bug 402392, add support for pluggable stores

But I don't recognize anything suspicious.
(In reply to Archaeopteryx [:aryx] from comment #15)
> In my humble opinion, a bug should get confirmed if someone is CCed who can reproduce the problem or experiences it recurrently.

OTOH, not being confirmed doesn't prevent any developer from looking at the bug - and indeed one is engaged.  conveniently, you do have privileges to confirm bugs as you see fit.
(In reply to larry.ackerman from comment #4)
> The configuration appears to be correct--I didn't change it. This started
> after the upgrade to 12.0  It has also happened on my home computer. I
> assume the prompt for compaction is directed at my InBox--the folder I am
> working in.
The prompt for compact applies to all your folders. Essentially, when you delete/move a message, if it's been more than an hour since the last time we checked, we look at each folder and see compacting that folder would reclaim some space. We sum up the amount of space that would be saved from each folder, and if it's greater than the threshold, we prompt the user and do the compaction. Compaction should set the space that would be reclaimed to 0 for each folder. If compaction fails for a folder, then the space that would be reclaimed (which is called "expungedBytes") would not be set to 0.

Are all your folders local/pop3, or do you have any imap accounts? We include the space taken up by deleted imap messages in offline stores in the total.
In the Options/Advanced/Network and Disk Space I have 1024 Mb cache and 40Mb for the Compact Folders--it was 20Mb in the past--

I use a POP Mail Server and have settings to NOT Delete any messages
All of the Local folders are archives of previous years and are never deleted. I do not believe I have IMAP accounts.

I looked at the size of my Inbox file before and after compacting and it was indeed changing but only by a few MB at most. The other folders should not change in size since nothing was deleted in them. So it appears that the "expungeBytes" is not getting reset to 0. Do you have any way to test for compaction success other than file size?
correction: I have found that my active files are in a folder named IMAP.UCSF.edu
--so apparently it is an imap account--I do not manage the exchange email from Thunderbird--simply download the emails.
I have no IMAP accounts. Even with taking Junk and message moving account, I doubt I got 5 MB (my threshold) of message moving/deleting.
Larry, do you also have a pop3 account, or access your messages from local folders? We don't tend to prompt you for offline imap cleanup very often, because the check for wasted space in imap only happens if you're offline. For pop3/local messages, it happens whenever you delete a message.

to people who see this issue, if you delete a message, get prompted for compact, let the compact finish, shutdown and restart, and delete an other message, do you get prompted right away?
For imap, we're double counting the wasted space for each message. I suspect we've always been doing that, and I realize not everyone is using imap.
I haven't set Thunderbird to get prompted for compacting, but see in the status bar and the message list and message content areas get blank.

After restarting Thunderbird, compacting doesn't kick in directly after deleting a message.

I saw it twice today:
1. After resuming from hibernation, I opened my Bugzilla folder and after deleting a message, compacting started.
2. Some hours later (I didn't do anything in Thunderbird in between), I switched to a different folder and deleting a message there also started compacting. Deleting more messages directly after this (independent of the folder) didn't trigger compacting anymore.
Reproducible here on trunk: 20120511030200
STR pop3 no filters,standard inbox,
Lowered threshold to 1 meg
Deleted enough messages to trigger compact (many bugmails needed)
Compaction complete shutdown TB for the day
Startup next day, receive a few messages (no where near a meg)
After some time maybe 2 hours delete a message
Received prompt for compaction

The only thing unusual is that I'm using "leave mail on server" for this system.
Status: UNCONFIRMED → NEW
Ever confirmed: true
This started when I upgraded to 12.0.1 yesterday. I have changed the threshold from 20 MB to 175 MB and it still happens. It appears to me the threshold is no longer being checked, only the hourly timer is active.
Report in http://gsfn.us/t/2tqse#reply_8912184 that toggling the auto-compact option in Tools > Options > Advanced > Network & Disk Space may reset the issue. He hasn't seen the notification after having switched off that function for a while and enabling it again (thus, multiple restarts between off and on).
Whiteboard: [gs][regression?] → [gs]
Blocks: 755993
I followed the suggestion in comment 21 and the symptoms ended.

delete a message, get prompted for compact, let the compact finish, shutdown the program and restart

thanks!
(In reply to Archaeopteryx [:aryx] from comment #23)

> 1. After resuming from hibernation, I opened my Bugzilla folder and after
> deleting a message, compacting started.
> 2. Some hours later (I didn't do anything in Thunderbird in between), I
> switched to a different folder and deleting a message there also started
> compacting. Deleting more messages directly after this (independent of the
> folder) didn't trigger compacting anymore.

In my case it is selecting an NNTP account, it is the only times I have ever seen the compact dialog appear.
(In reply to Matt from comment #28)
> (In reply to Archaeopteryx [:aryx] from comment #23)
> 
> > 1. After resuming from hibernation, I opened my Bugzilla folder and after
> > deleting a message, compacting started.
> > 2. Some hours later (I didn't do anything in Thunderbird in between), I
> > switched to a different folder and deleting a message there also started
> > compacting. Deleting more messages directly after this (independent of the
> > folder) didn't trigger compacting anymore.
> 
> In my case it is selecting an NNTP account, it is the only times I have ever
> seen the compact dialog appear.

Having the same issue everytime I open mdat or almost everytime.
fix for bug 725810 may have helped with this - can anyone who sees this try a tb 13 beta build?
Not tracking for now, as we fix this is possibly fixed. Please re-flag if there is still a significant problem with the latest betas.
tracking-thunderbird13: ? → -
tracking-thunderbird14: ? → -
I'm not interested in testing beta versions, the regular releases recently have been buggier than I like already. I just want to be able to use the software. I'd really appreciate this being tracked and to get a notice when it's confirmed that it's fixed in a regular release version.
Well, you are following this bug here by being on the CC list, thus you'll see if and when it resolves as "fixed". The other bug's fix listed in comment #31 should be included in 13.0 beta 3, which is currently available for download and testing from http://www.mozilla.org/thunderbird/channel/ (right button).

So, all that David and Mark are asking for is for someone affected to verify if the problem still exists in that beta build, or if it has been resolved by that other fix as well. If it turns out to be fixed, this bug here can be closed.
Confirming that 13.0 beta 3 appears to resolve the constant compaction request bug, at least insofar as it's possible to confirm a negative.  Thanks for fixing this.
Bug 762169 states that the problem is still observed after updating to 13.0, though there is no indication that any other action (toggling the preference, etc.) was performed prior to or after the update.
Sorry, there was an important piece of information I've missed:
> Only when a manual compact is done will it clear the flag.

Thus, a one-time manual compact may be necessary to reset the counter.
Blocks: 762169
I had this problem in TB 12 on all my machines, but it seemed to resolve itself once I performed a manual (on-demand) compact.  Immediately after upgrading to 13.0 on two machines, the problem recurred on both as soon as a move was done.  (The first move was huge, over 50mb total, so it was not unreasonable to ask, but subsequent moves were only a few kbs, and still it asked.)  I filed Bug 762169.

I ran a manual compact (as stated in my report) but now can tell you that the request to compact may STILL occur regardless -- which seems to be different (and worse) than the original TB 12 issue.  But it seems to depend where the cursor is when the manual compact happens:  On INBOX, the problem will recur (or not be cleared), but doing it on LOCAL FOLDERS so far has cleared the problem again.

IMHO, TB should compact only when needed (based on cumulative moves, ie- the amount of space to be recovered), not every time a message is moved or according to some arbitrary timer.
I have to retract comment #34.  TB 13.0 b4 is still requesting compaction of my Inbox each(?) time that:

1) An hour (or so) has passed; and
2) At least one message is deleted from the Inbox.

The compaction request comes when the message is deleted, regardless of whether enough messages have been deleted to exceed the threshold set in Options > Advanced > Network & Disk Space > Disk Space > Compact when . . .
(In reply to Dan Pernokis from comment #37)

> 
> IMHO, TB should compact only when needed (based on cumulative moves, ie- the
> amount of space to be recovered), not every time a message is moved or
> according to some arbitrary timer.

That's what we normally do - when it doesn't work that way, it's a bug.
This bug is still present in TB 14.0 beta
This only started happening to me with update to Thunderbird 13.0. The compaction request appeared whenever I moved an e-mail from the Inbox to any other folder. I only made it stop by unchecking the "Compact folders when it will save over . . ." box. When it did happen, it slowed down my computer considerably. And this computer is less than 9 months old (i7 2600, 8GB RAM, Windows 7).
Rod, Larry, do you still see this in version 13 ?
I've been seeing this too with 13, but never with 12. I have mixed accounts, but it appears when doing things to a pop account.
I have not tested version 13. I have had various problems with the last few upgrades so I have stopped simply upgrading when the new version comes out. One of those problems, where the upgrade messed with my filters (and I have a LOT of filters) is only fixable by my manually going through each and every filter and fixing it myself. I have chosen to not spend the multi-hours to just sit there fixing those, I'm fixing them as each filter gets used and I see the config needs to be fixed on it.

I am not going to be upgrading unless there is some clear value in doing so, and I have assured myself that it is not going to create other problems by doing so. I am going to wait until others have confirmed things like this asking every hour to compress has been fixed in some new version.
Reports in http://gsfn.us/t/2tqse continue, the most recent one stating that it just started after upgrading to 13.0.1. Please comment there if you want any of the users perform additional steps for testing.
(In reply to Magnus Melin from comment #43)
> I've been seeing this too with 13, but never with 12. I have mixed accounts,
> but it appears when doing things to a pop account.

clearly something is not fixed by bug 725810. The question is, is there a regression, or simply a testcase that was not considered by bug 725810?

It may be helpful to keep separate reports to avoid confusing of "seen in version 12", and "version 13 but never seen in version 12". I don't think we have a new bug report for the later.
(In reply to Wayne Mery (:wsmwk) from comment #46)
> "version 13 but never seen in version 12"

I have been labeling potential reports of such in gsfn as https://getsatisfaction.com/mozilla_messaging/tags/tb13compact
There was an update last week to 12.0.1 ? and the compact folders bug returned which I solved by allowing the compaction then immediately closed the program and restarted. Today I upgraded to 13.0.1 and have not seen the bug thus far.
Larry, are you still clean on version 13.0.1?  

Note, 13.0.1 contained NO specific patches related to compact afaik.  But were you ever on version 13.0?  Because version 13.0 did have a patch via Bug 725810 - Automatic Compact Folders Not Working ... to fix regression of bug 402392
Summary: Compact Folders window appearing very frequently → Compact Folders window appearing very frequently starting in TB12
For readers in this thread, I just posted this message to Bug 762169:

>>> Following each of several TB upgrades, starting with TB 12.0, I found the problem occurs and never resolves itself unless & until I run a manual compact with the cursor on LOCAL FOLDERS.  (I display "All" in view/folders.)  Sometimes I've had to close TB and relaunch it to make the fix stick, but not always.  However, once fixed -- as it is now -- the problem doesn't seem to recur UNTIL the next TB upgrade.  This was true starting with TB 12.0 and into 12.0.1, 13.0, and now 13.0.1.  That is, I can't make it not-work (ie- start prompting all the time) once fixed.  And this has happened identically on all three of my machines, starting with the 12.0 upgrade. <<<

Previously I thought (and reported in Comment 37) that simply doing a manual compact would clear the problem, but I found that cursor position matters -- must be on LOCAL FOLDERS.  Also found that sometimes I had to close & relaunch TB following the compact -- but not always -- or perhaps sometimes I had closed TB without realizing, and other times moved or deleted files and had to re-do the manual compact.

So there you have it -- from version 12.0 up to 13.0.1 -- and a procedure that seems to fix the problem without resolving the cause.
I upgraded to 13.0.1 and did the procedure of compacting folders after clicking on Local Folders and that seems to have cured the problem for me. Note for others who are wanting to apply this fix, Compact is not a choice on the context menu of Local Folders, you have to click File menu, then Compact Folders.
bienvenu, any thoughts on comment 50, 51?
(In reply to Wayne Mery (:wsmwk) from comment #52)
> bienvenu, any thoughts on comment 50, 51?

sounds like normal compaction wasn't getting to local folders. But, there was a fix for a bug where we were skipping the last account - perhaps that was the local folders account. Or perhaps there are still cases where we're missing an account to compact, perhaps because of errors compacting earlier accounts...
Compacting with the cursor over Local Folders has no effect whatsoever on the hourly compaction requests on my computer.  In fact, as far as I can tell, manually compacting with the cursor on Local Folders does not do anything at all (the folder is has no e-mail in it, just the default folders).

Tb 14.0 beta
(In reply to David :Bienvenu from comment #53)
> (In reply to Wayne Mery (:wsmwk) from comment #52)
> > bienvenu, any thoughts on comment 50, 51?
> 
> sounds like normal compaction wasn't getting to local folders. But, there
> was a fix for a bug where we were skipping the last account - perhaps that
> was the local folders account. 

right, and that fix appears in version 13 https://bugzilla.mozilla.org/show_bug.cgi?id=725810#c43

> Or perhaps there are still cases where we're
> missing an account to compact, perhaps because of errors compacting earlier
> accounts...

it would seem so from the likes of https://getsatisfaction.com/mozilla_messaging/topics/compact_madness  (one of the bugs in https://getsatisfaction.com/mozilla_messaging/tags/tb13compact ... which are not necessarily all valid or about frequency)
Reply to mgoldey (comment 54)...

But did you close/terminate TB and relaunch TB immediately after the compact?  I found that doing the close afterwards sometimes makes the difference.  The other variable is VIEW -- whether Local Folders is the top of the tree or not -- which may make a difference.  (Unfortunately once the problem is fixed, I can't break it again to re-test.)

And -- someone please correct me if I'm wrong -- I believe compacting works from the cursor on downward (ie all files & folders there and below), so "local folders" should include everything.
(In reply to Dan Pernokis from comment #56)
> Reply to mgoldey (comment 54)...
> 
> But did you close/terminate TB and relaunch TB immediately after the
> compact?  

Yes.  Makes no difference.  (I wish it were otherwise.)

>  The other variable is VIEW -- whether Local Folders is the top
> of the tree or not -- which may make a difference. 

I don't think it does.  My primary account/folder is at the top of the screen, and I've compacted it more times than I can count.  If compacting the topmost folder as displayed in the account/folder list had the effect of compacting everything below (I don't believe that it does, see below) and compacting everything solved this bug, I would have seen that behavior weeks ago when this all started.  

> And -- someone please correct me if I'm wrong -- I believe compacting works
> from the cursor on downward (ie all files & folders there and below), so
> "local folders" should include everything.

I don't think so.  This morning, I compacted an account below my topmost account/folder , and TB found something to compact and in fact did so.  The topmost account/folder has been compacted dozens of times without impacting the account/folder below.

FWIW, I have used Folderpane Tools a tool to re-arrange the account order as displayed on the leftmost pane.  But, I have compacted each and every account that I use in TB, and still get hourly compaction requests.  Therefore, even if "topmost" really means the account with the lowest ID #, I don't think that this trick works.  

Really, I wish that it were otherwise.  Would you like to compact now?
 mgoldey, sounds like you would be a really great test case :)


perhaps me too, though it comes in waves ... I am currently in a wave ... I answered the prompt a short time ago (but not sure if it was more or less than an hour ago), definitely didn't delete any messages because I wasn't even at the desk, but still got another prompt. And this time it wasn't because I was in a newsgroup.
(In reply to David :Bienvenu from comment #22)
> For imap, we're double counting the wasted space for each message. I suspect
> we've always been doing that, and I realize not everyone is using imap.

bienvenu, I totally overlooked this comment. Was this fixed, or do we have a bug for it?
(In reply to rsx11m from comment #36)
> Sorry, there was an important piece of information I've missed:
> > Only when a manual compact is done will it clear the flag.
> 
> Thus, a one-time manual compact may be necessary to reset the counter.

In my case, doing manual compact has made no difference.  FWIW, my activity is almost all imap. 


We have enough people here (magnus, ludo, myself, etc) who can reproduce - perhaps we can reproduce and shoot this problem dead in warsaw?
I extended the extra folder columns extension to show expungedBytes (attached). What I see is that after compaction, the expunged bytes do not change. On restart, they are still unchanged, but they change to zero when I actually click on the compacted folder in the folder pane.
I should note that the issue described in comment 62 only occurs for local folders. When I do the same thing for an IMAP (offline enabled) folder, the expungedBytes go to zero when I do a compact.

The offline compactor has these lines which are missing from the local compactor:

1021   // this forces the m_folder to update mExpungedBytes from the db folder info.
1022   uint32_t expungedBytes;
1023   m_folder->GetExpungedBytes(&expungedBytes);
1024   m_folder->UpdateSummaryTotals(true);

I would need to investigate some more to see if this is a real issue, and/or the root cause of this bug.
For me, on POP3 or Local Folders accounts, compacting a folder leaves the number to expunge shown. However it disappears after clicking the folder in the folderpane again but only after a small amount of time, varying from 0s to 60 seconds, no restart needed.
On Local folders Inbox it even isn't needed to click the folder again, the number disappears immediately by itself.
With aceman's confirmation, I can go ahead and implement the changes from my comment 63. AFAICT though this will only affect issues on local folders, not IMAP.

It would be interesting if anyone seeing this on an IMAP offline folder could use my test extension, and report whether compact is changing the expunged bytes number.
Status: NEW → ASSIGNED
Assignee: nobody → kent
Maybe while the number is still there (expunged bytes not flushed yet) and a new message is deleted, the compact confirmation is triggered again?
(In reply to Kent James (:rkent) from comment #62)
> Created attachment 657563 [details]
> Extension to show expunged bytes
> 
> I extended the extra folder columns extension to show expungedBytes
> (attached). What I see is that after compaction, the expunged bytes do not
> change. On restart, they are still unchanged, but they change to zero when I
> actually click on the compacted folder in the folder pane.

Using a pop3 account, I see precisely the same thing (except I didn't go through a restart).  TB 16.

I also have an imap account, which I don't use very often.  I sent myself a couple of large messages.  TB did not ask me to compact when I deleted the messages, although the size of the deleted messages is below the compaction threshold for the imap account. (Of course, my pop3 account would have prompted me to compact, regardless of the deleted messages' size or the compaction threshold, so long as an hour or so had passed since my last use of TB.) 

After deleting the messages from the imap account, the expunged bytes were displayed.  The counter went back to blank once I did a manual compaction on the imap inbox.  

So, there is a difference in behavior between imap and pop3 accounts.
installed rkent's addon on my work machine (I missed it somehow).

manual Compact Folders for both imap and pop account fails to decrement the expunge amount.  Only compacting single folder changes the expunge number.

And after Empty Trash the expunge amount is unchanged.
I'm encouraged that rkent has signed up here.

And it just occurred to me, that we really don't want this regression with many ugly heads to get out in ESR17.
tracking-thunderbird17: --- → ?
Severity: normal → major
Wayne have you tried restart or clicking again on a compacted folder?
So here is a partial analysis, which points to this being a regression from bug 402392 (which landed in TB12)

We have these lines of code in nsFolderCompactor.cpp:

464     nsCOMPtr<nsIMsgDBService> msgDBService =
465       do_GetService(NS_MSGDB_SERVICE_CONTRACTID, &rv);
466     NS_ENSURE_SUCCESS(rv, rv);
467     rv = msgDBService->OpenFolderDB(m_folder, true, getter_AddRefs(m_db));
468     m_db->SetSummaryValid(true);
469     m_folder->SetDBTransferInfo(transferInfo);
470 
471     nsCOMPtr<nsIDBFolderInfo> dbFolderInfo;
472 
473     m_db->GetDBFolderInfo(getter_AddRefs(dbFolderInfo));
474 
475     // since we're transferring info from the old db, we need to reset the expunged bytes,
476     // and set the summary valid again.
477     if(dbFolderInfo)
478       dbFolderInfo->SetExpungedBytes(0);

All of this is new in bug 492392 except for lines 469, 477, and 478.

OpenFolderDB creates a new message database, assigning it to m_db, but does not save that in the folder.  m_folder->SetDBTransferInfo(transferInfo) needs a message database, so it creates a SECOND database with a the transferInfo (which has the old non-zero expungedBytes) and that becomes the "real" message database and associated real dbFolderInfo object. Yet we then get the dbFolderInfo object from the dangling database, and that is the one where expungedBytes = 0 is set.

Both database objects and associated dbFolderInfo are pointing to the same file, which could create all kinds of fun as they compete to update the file. That might be the source of other issues, and could lead to some indeterminacy in this.

I'm not ready to propose a fix, as I don't really understand why these lines were added in the first place, but I'm confident there are some real issues here that need to be solved.
Kent, thanks for posting this. I've been looking in exactly the same place for bug 710056, but hadn't spotted the creation of two copies of the data structure. I think we'll need to get Bienvenu to review the patch.
(In reply to :aceman from comment #70)
> Wayne have you tried restart or clicking again on a compacted folder?

My results depend on which profile I use.  

work profile (at work) bug doesn't show - File | Compact Folders causes expunged to go to zero for pop, imap and local.  Note: Empty Trash leaves an expunged count   (profile has mix of news, imap, pop, rss)

work home (at laptop) bug shows - File | Compact Folders doesn't affect expunged count for pop, local AND imap.   Note: here too, Empty Trash leaves an expunged count   (profile has mix of news, imap, pop, rss)
Attached patch WIP patch (obsolete) — — Splinter Review 
This patch adds one line that fixes the issues reported earlier:

+    m_folder->SetMsgDatabase(m_db);

plus leaves in some of my debugging code (the unit test changes do not effectively test for this bug yet). This fixes the expungedBytes issue. However, I an seeing the following errors about 1/3 of the time when I copy a message to a local folder, then compact:

###!!! ASSERTION: whoops, something wrong with previous copy: 'mCopyState->m_leftOver == 0', file c:/tb/central/src/mail
news/local/src/nsLocalMailFolder.cpp, line 2231
91433e0 New nsDBFolderInfo m_expungedBytes = 0
91433e0 LoadMemberVariables m_expungedBytes = 0
5b9f0c0 SetExpungedBytes to 3003
9143480 New nsDBFolderInfo m_expungedBytes = 0
9143480 LoadMemberVariables m_expungedBytes = 0
WARNING: Must complete empty transaction when compositing!: file c:/tb/central/src/mozilla/layout/base/nsPresShell.cpp,
line 5301
9143660 New nsDBFolderInfo m_expungedBytes = 0
9514030 New nsDBFolderInfo m_expungedBytes = 0
###!!! ASSERTION: temp file not of expected size in compact: 'tempFileRightSize', file c:/tb/central/src/mailnews/base/s
rc/nsMsgFolderCompactor.cpp, line 424
9143660 New nsDBFolderInfo m_expungedBytes = 0
9143660 LoadMemberVariables m_expungedBytes = 3003

I was also seeing this before my fix, so I think this is a separate issue, yet intimately related to the same problems. I want to figure out this issue as well before proceeding with a patch for review.
The "something wrong" warning occurs when nsMsgLocalMailFolder::CopyData exits after executing this code, which occurs only when the data for the message does not have a \n or \r terminator:

2090       else
2091       {
2092         memmove (mCopyState->m_dataBuffer + 1, start, mCopyState->m_leftOver);
2093         break;
2094       }

I'm not yet ready to say what it is supposed to be doing.
The problem with my analysis in comment 71, and possibly the problems I was having with other issues, is that there are some calls to get the database that use the cache, but not the folder, so simply adding the db to the folder is insufficient.

In particular, calls through OpenMailDBFromFile (such as we see in nsMailboxUrl::GetMsgHdrForKey) do not rely on the mDatabase in the folder object, but have to have the object in the cache to prevent opening a second database object. Plus it is possible to null the mDatabase in the folder object, so that is not a reliable solution.

So once an invalid database becomes valid, there needs to be some mechanism to get it added to the cache, else subsequent callers will open a second copy of the database. I don;t see any mechanism in the current code. So in the current patch, I simply propose doing that in SetSummaryValid(true).

I will also take Irving's unit test from bug 710056 and adapt it here is a separate patch. I think that bug 710056 will end up being duped to this one.
The root cause of this and bug 710056 are the same, so I'm going to dup this to that one. However, I have been unable to get a unit test to fail for this expungedBytes issue, while the unit test in bug 710056 fails reliably for that bug's column issue. Yet I can see the expunged bytes issue very clearly in the attached test extension when I run Thunderbird, and the patch I will submit on bug 710056 fixes both issues.

I'm not entirely comfortable this, but I've spent way too much time already trying to get a unit test to fail, and I don't think further time is warranted.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → DUPLICATE
Confirming that the bug is fixed and no longer present in TB17.  Hallelujah!  And thank you.
Status: RESOLVED → VERIFIED
Assignee: rkent → nobody
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: