439089 - Better Faster IMAP: Auto compact on idle time with better threshold

Status: RESOLVED WONTFIX

(MailNews Core :: Backend, defect)

Description

[Emre Birol]

Enabling offline mode for all folders by default in TB requires better auto compacting mechanism. The focus of this task will be to improve the existing implementation in 3 areas:
 o timing (on TB idle)
 o threshold 
 o divisibility (doing in multiple passes if possible)

Comment 1

[Emre Birol]

Prefs need to be changed:

pref("mail.purge_threshhold",              1024);
pref("mail.purge.ask",                     false);

Comment 2

[David :Bienvenu]

just a note - autocompacting is also (one might say primarily) used for local/pop3 folders. It's also used for offline news stores.

Comment 3

[Magnus Melin [:mkmelin]]

Bug 437657 is about the purge threshold.

Lots of related bugs - bug 205756, bug 152675, (suite) bug 188728, bug 286888. Also bug 324576.

Comment 4

[Emre Birol]

This is how Compacting works originally;

Every local and offline folder has an expunged-byte counter, and they update this value after every delete operation. When an imap folder is selected while TB is in offline state, UpdateFolder() method starts an all-folders compacting operation via AutoCompact() method. Basically, AutoCompact goes through all the accounts and their folders to collect total expunged byte count. If it is bigger than the threshold set in the prefs, it shows a dialog to ask whether the user wants to execute the compacting op or not. If yes, it starts compacting local/pop3 and offline folders using nsFolderCompactState class that controls the compacting state machine. 

Possible changes are;
1. Executing compacting operation automatically based on idle time
2. Increasing the threshold value to something like 5MB
3. Not showing the message dialog by default
4. Executing compacting operation incrementally
5. Determining the duration and gain before the operation - I think David has mentioned this one. Not sure. David can you elaborate more.
6. Implementing the notification mechanism to meet Global Database's needs.

These are two possible approaches to implement #4 and #6; 
o Implementing the logic in nsFolderCompactState
o Implementing as a separate service

I like the second option. One central location to consume IDLE service, one central location to suspend/resume compacting and one central location for the rest of the system to hook in for notifications.

Thoughts?

Comment 5

[David :Bienvenu]

Emre, this is how it works for IMAP offline stores, but you should also take into account how it works for local/pop3 folders. For them, AutoCompact is triggered after a message has been deleted. News works similarly to IMAP - in UpdateFolder, though it ignores the offline state. Just use mxr and look for AutoCompact.

To prioritize which folders to compact automatically, I would suggest looking at the ratio of the expunged bytes to the total size of the mailbox/offline store. The higher that ratio, the bigger the payoff is to compact that folder. And conversely, if we have a 1MB folder with 10k worth of expunged messages, automatically compacting that folder is not very interesting.

>I like the second option. One central location to consume IDLE service, one
>central location to suspend/resume compacting and one central location for the
>rest of the system to hook in for notifications.

I partly agree, but not completely. I think a separate service to hook into the idle service is ok, and it should handle suspect/resume of compaction. But the folder compact state object has all the knowledge of how to suspend/resume, so I believe the service should just make Suspend/Resume calls to the folder compact state, not duplicate all the logic about the where we are in the compact process. And the folder compact state should be the one that figures out the order to compact folders, etc. Also, the notifications have to happen from the folder compact state because when the user forces an explicit compact, that's going to invoke the folder compact state directly, skipping the service, which is kicked off by idle. And in fact, the folder compact state already sends a notification (see NotifyCompactCompleted).

Given that, would you still want a separate service, or would you just convert the folder compact state to a service?

Comment 6

[Emre Birol]

>..this is how it works for IMAP offline stores 
>...you should also take into account how it works for local/pop3 folders.
yes, I was scoping this task just for imap. Makes sense to cover local/pop3 as well.

>But the folder compact state object has all the knowledge of how to >suspend/resume, so I believe the service should just make Suspend/Resume calls >to the folder compact state, not duplicate all the logic about the where we >are in the compact process.
Agreed. This is exactly what I was thinking, I am not suggesting any kind of code/logic duplication.

>the notifications have to happen from the folder compact state because when >the user forces an explicit compact, that's going to invoke the folder compact >state directly, skipping the service, which is kicked off by idle. And in >fact, the folder compact state already sends a notification (see >NotifyCompactCompleted)
Ok, based on your response to Andrew on m.d.a.thunderbird, I assumed that compacting doesn't generate any notification - except progress bar stuff.

>Given that, would you still want a separate service, or would you just convert
>the folder compact state to a service?
Currently, as I understand it, every folder creates an instance of nsFolderCompactState before compacting, executes the compact op, and releases it when done. Not sure how converting it to a service would help us in this case. If we are going to resume/suspend the compacting op, we should keep a state object per folder hierarchy - lazily, at least until it is completed. Do I miss something? Appreciate if you can elaborate more.

Comment 7

[David :Bienvenu]

>Ok, based on your response to Andrew on m.d.a.thunderbird, I assumed that
>compacting doesn't generate any notification

It doesn't generate a nsIMsgFolderListener notification, but it generates an other kind of notification.

>Currently, as I understand it, every folder creates an instance of
>nsFolderCompactState before compacting, executes the compact op, and releases

nsFolderCompactState can either compact all folders, or a single folder. In the compact all case, there's still just one nsMsgFolderCompactState object - see nsFolderCompactState::CompactAll().

Comment 8

[Emre Birol]

>nsFolderCompactState can either compact all folders, or a single folder. In the
>compact all case, there's still just one nsMsgFolderCompactState object - see
>nsFolderCompactState::CompactAll().
I didn't realized that one instance of nsFolderCompactState can work on all kind of folders from any account. It is clear now. 

So, converting nsFolderCompactState into a service makes sense.  Thanks.

Comment 9

[David :Bienvenu]

(In reply to comment #8)

> I didn't realized that one instance of nsFolderCompactState can work on all
> kind of folders from any account. It is clear now. 

Sorry, yes, I should have been more clear about that. It's a bit odd, but it will be less odd when you make it a service :-)

Comment 10

[David :Bienvenu]

moving to b2

Comment 11

[David :Bienvenu]

taking, marking blocking of space/time threshold for autosync. If we ever reach the autosync space threshold, we will need to compact in order to get under the threshold. Being able to do background compact on idle would be a big win for that. I guess it doesn't technically block, but it would make for a much better user experience.

Comment 12

[Wayne Mery (:wsmwk)]

Will it make sense to target this for 3.1 time frame?

Should we expect after 3.0 that people are more likely to encounter problems related to speed of compactions due to larger local storage .. problems that also will impact "compact in progress" issues like Bug 498274 -  [Meta] Issues around simultaneous "delete/append of mail, update of mail data, read by other" and "compact folder" of local mail folder (which I doubt is a complete accounting of the issues)

Comment 16

[Tim Deaton]

I was the one who opened Bug 471937, which has now been closed into this one.  (It's very discouraging that no one other than Wayne has EVEN POSTED to this bug in TWO YEARS.)

To recap what I said in 471937:
I'm currently using Thunderbird v3.1.10 on Windows XP SP3 on a machine with an AMD quad-core processor and 4 gig of ram.  My incoming mail server is POP3.

Currently, Thunderbird compacts folders at various times while I am reading/creating/deleting emails.  There seems to be no way to control this other than the one entry under |Tools|Options|Network & Disk Space| to "Compact folders when it will save over ___ kb".  

Meanwhile, every time it compacts folders, it appears to confuse the indexing and it forgets what message I was on.  I have to quit the message to recover, and Thunderbird no longer knows which message it was: no message is highlighted.  The message I was reading will usually (but not always) be marked as read.  Most commonly this happens the first time I delete a message during a session.

My suggestion is that the default should be that compacting ONLY happens on shutting down (or perhaps when starting up) Thunderbird.  Another option could allow compacting ONLY when no other activity has occurred for one minute.  That option (along with "Compact at startup" and "Compact at shutdown") should appear in the "Disk Space" section under |Tools|Options|Network & Disk Space|.  I see no reason why compacting should ever occur except when the program is sitting idle. 


Reproducible: Always

Steps to Reproduce:
1. Open Thunderbird
2. Read & delete one message.
2a. (Compacting happens after the 1st message is deleted and the focus moves to the next message.)
3. Continue working in Thunderbird for an extended time.
3a. (Compacting reoccurs at seemingly random times.)
Actual Results:  
Thunderbird forgets its indexing, you cannot delete your current message or move between messages, and when you close that message Thunderbird doesn't tell you where in the message list you were.

Expected Results:  
Compacting should ONLY happen at unobtrusive times, like on program start-up or shutdown.

Comment 17

[Wayne Mery (:wsmwk)]

filed bug 759360 which addresses part of comment 5 / "5. Determining the duration and gain before the operation - I think David has mentioned this one...."


(In reply to Tim Deaton from comment #16)
> To recap what I said in 471937: ...
> Currently, Thunderbird compacts folders at various times while I am
> reading/creating/deleting emails.  There seems to be no way to control this
> other than the one entry under |Tools|Options|Network & Disk Space| to
> "Compact folders when it will save over ___ kb".  
> 
> My suggestion is that the default should be that compacting ONLY happens on
> shutting down (or perhaps when starting up) Thunderbird.  

this is covered by bug 401295. But if anything is done, there are more favored choices than shutdown/startup.

Comment 20

[Magnus Melin [:mkmelin]]

I don't really see us doing this. Closing.