1719072 - Compact should never run on (or near) startup

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Wayne Mery (:wsmwk)]

Steps to reproduce:

• get to a point where the compact threshold has been reached, by constantly deferring compact by picking Cancel at the compact prompt
• restart Thunderbird

Result: compact prompt occurs on startup

Expected results: Compact does not run on startup, or shortly thereafter (say within 15-20 minutes of startup)

I can consistently produce the compact prompt at startup.

We should want to do this regardless of whether the compact prompt is enabled or not, for the following reasons:

1. For performance reasons, we should provide significant time for a user to do some productive work before the performance impact of compact kicks in - on top of the high IO and other resource usage of downloading many messages, and the resulting filtering and indexing of messages

2. To avoid locking contention during this high IO period, which might cause filtering errors (although the code should already be preventing "most" filter errors) or causing incomplete compacting

3. To avoid interrupting user work with compact prompt, for those users who have decided not to suppress the prompt

4. Even if the compact threshold has not been met immediately on startup, a user can hit the threshold during startup because of message filtering, spam filtering, and manually deleting messages from Inbox.

I suspect this behavior might rarely affect the average user. But there is a higher probability of impact for users of high message traffic, and use filters. And moreso if messages tend to be large.

Can this be accomplished by changing the initial value of gtimeOfLastPurgeCheck to a negative value ??

• The value is set at https://searchfox.org/comm-central/rev/4c5a16a4e9a7d5e873e2518c743e231d9385f385/mailnews/base/src/nsMsgDBFolder.cpp#202
• The value is used at https://searchfox.org/comm-central/rev/4c5a16a4e9a7d5e873e2518c743e231d9385f385/mailnews/base/src/nsMsgDBFolder.cpp#1815

Also, I wonder whether this compact behavior will occur when waking a laptop after it has been sleeping over an hour. (the compact can't happen more than once an hour - https://searchfox.org/comm-central/rev/4c5a16a4e9a7d5e873e2518c743e231d9385f385/mailnews/base/src/nsMsgDBFolder.cpp#1822 )

Comment 1

[Thomas D. (:thomas8)]

Given the incompatibility of Compact action with some other concurring actions like Filters (see related/dependent bugs here), this bug has my full support from an UX perspective. As Wayne hinted, enterprise consumers with more and larger messages and much filtering are likely to be affected by this.

Comment 2

[Wayne Mery (:wsmwk)]

A few background notes ...

• The code which implements the compact at startup and the one hour interval is bug 17204.
  ** This was 20 years ago when message stores and disks were much smaller (and, yes, much slower), and it was also not normal to be running a laptop on battery. SSD, with its limited lifetime writes, was in its infancy.
  ** There was no real discussion of the wisdom of compact on startup in that bug report. Only a couple comments about running it at startup and about users who might not be online for long https://bugzilla.mozilla.org/show_bug.cgi?id=17204#c65 - where thunderbird is running for a very short time. There is no mention of other possible complications, of any type.
  ** The was no discussion about the one hour interval - so we don't know the reason for choosing it over a longer or shorter value. Is one hour still a suitable value? (perhaps a question for a different bug)

• https://bugzilla.mozilla.org/show_bug.cgi?id=439089#c4 broadly describes how compacting works, at the time bug 439089 was implemented: "Every local and offline folder has an expunged-byte counter, and they update this value after every delete operation. When an imap folder is selected while TB is in offline state, UpdateFolder() method starts an all-folders compacting operation via AutoCompact() method. Basically, AutoCompact goes through all the accounts and their folders to collect total expunged byte count. If it is bigger than the threshold set in the prefs, it shows a dialog to ask whether the user wants to execute the compacting op or not. If yes, it starts compacting local/pop3 and offline folders using nsFolderCompactState class that controls the compacting state machine."

Comment 3

[Wayne Mery (:wsmwk)]

We're not going to track a non-critical, non-regression bug.

Comment 4

[Tom]

Wayne, we've just duplicated status-thunderbird91 and tracking-thunderbird91 to status-thunderbird_esr91 and tracking-thunderbird_esr91 to not have to query on two different flags. BTW, after comment #1 you added tracking-thunderbird91: --- → ? yourself.

Comment 6

[Benjamin Melançon]

The feature request i came to file, and found this bug, is that Thunderbird should compact when i am inactive. Instead it reliably begins to compact the moment i return to Thunderbird, like i may have had it open and unused for three hours, and the moment i go to use it i cannot because it begins compacting. So if moving compacting to shutdown as suggested in bug 401295 works, that's great. Based on the current behavior this proposal to delay from the start would not be enough, we would have to at least have some delay from when it detects the need for compaction (since interacting with Thunderbird seems to trigger detecting) or, ideally, have the check for and compact action take place explicitly when no action has happened for five or fifteen or thirty minutes.

Bug 286888 would be dependent on a solution like this one (comments in that thread raise the idea of an option to never compact at all… if the only benefit is disk space and not faster indexing somehow then i would like that option please!)

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to Benjamin Melançon from comment #6)

Based on the current behavior this proposal to delay from the start would not be enough, we would have to at least have some delay from when it detects the need for compaction (since interacting with Thunderbird seems to trigger detecting) or, ideally, have the check for and compact action take place explicitly when no action has happened for five or fifteen or thirty minutes.

No, the idea behind this bug report is to delay initial compact by 5 minutes or more, regardless of user interraction. Except an explicit user compact request.

The feature request i came to file, and found this bug, is that Thunderbird should compact when i am inactive. Instead it reliably begins to compact the moment i return to Thunderbird, like i may have had it open and unused for three hours, and the moment i go to use it i cannot because it begins compacting. So if moving compacting to shutdown as suggested in bug 401295 works, that's great.

Bug 286888 would be dependent on a solution like this one (comments in that thread raise the idea of an option to never compact at all… if the only benefit is disk space and not faster indexing somehow then i would like that option please!)

Neither of those changes is likely to ever happen, because mbox should be on it's way out in version 114 next year, and compacting on shutdown brings it's own complications.

Comment 8

[Ben Campbell]

I just had a look into the autocompact system, and it looks like it might be an easy change to make.
The go/no-go check seems to be done by nsMsgDBFolder::AutoCompact(), which is called after messages are deleted.
And it looks like it'd be easy enough to add in a no-autocompact-within-X-minutes of startup clause to that check.

nsMsgDBFolder::AutoCompact(): https://searchfox.org/comm-central/source/mailnews/base/src/nsMsgDBFolder.cpp#1950

The trick then is to get the time-since-startup. It'll be there somewhere, I just don't know what the correct approach is!
There is a StartupTimeline START timestamp being recorded:

https://searchfox.org/comm-central/source/mail/app/nsMailApp.cpp#380

I'm not quite sure what the StartupTimeline intention is - it sounds a bit debug/telemetry-oriented, which makes me hesitant to use it.
Maybe there's a "better" timestamp available somewhere?

QUESTIONS for Geoff & Magnus:

1. Are there any other places which deal with autocompaction that I've missed (I'm thinking of JS land here)? Or is it all done in C++ via AutoCompact().
2. Where should I get a time-since-startup value? From the StartupTimeline START event? Somewhere else? Surely there's an existing example somewhere in C-C that checks how long we've been running?

Comment 9

[Geoff Lankow (:darktrojan)]

1. Are there any other places which deal with autocompaction that I've missed (I'm thinking of JS land here)? Or is it all done in C++ via AutoCompact().

I'm not aware of any. That doesn't necessarily mean they don't exist.

2. Where should I get a time-since-startup value? From the StartupTimeline START event? Somewhere else? Surely there's an existing example somewhere in C-C that checks how long we've been running?

It'd be worth checking out nsIAppStartup. There's getStartupInfo() although that's a bit C++ unfriendly, perhaps it can give some hints. We probably also should use that interface to make sure we're not compacting at silly times like during shutdown.

The user idle service might be useful. It gives a notification when the user's not done anything for a given length of time.

Comment 10

[Magnus Melin [:mkmelin]]

Agreed nsIUserIdleService could be useful.

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #2)

• https://bugzilla.mozilla.org/show_bug.cgi?id=439089#c4 broadly describes how compacting works, at the time bug 439089 was implemented:

Correction, at the time that bug 436615 was implemented, not bug 439089.

And FWIW bug 439089 originates the idea of compacting on idle.