1942554 - Better prioritise closing of idle message databases

Status: RESOLVED FIXED

(MailNews Core :: Database, task)

Description

[Geoff Lankow (:darktrojan)]

We close idle message databases to avoid having too many file handles at once.

The number of databases to keep open was set at 30 (mail.db.max_open) in 2012 which could probably be raised somewhat now. Apparently the default number of files we're allowed to open per process is 512 on Windows and we could raise that by asking, although that would include every other file handle we have open for other reasons.

If we decide to close some databases, we should prioritise better. Currently we close the least-recently-used file first, but I think we should close the smallest databases first (or some combination of both measures) as the cost of reopening very large databases is high. This would be a simple change that helps users with a lot of mail and doesn't really affect other users.

Comment 1

[Geoff Lankow (:darktrojan)]

Note to self: we'd also have to change this part of the algorithm which just closes databases unused for 5 minutes.

Comment 2

[Wayne Mery (:wsmwk)]

I vaguely remember going through this area with Irving and rkent, indeed as an attempt to be preemptive. 30 could be low by today's standard equipment and some users' profile sizes.

(In reply to Geoff Lankow (:darktrojan) from comment #1)

Note to self: we'd also have to change this part of the algorithm which just closes databases unused for 5 minutes.

IIRC that mean all databases close if there has been zero activity in 5 minutes. (except Inbox iirc). I don't recall that 5 minutes was arrived at by anything other than seat of the pants judgement.

There is also a small matter (on Macs I think) that it opens and keeps open every font file. Or did at one time.

Comment 3

[Wayne Mery (:wsmwk)]

Forgot to mention, that underlying folders of unified folders and virtual folders may also factor in a higher active folder count.

Comment 4

[Geoff Lankow (:darktrojan)]

While we're at it we should acknowledge that IsFolderOpenInWindow hasn't worked for a very long time. nsIMsgWindow.setOpenFolder, which it relies on, doesn't get called by very much at all. I thought it was broken by the 2023's front-end changes, but it looks like it's been much longer than that.

Comment 5

[Geoff Lankow (:darktrojan)]

On closer inspection, I don't think this will do what I want it to do. Databases that aren't the currently open folder are getting cleaned up by garbage collection, which defeats my wanting to keep large ones open.

Comment 6

[Magnus Melin [:mkmelin]]

See bug 1941332. nsIMsgWindow.setOpenFolder seemed a bit dead to me too, but on closer inspection it's actually called a bunch

Comment 7

[Geoff Lankow (:darktrojan)]

I think I finally understand what I'm seeing.

I can add a threshold and prevent MsgDBCacheManager closing idle databases larger than it unless there are too many databases open. If there are too many databases open I can make MsgDBCacheManager prioritise closing smaller databases.

But neither of these things prevent the database from being closed by other things. If you have the folder open in the UI and switch to another folder, the database closes. I'm sure there are other reasons. I'm thinking that instead of a service to close databases, we need a service to keep large databases open!

Comment 8

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1942554 - Better prioritise closing of idle message databases. r=mkmelin

Comment 9

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1942554 - When closing a view, keep the database open if it's large. r=mkmelin

As well as not closing large databases when they're idle, let's not close then when the user switches folder.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/92e270c554f6
Better prioritise closing of idle message databases. r=mkmelin
https://hg.mozilla.org/comm-central/rev/2b0986ab4813
When closing a view, keep the database open if it's large. r=mkmelin

Comment 11

[Geoff Lankow (:darktrojan)]

Release Note Request (optional, but appreciated)
[Why is this notable]: It should improve the performance of Thunderbird for heavy users. It could also cause problems in particular circumstances, but I don't expect it to.
[Suggested wording]: Changed: criteria for closing idle message databases.
[Links (documentation, blog post, etc)]: None, but I'll write some stuff here for people who find this bug.

---

Thunderbird has a message database file for every folder. Once a minute it checks any open databases to see when they were last used, and closes idle databases (more than mail.db.idle_limit ms since last use), so that it doesn't have problems with too many open files. If there's still more than mail.db.max_open databases open, it closes the oldest ones until that number is reached.

I've changed this. The automatic clean up will now not close idle databases if they're larger than mail.db.keep_open_size bytes. To bring the number of open databases down it is now more likely to close small databases than ones that haven't been used lately. Thunderbird now also avoids closing large databases when switching folders.

Hopefully this will reduce the need to reopen large databases, which can take a long time. You can tune the preferences to suit your situation, at your own risk, of course. Also, you can get some logging info by changing mailnews.database.dbcache.loglevel to "All".