347837 - Slow switching between (loading) a largely-populated folder - change to lazy db closes during idle/less aggressive closing of dbs

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Adam Guthrie]

I have a folder on a POP account that contains about 50,000 e-mails. When I get new e-mail and I am still in this folder I like to switch to another folder and then back to it so the threads get pushed to the top.

Lately, I've noticed that this takes /a lot/ longer than it used to. I have to wait almost 1-2 full seconds before the folder is "selected".

I think I've tracked down the regression range for this to between 2006-07-21 04:00 and 2006-07-22 04:00.

Checkins: http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=ThunderbirdTinderbox&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2006-07-21+03&maxdate=2006-07-22+05&cvsroot=%2Fcvsroot

I'm not 100% this is when it regressed, because it's sometimes sporadic, although the fact that bug 345251 landed during this time makes me somewhat more sure.

If anyone needs a large folder to test this with I can send them mine (it's bugmail).

Comment 1

[Adam Guthrie]

Attachment: stack from the hang?

I can manage to attach to the process when it's hanging. It looks like it's "hanging" in mork code.

Maybe I could do some profiling and see what functions are being called the most. I'm not sure how to do this, though.

Comment 2

[David :Bienvenu]

Adam, I suspect the issue is that we're opening the db at all in this situation, and not any particular slowdown in open a db for 50K messages.

When you leave a folder, we null out the cached db pointer in the folder, and eventually the db will get closed (I say eventually because sometimes js will be holding onto the db until gc, and there are a couple other things that temporarily hold open dbs, like reading a message in the message pane seems to hold a db open until a new message is read). I suspect what used to happen in your scenario was that when you clicked back on the original folder, the db was still open, so opening the folder was faster. Now, it appears that the db is getting closed. You might be able to verify this by using some unix tool to see if the file handle for the .msf file is still open after you switch folders or not.

In a way, we'd rather that db's get closed sooner rather than later, though it's a bit of a drag for your usage scenario.

Comment 3

[Wayne Mery (:wsmwk)]

Adam, Threadbubble extension will bring threads with newest message to the top.

http://micropipes.com/code/threadbubble/

https://bugzilla.mozilla.org/show_bug.cgi?id=262319 

Comment 4

[Wayne Mery (:wsmwk)]

David, are folders closed based on LRU?  Is it possible to age them based on size, so that bigger folders are somewhat less likely to be closed?

Comment 5

[David :Bienvenu]

No, we close db's aggressively (or, rather, clear out folders' cached db's aggressively, which usually releases the final ref to the db, and closes it), because there's no central piece of code to cleanup cached/open db's.

I think using the idle service to close db's that haven't been touched recently is an interesting idea. I think we'd need to keep track of the last use time in the db, since the folder's time stamp isn't quite the same - the latter tries to keep track of user interactions with the folder, and there are db uses that don't really correspond to user interactions.

Currently, the folders cache their db's and code throughout the product knows to clear the folder's cached db pointer in order to avoid holding the db open in memory. In some cases, it's very important that this happen right away (e.g., there are components that iterate over all the folders, opening their db's, doing something, and then clearing the folder's cached db pointer - if they didn't clear the cached db immediately, we'd have runaway resource usage). I think the easiest way forward would be to have some code triggered on idle that iterates over folders, releasing cached db pointers for db's that haven't been used recently, and aren't open in the UI. And then go through the code removing calls to nsIMsgFolder::SetMsgDatabase(nsnull) which aren't needed if we can close db's on idle. I suspect only a handful of calls would actually end up getting removed, e.g., when you click away from a folder, or when a message is moved/copied into a folder, it's OK to leave those db's open for a while under the assumption that we might need that db open again relatively soon.

Currently, we leave all INBOX and Trash db's open because we figure those are going to be used most often. I could definitely see relaxing this if we had some cleanup task.

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to comment #4)
> David, are folders closed based on LRU?  Is it possible to age them based on
> size, so that bigger folders are somewhat less likely to be closed?

(In reply to comment #5)
> No, we close db's aggressively (or, rather, clear out folders' cached db's
> aggressively, which usually releases the final ref to the db, and closes it),
> because there's no central piece of code to cleanup cached/open db's.
> 
> I think using the idle service to close db's that haven't been touched recently
> is an interesting idea. I think we'd need to keep track of the last use time in
> the db, since the folder's time stamp isn't quite the same - the latter tries
> to keep track of user interactions with the folder, and there are db uses that
> don't really correspond to user interactions.
> 
> Currently, the folders cache their db's and code throughout the product knows
> to clear the folder's cached db pointer in order to avoid holding the db open
> in memory. In some cases, it's very important that this happen right away
> (e.g., there are components that iterate over all the folders, opening their
> db's, doing something, and then clearing the folder's cached db pointer - if
> they didn't clear the cached db immediately, we'd have runaway resource usage).
> I think the easiest way forward would be to have some code triggered on idle ...

With much work on idle and aggressive db closes happening in other bugs, is this idea made more feasible, or subsumed by the work in other bugs?

Comment 7

[Kent James (:rkent)]

David in comment 2 said "I suspect the issue is that we're opening the db at all in this situation, and not any particular slowdown in open a db for 50K messages"

That implies that opening the database is independent of folder size. I have not tested it, but I doubt that that is true. Is that really what you meant David, and if so are you sure it's true?

Also, I don't know what Waye is referring to in comment 6 with "much work on idle and aggressive db closes happening in other bugs"

Comment 8

[David :Bienvenu]

Kent, the assertion was that it had gotten a lot slower to open a 50K folder, and I was asserting that I don't think it had gotten slower, not that opening a database is independent of folder size (or, more accurately, the number of messages in the folder).

Opening a database is indeed directly proportional to the number of messages in the folder, and has been for a very long time.

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to comment #7)
> Also, I don't know what Waye is referring to in comment 6 with "much work on
> idle and aggressive db closes happening in other bugs"

this refers to compact, gloda and imap bugs which are improving idle detection/usage and/or db close behavior related to background tasks - as opposed to rather simpler user UI actions of moving messages and switching folders.

Comment 10

[Wayne Mery (:wsmwk)]

morphing this bug and adding "change to lazy db closes during idle/less aggressive closing of dbs" to summary

I'm not adding to the summary how we decide which dbs to close - whether it's strictly LRU, size weighted, etc.

Comment 11

[WADA:World Anti-bad-Duping Agency]

User can keep Db open, by keep folder open.
  Folder context menu, Open in New Tab(or Open), instead of folder switch.
As Db is kept open if folder is opened at a tab or a window, thread pane is immediately shown by tab switch or window switch. This can be a workaround by user.