218477 - opening inbox view with ~10000 messages is very slow (20 seconds CPU time), locks mozilla

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[duraid]

User-Agent:       Mozilla/5.0 (X11; U; Linux ia64; en-US; rv:1.5b) Gecko/20030906
Build Identifier: Mozilla/5.0 (X11; U; Linux ia64; en-US; rv:1.5b) Gecko/20030906

When I run mozilla mail, if I hit 'read mail' (to be taken to my inbox with
~10000 entries), CPU usage hits 100% for around 20 seconds, and mozilla locks
completely. If instead I hit e.g. 'junk' (to view a folder with fewer messages),
the view (list of subjects/senders/ etc) shows up faster. It looks as if an
algorithm that was once log or constant time, is now linear in the number of
messages in a mailbox.


Reproducible: Always

Steps to Reproduce:
1. run mozilla mail
2. hit 'read' or 'inbox' or whatever to attempt to view a mailbox with many messages


Actual Results:  
mozilla locked completely, using 100% cpu time for ~20 seconds.

Expected Results:  
brought up the mailbox's view much faster

This doesn't happen with 1.3, and this problem is happening with 1.5b (built
from source)

Comment 1

[Matthias Versen [:Matti]]

-> Databse

Comment 2

[David :Bienvenu]

afaik, nothing has changed in this area. Are you using a non-default view, or
View : All? How is the folder sorted? Are you using IMAP or POP3?

Comment 3

[duraid]

Sorry for the extreme delay in getting back to this. (On the plus side, I can
confirm this bug is still present in Mozilla/5.0 (X11; U; Linux ia64; en-US;
rv:1.6a) Gecko/20031209)

I am not using a non-default view. The folder is sorted by order received, and
I'm using POP3.

Comment 4

[duraid]

I am now in the situation where I have two main boxes I read, one has ~30,000
messages and the other has ~10,000. The difference in the delay in opening each
is about a factor of 10, making me wonder if there is some O(n^2) operation
going on before any messages are displayed.

Surely it would be better to immediately display an empty view and populate it
with messages, so the first 30 or so that a user can see appear instantly, and
the display of other messages can be done in a lazy manner (when the user
scrolls), or perhaps in a background thread. I've no idea if this is what
mozilla is supposed to do or not. However, sorting the _entire_ mailbox to
determine the first ~30 messages to display seems a bit weak. Surely this isn't
the current approach?

Comment 5

[duraid]

I *believe* this bug doesn't occur with 1.4, so whatever went wrong probably did
so between 1.4 and 1.5. Not 100% sure though.. stupid memory ;(

Comment 6

[David :Bienvenu]

the sorting is extremely fast. the time to open a large folder is usually
dominated by the time to load the database. My 17K folder takes about 3 seconds
to open, in a debug build, on my 1.6MHz pentium. You must be seeing something
different from normal usage. Is it possible that the INBOX is getting reparsed
whenever you open the folder? Is there a status message that says - "building
summary file"? Are you running a 64 bit linux? That might cause some strange
edge case...how fast is your cpu? How much RAM?

Comment 7

[duraid]

I don't think the INBOX is getting reparsed - I think I know what that looks
like (there's usually a lot of disk activity and a creeping progress bar, right?)

No message about building a summary file. Yes, I'm running a 64 bit linux. Since
I first reported this bug my CPU speed has changed, but not drastically. At the
moment it is a 1.3GHz Itanium 2, the system has 2GB of RAM (i.e. plenty).

What changed in the database after 1.3, I wonder?

Comment 8

[David :Bienvenu]

nothing signficant changed in the database. I can't think of anything that
changed anywhere that would produce a 20x performance degradation. Is the folder
just as slow to open if you're sorted by subject, for example? Is it just the
first time opening the Inbox that's slow, or is it every time in a given mozilla
session? Is it possible that it's not the size of the inbox, but the fact that
you're getting new mail that's causing the slowdown? Do you use the junk mail
feature? Do you have a large training set? It's training.dat in your user
profile directory. In some cases, people have trained on so many messages that
training.dat gets to be 10's of megabytes and the first time we load it,
everything grinds to a halt. (There's already a bug about limiting the size of
this file)

Comment 9

[duraid]

Yes, it's just as slow to open regardless of sort. The slowdown/lockup happens
every time I hit 'inbox' from the front mail view. If I hit e.g. 'sent' (fast
enough) and then hit 'inbox'... it's very slow. If I keep the mail window open,
and hit e.g. 'get msgs', that's just fine.

I can certainly watch new mail come in, it's fast (it comes in _after_ the inbox
is displayed, of course.)

My training.dat is 470kB.

I guess I have to build a mozilla with symbols (stupid gentoo stripping
everything) and try to profile it. I'll try to do that today or tomorrow.

Comment 10

[David :Bienvenu]

If Mozilla is locking up, you don't need to run a profiler if you can just break
into the debugger while it's locked up and see what the stack trace is.

How big are your INBOX and INBOX.msf files? Do you ever compact your INBOX? You
might try moving away your INBOX.msf file and see if there's something about it
that's taking a very large time to load. Though if clicking on the INBOX is slow
every time, that sounds like it's the building up of the view, not the loading
of the db, because we tend to keep the INBOX db in memory...

Comment 11

[duraid]

Well, this is going to sound crazy, but I ran a "profiler" (no call graphs, just
%-time spent vs location) and, to my surprise:

all this wasted time is being spent in memmove. (!)

I'll try and get a call stack as soon as I can figure out how to debug mozilla
properly (a bit nasty since it's multithreaded ;)

Comment 12

[WADA:World Anti-bad-Duping Agency]

I have been experiencing short freeze(severarl seconds) of mail folder for these
some months and am still experiencing.
Similar bugs are opened for Thunderbird, Bug 222886 and Bug 225315. 

Secenario is usually as follows. (Win-Me)

(1) Mails are downloaded from POP3 server to Inbox by automatic download.
(2) Junk mails are moved to Junk folder by Junk Mail Control.
(3) Some mails are moved to other folders by my message filter.
(4) Mail arrival notification is displayed.
(5) Go Mail&News and try to open a folder including new mail(not Inbox usually).
    Click a folder ==> Mozilla freezes and mail list pane is kept blank.
    After short freeze, mail list pane is successufully displayed.
    No problem here after.

Since my mail folders are not so large, maximum of some thousands mails per a
folder, this short freeze is not so big problem, but flustrating.

As far as I can remenber, this short freeze occurs on only folders(sub-folders
in folder) which contains unread mails newly moved by message filter.
I did not experience this short freeze on Junk folder nor Inbox folder nor
manual mail move nor manual messege filtering nor automatic mail move by manual
Junk marking/unmarking.
And this short freeze seems to occur only when I try to read these filtered
mails just after mail arrival notification displayed.

Some guess can be done from these insidents :
(1) ".msf" re-construction is not done perfectly during mail download
    for some reasons.
(2) ".msf" re-construction is done on folder open.
      (But ".msf" re-contructuion message was never displayed in status bar.)
      (I saw this message only on Mozilla restart after system crash.       )
(3) ".msf" re-construction is also kicked by other triggers,
    for example, a kind of timer pop event, as a result of other user actions.

To reporter :

Since temporary freeze seems to depend on mail folder size, try to move all
mails in "Inbox" to other folder such as "Inbox-processed".
I belive keeping "Inbox" for download puropose only will reduce temporary freeze
impact on you.

Comment 13

[David :Bienvenu]

As I said in a private e-mail, I think I know why we're spending so much time in
memmove, and I'll try to attach a patch when I can find a fix.

Comment 14

[David :Bienvenu]

Attachment: proposed fix

this restores the old code that handled this, with a comment.

Comment 15

[David :Bienvenu]

fix checked in. will be in tomorrow's 1.7 nightly build.

Comment 16

[[e-mail]]

Would it help to do this for the kExpandAll flag too?

Comment 17

[David :Bienvenu]

that's a good idea, Neil. I'll try it.

Comment 18

[duraid]

David, thanks for patching this one up. Why was the original code removed, I
wonder? Also, apologies for the newbie question, but what happens now: will the
Thunderbird guys pick this up magically, or should I file a bug there, or...?

I'll test the patch soon and report back.

Comment 19

[David :Bienvenu]

I believe that code was removed because I incorrectly thought it was responsible
for a bug where threads ended up expanded when they shouldn't. It turns out it
wasn't.

trunk Thunderbird builds will pick up this fix for free.

Comment 20

[duraid]

Sorry for the late response, hosed my build tools a while ago and only just got
around to fixing them.

Well, good news: the bug seems to be fixed. Opening the inbox isn't
instantaneous but it is much, much faster than it was previously.

Comment 21

[David :Bienvenu]

fixed on m4 branch

Comment 22

[David Mytton]

(In reply to comment #21)
> fixed on m4 branch

This is still present in the latest Thunderbird build as per bug 
http://bugzilla.mozilla.org/show_bug.cgi?id=222886