37485 - Slow loading large IMAP folder (major regression)

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect, P1)

Description

[Jim Roskind]

I have been having trouble, and FINALLY today I got the verification build to 
run (by blowing away my moz*.* in the WinNT directory.
I have a large inbox (about 8000 messages).
I've been running about an hour now, and I've gotten about 2700 loaded.
At this rate, I'll have my mailbox ready to read in about 3 hours.
PR1 was no where near this slow.  It might have to do with scrolling of the pane 
as the messages arrive.  It appears that each new message is sorted into my 
list, and the introduction takes roughly 2-3 seconds per message (at this 2800 
point in the load :-(  ).  

I also noted that the far right column of the message list pane contains the 
"message number" of the most recently loaded message (strange to see the same 
number repeated a ton of times :-/).

I'll marke this as dogfood as soon as it gets entered.  Also note that I'm 
running at 100% CPU utilization (presumably the sorts??), and no visible disk 
activity (i.e., I'm not near thrashing yet.  I have 256Megs, and the total 
memory usage on the system is currently about 188M).

Comment 1

[Jim Roskind]

Making dogfood nomination.  My inbox is up to 3150 now... and climbing ;-)

Comment 2

[lchiang]

Is this the first time loading the folder?  (ie. no corresponding .msf file 
created in users directory)

Comment 3

[Jim Roskind]

Yes, it was the first time.  In fact, it had to migrate over my prefs from 4.x 
just before doing the load.  It took me a while to realize that it had migrated 
over the "use SSL" which was not functioning.  Once I turned that off, I was 
able to see more than the INBOX (it loaded my whole list of folders).  I then 
asked that it try to load my INBOX... and that was when the problem appeared :-(

Comment 4

[David :Bienvenu]

Jim, it's useless to assign dogfood bugs to me - I've been sick for a week and 
might be out for another week. Also, I can load a 3000 msg folder in about 10 
seconds. I noticed yesterday that downloading headers was extremely slow, but 
seems fine today, all without the magic of any imap changes. My suspicion is 
some RDF/tree control algorithm has gone N2. I've noticed that deleting a 
message in a large folder has become painfully slow, for both local AND imap, 
whereas deleting a message in a small folder behaves normally.

Comment 5

[Chris Waterson]

I've botched updates on threading counts. See bug 37290. This should cause grief 
once you start to read messages.

Comment 6

[Chris Waterson]

taking bug...my fault.

Comment 7

[Chris Waterson]

beinvenu is correct re: O(n^2) in RDF. see...

http://lxr.mozilla.org/mozilla/source/rdf/content/src/nsXULTemplateBuilder.cpp#4
827

This code gets run each time a new message is added to the folder, and will 
cause every *other* message to be re-iterated. Zoiks! I'm working on a fix.

Comment 8

[Chris Waterson]

oops, dropped bienvenu from cc list...

Comment 9

[scottputterman]

adding myself to cc

Comment 10

[David :Bienvenu]

*** Bug 17470 has been marked as a duplicate of this bug. ***

Comment 11

[leger]

Putting on [dogfood+] radar.  ccing suresh. SURESH, what does your latest perf 
testing in this are show on an NT box?

Comment 12

[Suresh]

Jan, Using 2000-05-02-09-M16 build on Win NT (400 Mhz, 320 MB RAM) time taken to
load a folder with 6650 messages(no summary file) is 63.92 seconds.

Comment 13

[Chris Waterson]

Checkin for 37290 fixed the O(n^2) problem. I'm marking this fixed, as my 300 
Mhz machine is showing load times of ~5s for a 6,000 message local folder.

Comment 14

[lchiang]

suresh - can you put a new time in here for today's (or yesterday's) build and 
verify? Thanks.

Comment 15

[Suresh]

Using 5/22 commercial build on Win NT (400 Mhz, 320 MB) time to load 7275 IMAP
folder (no summary file) is 34.58 seconds, which is ~50% improvement from my
previous timing. :)

Marking as Verified.