101315 - summary file creation is very slow, speedup possible

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Message Display, defect, P1)

Description

[Stefan Magdalinski]

build id 2001091903

not sure if this is a backend, database or frontend problem.

I've noticed a possible great improvement in mailnews summary creation performance:

after deleting a summary file on a folder with 4260 messages, rebuilding on my
system took 270 seconds.

repeating the procedure, but immediately selecting a different folder once the
rebuild starts, and the rebuild takes only 90 seconds.
looks like the endless repaints on the message list pane takes 2/3 of the time!

Comment 1

[Stefan Magdalinski]

I've been trying to do some informal benchmarking of how much time this could save. 

I don't know about anyone else, but I seem to drive mailnews quite hard. Got
maybe  30 folders with 500-3000 messages in each. And the program seems to need
to rebuild  its indexes from scratch from all the time? what is the criteria for
saying do a rebuild? 

my experience is that would make a big perceptual difference to how much time I
spend thumb-twiddling waiting for mozilla to come back to me, over the course of
a working day.

Comment 2

[scottputterman]

adding some more developers to this.  Any thoughts on this and if you agree is
there anything that can be done about the extra painting?

Comment 3

[Navin Gupta]

We build summary file way too often, and i have seen this because 
i use pop3 as my primary acct. I will try to log some bugs to track this 
problem. 

Comment 4

[David :Bienvenu]

yes, we could avoid displaying any headers until the folder was reparsed. Navin
is right; there are situations where we were leaving the db out of sync - one
was mark all read, which has since been fixed. But we have to reparse the db
once it gets out of sync or we'd be very vulnerable to data loss.

Comment 5

[scottputterman]

Navin, were you able to log some bugs about reparsing the summary file too
often? Those might be worth looking into since it'll improve folder loading perf
in those cases.

Comment 6

[David :Bienvenu]

Be aware that running 4.x and 6.x/mozilla against the same profile will
guarantee lots of reparsing, and there's nothing we can do about that.

Comment 7

[scottputterman]

right, I don't care about the 6.x/4.x case.  Navin's and your comments made it
sound like we are getting out of sync in other cases that we might be able to
prevent.

Comment 8

[Julien Pierre]

Adding myself to this bug. I too am hitting hard performance problems with mail
using POP3. I have about 500 MB of personal mail at work, and another 500 MB at
home. The constant rebuilding of index files is very painful. Some of my folders
have 5000 to 50000 messages in them and it just takes forever ... When I do it
in 4.x it is an order of magnitude faster, and the reindexing only appears when
the db gets corrupt somehow, which is very rare.

In recent builds of mozilla (at least 0.9.5 and 0.9.6 on os/2), my folders
*always* get reindexed when they are first accessed in the mail application
after the brower has started. If I just exit mozilla , restart it, and go back
to the same folder, it reindexes it. This problem has forced me to stay with
4.61 for os/2 at home instead of mozilla 0.9.x . It was not always that bad in
mozilla, it seems that the reindexing problem has been getting much more
frequent over the past few releases.

FYI, I am not experiencing that constant reindexing bug in the Netscape 6.2
release on Windows on my work machine. So it may be a post 0.9.4 regression.
I also want to add that I'm not sharing profiles between 4.x / 6.x at all
(either at home on os/2 or at work on windows). I always have a separate
migrated profile for mozilla/netscape6. But it is only for testing and I always
"leave message on server" in them, with the actual remove always occuring in the
4.x browsers, until I pick up the next build of mozilla/netscape6 and re-migrate ...

Comment 9

[David :Bienvenu]

*** Bug 90541 has been marked as a duplicate of this bug. ***

Comment 10

[David :Bienvenu]

Attachment: db part of fix

this patch adds a method that clients can use to see if a db is valid. The sole
client (patch upcoming) is for local mail to know if a folder is getting
reparsed, but I can imagine other clients in the future. The other dbs (beside
local mail) will probably want to check the version but I'll do that later.

Comment 11

[David :Bienvenu]

Attachment: check if folder is being reparsed before sending folder loaded notification

Comment 12

[(not reading, please use [e-mail] instead)]

Comment on attachment 60868 [details] [diff] [review]
db part of fix

sr=sspitzer

Comment 13

[(not reading, please use [e-mail] instead)]

Comment on attachment 60869 [details] [diff] [review]
check if folder is being reparsed before sending folder loaded notification

sr=sspitzer

any reason not to put 
"rv = mDatabase->GetSummaryValid(&valid))"
on the line above,  it makes debugging easier.

Comment 14

[Navin Gupta]

I don't see how this will speed up folder loading. it may solve the problem
that I was trying to fix in some other bug using that parsingFolder boolean

Comment 15

[Navin Gupta]

s/folder loading/ building summary file/

Comment 16

[David :Bienvenu]

It's complicated, Navin, but believe me, it cuts the time by 10x or more.
Essentially, we no longer send notifications to the front end to adjust the
counts on each message we reparse. This is in conjunction with the fix for bug
112761, which relies on us not sending a folder loaded notification before we're
done parsing. Once the FE receives a folder loaded notification, it turns on all
the count updating.

Comment 17

[David :Bienvenu]

I believe this was slow because we were updating the msg counts, not the thread
pane.  The checked in patch speeds it up dramatically for me.

Comment 18

[Navin Gupta]

sorry for the delay, r=naving, but I was just trying to figure out how we 
are saving much because we are just preventing one NotifyFolderEvent from
being triggered.

Comment 19

[David :Bienvenu]

like I said, it's complicated, but if we call StartFolderLoading, it turns off
notifications, until EndFolderLoading gets called, and sending a folderloaded
notifiation makes EndFolderLoading get called.

Comment 20

[[e-mail] (gone - use [e-mail] instead)]

Verified FIXED using build 2001-12-16-08 and a build from the 7th to compare
behaviors.

Setup: 8,027 message local folder, deleted the .msf for that folder and then
clicked on the folder, timed how long it took to fully load the folder's
contents and stop the meteors.

12-16         |   12-07
------------------------
20.80 seconds | 50.90 seconds

This was on a really fast machine, so we cut building (and I'm assuming parsing
too, since after building we won't have the message headers cached) the summary
file by 59% on a Pentium 733 with 384 megs of RAM.