summary file creation is very slow, speedup possible

VERIFIED FIXED in mozilla0.9.7


MailNews: Message Display
17 years ago
14 years ago


(Reporter: Stefan Magdalinski, Assigned: Bienvenu)




Firefox Tracking Flags

(Not tracked)



(2 attachments)

5.53 KB, patch
(not reading, please use instead)
: superreview+
Details | Diff | Splinter Review
809 bytes, patch
(not reading, please use instead)
: superreview+
Details | Diff | Splinter Review


17 years ago
build id 2001091903

not sure if this is a backend, database or frontend problem.

I've noticed a possible great improvement in mailnews summary creation performance:

after deleting a summary file on a folder with 4260 messages, rebuilding on my
system took 270 seconds.

repeating the procedure, but immediately selecting a different folder once the
rebuild starts, and the rebuild takes only 90 seconds.
looks like the endless repaints on the message list pane takes 2/3 of the time!
Ever confirmed: true
Keywords: perf


17 years ago
Blocks: 63759

Comment 1

17 years ago
I've been trying to do some informal benchmarking of how much time this could save. 

I don't know about anyone else, but I seem to drive mailnews quite hard. Got
maybe  30 folders with 500-3000 messages in each. And the program seems to need
to rebuild  its indexes from scratch from all the time? what is the criteria for
saying do a rebuild? 

my experience is that would make a big perceptual difference to how much time I
spend thumb-twiddling waiting for mozilla to come back to me, over the course of
a working day.

Comment 2

17 years ago
adding some more developers to this.  Any thoughts on this and if you agree is
there anything that can be done about the extra painting?

Comment 3

17 years ago
We build summary file way too often, and i have seen this because 
i use pop3 as my primary acct. I will try to log some bugs to track this 
QA Contact: esther → stephend

Comment 4

17 years ago
yes, we could avoid displaying any headers until the folder was reparsed. Navin
is right; there are situations where we were leaving the db out of sync - one
was mark all read, which has since been fixed. But we have to reparse the db
once it gets out of sync or we'd be very vulnerable to data loss.
Assignee: sspitzer → bienvenu


17 years ago
Keywords: nsbeta1

Comment 5

17 years ago
Navin, were you able to log some bugs about reparsing the summary file too
often? Those might be worth looking into since it'll improve folder loading perf
in those cases.

Comment 6

17 years ago
Be aware that running 4.x and 6.x/mozilla against the same profile will
guarantee lots of reparsing, and there's nothing we can do about that.

Comment 7

17 years ago
right, I don't care about the 6.x/4.x case.  Navin's and your comments made it
sound like we are getting out of sync in other cases that we might be able to

Comment 8

17 years ago
Adding myself to this bug. I too am hitting hard performance problems with mail
using POP3. I have about 500 MB of personal mail at work, and another 500 MB at
home. The constant rebuilding of index files is very painful. Some of my folders
have 5000 to 50000 messages in them and it just takes forever ... When I do it
in 4.x it is an order of magnitude faster, and the reindexing only appears when
the db gets corrupt somehow, which is very rare.

In recent builds of mozilla (at least 0.9.5 and 0.9.6 on os/2), my folders
*always* get reindexed when they are first accessed in the mail application
after the brower has started. If I just exit mozilla , restart it, and go back
to the same folder, it reindexes it. This problem has forced me to stay with
4.61 for os/2 at home instead of mozilla 0.9.x . It was not always that bad in
mozilla, it seems that the reindexing problem has been getting much more
frequent over the past few releases.

FYI, I am not experiencing that constant reindexing bug in the Netscape 6.2
release on Windows on my work machine. So it may be a post 0.9.4 regression.
I also want to add that I'm not sharing profiles between 4.x / 6.x at all
(either at home on os/2 or at work on windows). I always have a separate
migrated profile for mozilla/netscape6. But it is only for testing and I always
"leave message on server" in them, with the actual remove always occuring in the
4.x browsers, until I pick up the next build of mozilla/netscape6 and re-migrate ...


17 years ago
OS: Windows ME → All
Hardware: PC → All

Comment 9

17 years ago
*** Bug 90541 has been marked as a duplicate of this bug. ***


17 years ago
Keywords: nsbeta1 → nsbeta1+
Priority: -- → P1
Target Milestone: --- → mozilla0.9.7

Comment 10

17 years ago
Created attachment 60868 [details] [diff] [review]
db part of fix

this patch adds a method that clients can use to see if a db is valid. The sole
client (patch upcoming) is for local mail to know if a folder is getting
reparsed, but I can imagine other clients in the future. The other dbs (beside
local mail) will probably want to check the version but I'll do that later.

Comment 11

17 years ago
Created attachment 60869 [details] [diff] [review]
check if folder is being reparsed before sending folder loaded notification
Comment on attachment 60868 [details] [diff] [review]
db part of fix

Attachment #60868 - Flags: superreview+
Comment on attachment 60869 [details] [diff] [review]
check if folder is being reparsed before sending folder loaded notification


any reason not to put 
"rv = mDatabase->GetSummaryValid(&valid))"
on the line above,  it makes debugging easier.
Attachment #60869 - Flags: superreview+

Comment 14

17 years ago
I don't see how this will speed up folder loading. it may solve the problem
that I was trying to fix in some other bug using that parsingFolder boolean

Comment 15

17 years ago
s/folder loading/ building summary file/

Comment 16

17 years ago
It's complicated, Navin, but believe me, it cuts the time by 10x or more.
Essentially, we no longer send notifications to the front end to adjust the
counts on each message we reparse. This is in conjunction with the fix for bug
112761, which relies on us not sending a folder loaded notification before we're
done parsing. Once the FE receives a folder loaded notification, it turns on all
the count updating.

Comment 17

17 years ago
I believe this was slow because we were updating the msg counts, not the thread
pane.  The checked in patch speeds it up dramatically for me.
Last Resolved: 17 years ago
Resolution: --- → FIXED

Comment 18

17 years ago
sorry for the delay, r=naving, but I was just trying to figure out how we 
are saving much because we are just preventing one NotifyFolderEvent from
being triggered.

Comment 19

17 years ago
like I said, it's complicated, but if we call StartFolderLoading, it turns off
notifications, until EndFolderLoading gets called, and sending a folderloaded
notifiation makes EndFolderLoading get called.
Verified FIXED using build 2001-12-16-08 and a build from the 7th to compare

Setup: 8,027 message local folder, deleted the .msf for that folder and then
clicked on the folder, timed how long it took to fully load the folder's
contents and stop the meteors.

12-16         |   12-07
20.80 seconds | 50.90 seconds

This was on a really fast machine, so we cut building (and I'm assuming parsing
too, since after building we won't have the message headers cached) the summary
file by 59% on a Pentium 733 with 384 megs of RAM.
Product: Browser → Seamonkey
You need to log in before you can comment on or make changes to this bug.