Closed Bug 172786 Opened 22 years ago Closed 21 years ago

Building mail summary file takes forever

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rob, Assigned: Bienvenu)

References

Details

Attachments

(2 files)

partial fix
3.10 KB, patch
Details | Diff | Splinter Review
don't limit ODA to 16K
967 bytes, patch
darin.moz
: review+
sspitzer
: superreview+
sspitzer
: approval1.4b+
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.1) Gecko/20020826
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.1) Gecko/20020826

Sometimes, when I have to shut Mozilla down and restart, it can take 5-10
minutes to build the summary file for my Inbox. 

There's more examples of users complaining about this here:
http://groups.google.com/groups?q=mozilla+building+summary+file&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=353D2505.C8304A20%40netscape.com&rnum=3
http://www.geocrawler.com/archives/3/116/1999/2/0/592662/

Reproducible: Sometimes

Steps to Reproduce:
1.Open mail window
2.Try to get mail or click on inbox (other folders don't seem to have this
problem - I have only 784 messages in my inbox)


Actual Results:  
3.Get "Building mail summary file..." for 5-10 minutes and hard drive crunches
away... it's naptime

Expected Results:  
Summary file should already be there and be built. Even if it's not or is
corrupted, it should take that long to build. There have been times where I see
each message come into the inbox and it takes about 1 second per message. That
is awful!

 I have a 1.4GHz Athlon and reasonably fast hard drive.
This message are VERY old. (1998 and 1999).
Mozilla doesn't rebuild the .msf files unless they are corrupted or deleted.
I don't have this problem with my bugzilla mail folder with 11250 messages.
(Athlon 1.3Ghz and a fast HDD).

A complete rebuild needs of course it's time but you should not get this unless
something special happens (i never got this).

Have you ever compacted the folders (a deleted message is only marked as deleted
and will only physical deleted if you compact the mail folder) ?
How big is your mail file in your profile for that folder ?

Just checked main "mail" folder sizes and it's 443MB, which is pretty big. Did
an "empty trash" and it's down to 154MB, which is more reasonable and expected.

My in-box folder is 128MB, where 95MB of that is a set of e-mails with 1MB
attachments that I need to hang onto. I don't think there's anything too wild
about that.

I'm not sure if emptying the trash will fix the odd "building file..." or not.
One point I was hoping to make with the old '99 and '98 messages is that this is
not a new issue, and it can still happen, and still causes problems. I will be
watching for the situation to occur again and add more detail as I encounter it.
I don't restart Mozilla that often - once every many days, usually after it
crashes. It could be that it's corrupting the in-box file when it crashes. I
don't know.
It could be the problem that you have all this big files in one folder.
Mozilla will only close the folder file if you don't use it.
If you look in the inbox the whole time and mozilla crashes it's possible that
mozilla must rebuild the .msf file.
If the .msf file is open while Mozilla crashes it must rebuild the .msf file or
you could get corrupted mail folders.
Mozilla must read the complete mail file to generate the msf and with 450MB it
is of course not very fast.

but i don't more details but bienvenu should know more.
QA Contact: gayatri → esther
yes, we have to read through the whole file to regenerate the summary file, so
if it's huge, we're going to spend a lot of time in just diskio.

we shouldn't have to regenerate summary files very often. Do you have your inbox
on a file server, or on your local machine? If the former, that would explain
both  why your .msf files are getting out of date, and why it's so slow to
regenerate them.

there's also a bug where we update the message display while regnerating the
.msf files, which we should fix, but I doubt that's what's taking so long.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Local machine.

This would fix the problem:
Store attachments outside of the "Inbox" file (or folder file) so that
rebuilding the .msf file doesn't require scanning through MBs and MBs of
attachment data.

It is entirely useless to scan 50 or 100MB worth of attachments when you are
just trying to get the header information for 700 messages. That's the only
reason I can imagine for such slow performance during rebuilding. Instead, make
1 file (or a separate directory) for attachments and one for actual messages, so
my 100MB of attachments don't slow down the rebuilding of this inbox.msf file
that does get corrupted from time to time.
we use the berkeley mailbox format for compatibility with other mail readers -
it's a standard mailbox format, but it requires us to store the attachments
inline. 5-10 minutes is still an incredibly long time to read through a 128 MB
file - it takes a fraction of that time on my machine (1.6 GHz with a reasonably
fast hard drive). So I suspect something else is contributing to the slowness,
besides the size of the file. Things like a virus checker, or horrible disk
fragmentation.
I don't have a virus checker. Could be some fragmentation. 

OK, clear on the format standard. I give up for now.
you might defrag. I'll do some tests here on a file that size when I get a
chance for comparisons sake.
I just upgrade to Mozilla 1.2 from Netscape 4.8. Building email summary takes
AGES now, compared to old Netscape.

Since mail folders are the very same, and bout programs (NS and Mozilla) read
the entire folders in order to rebuild summary, i'd expect this operation takes
about the same time. It doesn't seem the case, nevertheless.

I'd say about 1:2 performance hit for Mozilla.
This has happened to me several more times since I posted this boog. I've
defragged the hard drive and doesn't help much. What appears to happen is:

1. Mozilla / Mozilla Mail crashes
2. Getting back into Mozilla after the crash, the mail file index is rebuilt -
which takes several minutes for probably a few hundred e-mails. I'm talking full
hard drive running crazy while I sit and watch the messages enter the index 1 by
painful 1. Each only takes a second or less, but I say again that this is way
too slow. It needs to be tuned up.
Rob and Jesus, have you tried 1.3a? It should be quite a bit faster than 1.2 for
rebuilding the summary files.
I have not yet. I will get that installed soon and report findings.
MS Windows 98.

Mailbox: 12.5 MB

Rebuild mailbox database:

- Netscape 4.8: 15 seconds.

- Mozilla Build 2002122208: 45 seconds

:-((

*** Bug 186902 has been marked as a duplicate of this bug. ***
100% reproducible on Linux, build 2003012322

Environment:
OS: Mandrake Linux 9.0
Hardware:
  Pentium III 666 MHz,
  512 MB RAM,
  hard disk storing the Mozilla profile: SAMSUNG SV2042H, ATA DISK drive (20411
MB in size), UDMA 100

A folder in Local Folders has 38M in size. Its name is 
"NMAIL Message Notifications". For your testing you can download a gzipped
archive (3.8M) here:
<http://olo.office.altkom.com.pl/domowa/qa/mozilla/2003_01_24_Large_Mail_Folder_loading/>

Mozilla's behaviour:
When I enter "Local Folders/Admin/NMAIL Message Notifications", Mozilla starts
to build the summary file. Mozilla consumes all available CPU time. After a long
time (~20 minutes) progress bar eventually stops moving. After some more time
Mozilla hangs completely and even doesn't update its window when I switch from
other apps.

The summary file has a size of 0 bytes all the time.

BTW, I have a message filter rule that moves messages to this folder, and new
messages keep coming very frequently - about 5 msgs per 1 minute during peak
hours - those are admin notifications from a 500 user mail server.
*** Bug 182907 has been marked as a duplicate of this bug. ***
BTW I've placed more testcases at the location from comment #15, all are
compressed archives containing versions of the big folder I have trouble with.

The latest ones will be compressed with bzip2 instead of gzip - now they only
take 1.9MB.
ok i get this problem too, on both win98 se and linux, but its a tad slower to
build on linux (both have mozilla 1.3 release), i should mention that i
symlinked the 'Mail' directory from windows to linux so i use the same folders
for both... i dont know if that could be a problem, because it seems to build it
everytime i reboot computer.. maybe every time i switch OS it thinks the summary
file is not up to date ?!?!

btw, i have about 300 messages, and it takes about a minute to build summary
file for inbox, size of Inbox file: 26309K

how do i prevent it from rebuilding every time?

Nehal
yes, the symlink is what's causing the problem. The timestamps stored in the
.msf files aren't agreeing with the timestamp of the file on the disk,
presumably becausee the two os's aren't in sync as far as the times are concerned.
so what would be the fix? (other than not symlinking, i would like to use common
folders)
I would like to lend my support to this bug report.  I get "Rebuilding summary
file" at least a few times a week (probably because Mozilla or Windows hangs
just as often).  It takes eons to complete.  When it's done, there is no visible
difference, but OK, maybe it's repairing something.  Surely, there must be a way
to accelerate this.  Maybe the answer is not to let the mail file get so big so
quickly in the first place (mine is well under 100MB, and it still takes several
minutes on a 1999 PC).  Perhaps the answer is to index the mail file and only
rebuild those parts of it which require rebuilding.  Please, someone optimize
this.  Thank you.




I am sitting here right now waiting. I have a "backups" folder off of my inbox
that has serveral hundred e-mails with attached files. The attached files are
1-2MB apiece. For some reason, it is taking 1 second PER e-mail to rebuild the
index. This is ridiculous. 

For some reason, Mozilla e-mail is so incredibly unstable that I can't run my
e-mail for more than a few days without needing to rebuild all of my indexes.
This is a complete mess.
In case it wasn't clear, it seems like the time it takes to index somehow
depends on the size of the attachment, which seems wrong to me.
parsing a mail folder/mail message is dependent on the size of the message - we
have to read through the message to find the beginning of the next message. 1MB
a second is still slow, however.

My summary files are never invalidated. There seems to be something about some
people's configurations or useage models that invalidate summary files. If I had
reproducible steps on my machine that caused this bug, I'd have a much better
chance to fix it.
can someone tell me how it decides whether it should rebuild or not, 
so i can fix for my problem (comment #19), does it compare dates 
of files or what?

thankyou, Nehal

Fixing bug 58308 is likely to provide a satisfiable fix to this bug.

Adding bug 58308 to deps.
Depends on: 58308
I think the problem is not frequent rebuilds but the time spent doing them.
Rebuilding a mailbox under old Mozila 4.8 is THREE times faster. The very same
mailbox! :-(. See comment #13

This issue and general inestability in email/news component keeps me using NS
4.8 for email. Mozilla is superbe as a browser, nevertheless.

And no, Maildir is not an option for me. I'd like to see Mozilla reindexing a
mailbox, at least, as fast as NS 4.8 :-). It's clearly doable, since NS 4.8 is
already doing it :-).

The limiting factor should be HD bandwidth, not CPU or memory. Currently HD
transfer rates of 20 MB/s are "normal".
mozilla decides to rebuild the summary file based on two criteria - the last
modified time, and the size of the mailbox file. When we make a change to the
mailbox file, we sync the file and then get the last changed time and the file
size, and write it to the .msf file, and save the .msf file. The next time we
open the .msf file, we compare the size and last modified time of the mailbox
file with what we have in the .msf file. If either doesn't match, we assume an
external agent has changed the mailbox file, and thus our summary file is
invalid. Outside of Mozilla crashing while writing out the .msf file, this
should be relatively robust. 

There are a few possibilities for the .msf file getting out of sync:

1. Some other program (e.g., another e-mail program, a virus checker, etc.) is
changing the last modified date or file size of the mailbox.
2. Mozilla is changing the file but not writing out the new last modified date
to the .msf file for some operation.
3. The OS is reporting one last modified date when we ask for it after making a
change, and then reporting a different last modified date later when we open the
file again.

Daylight savings time seems to cause this on some OS's. Having your mailboxes on
a networked drive can also cause this problem.
I have a patch that helps somewhat, which I'll attach. But I believe the real
reason we're slower than 4.x is that nsInputStreamPump::OnStateTransfer() limits
itself to 16K when sending OnDataAvailable - if I comment out this check, I get
speeds similar to 4.x. I'll investigate more, and talk to someone (Darin?) about
this:

            // XXX need to make max ODA size configurable
            if (avail > 16384)
                avail = 16384;
Status: NEW → ASSIGNED
Attached patch partial fix — — Splinter Review 
this patch makes it so we delay displaying the thread pane when reparsing a
local folder (so we don't waste time updating the thread pane), and I removed a
status update that wasn't correct because contentlength was always 0 and slowed
us down as well.
this patch removes the 16K limit on ODA data - this speeds up reparsing quite a
bit. I'm going to run with it a bit to make sure it doesn't break anything.
the reason Aleksander's mail folder takes so long to parse is that it's
basically one giant thread, if we thread by subject, and our code that adds
messages to threads breaks down performance-wise when the threads are huge.

If you turn off threading by subject without_re, it should be a lot faster. I'll
try to figure out a way to improve this code. 
*** Bug 196607 has been marked as a duplicate of this bug. ***
Darin, I haven't had any problems with the patch that removes the 16K throttle -
does it look OK to you? Are there any tests you want me to try? I'd like to
consider checking this in for 1.4, or does that seem crazy?
1 folder 68 MB (16916 messages, 500 has small text attachements) all same two
subjects:
Ask A Question and Re: Ask A Question
Just imported from Outlook Express
Summary took 100% CPU for 4 hours
XP pro Pentium III 600MHz 7200rpm new harddisk never used
1.4a Build 2003040105
I have a sent folder of 10000 messages I want to do the same to. Should I wait
so it can be done using a psossible patch? Or will deleting the msf file be
enought for testing?

Michael, see comment 33 - http://bugzilla.mozilla.org/show_bug.cgi?id=172786#c33
- I have a patch that will allow you turn off threading by subject without re:,
with a hidden pref, that I will check in when the tree opens again for general
development.
Comment on attachment 121077 [details] [diff] [review]
don't limit ODA to 16K

r=darin (without this check, max ODA is 64k... and i think that should be ok
and probably good for most stream listener implementations)
Attachment #121077 - Flags: review+
Comment on attachment 121077 [details] [diff] [review]
don't limit ODA to 16K

sr/a=sspitzer
Attachment #121077 - Flags: superreview+
Attachment #121077 - Flags: approval1.4b+
fix checked in for 1.4 final - should speed up folder parsing substantially.
Status: ASSIGNED → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED
Tp on linux and mac improved quite a bit!  :-)

btek (linux)
----------------------
Tp before: 1078
Tp after:  1017   ( -61ms, or 5.67% improvement )

silverstone(mac osx)
----------------------
Tp before:  593
Tp after:   539   ( -54ms, or 9.1% improvement )

monkey (mac osx)
----------------------
Tp before:  664
Tp after:   584   ( -80ms, or 12.05% improvement )

fuego (linux)
----------------------
Tp before: 2655
Tp after:  2596   ( -59ms, or 2.22% improvement )


creature (win2k)
----------------------
Tp before:  270
Tp after:   269 and 268  ( no difference )
great!
So, what happens if we make ODA bigger than 64K? Would Tp go down even more?
Quite possibly - Darin talked about increasing the number of 16K segments. The
issue is that we'd need to make sure all ODA handlers can deal with getting more
data than 64K at once. I think we should try it. I suspect it would help with
local file operations the most, since those operations are most likely to have
more than 64K of data at a time.
roc, bienvenu: the issue is that a lot of ODA impls will take the count
parameter and pass that straight to malloc (either directly or indirectly). 
gzip stream converter is a good example.  of course, such impls could be revised :-/

anyhow, the parameters of interest are stored in
netwerk/base/src/nsNetSegmentUtils.h

the default values are:

#define NET_DEFAULT_SEGMENT_SIZE  4096
#define NET_DEFAULT_SEGMENT_COUNT 16

also, nsIOService owns a cache of these segments.  the default number of cached
segments is:

#define NS_NECKO_BUFFER_CACHE_COUNT 24

with a 15 minute expiration.

if someone has spare cycles, it'd be worthwhile to play around with different
configurations of these settings.

fwiw: in the past i tried upping the max buffer size, but didn't see much
improvement beyond 32k.  i also have a bug somewhere about making these buffer
sizes more configurable (per channel, transport, stream pump, etc.).
Tp on Windows (creature) has not improved but it has on Linux (btek). 
Why ? Do btek and creature have the same internet connection speed ? 
No, I think Windows is just better behaved in this respect. Any time you have a
lot of thread switches/interactions like this, Mac and Linux seem more adversely
affected. At least, that's one possibility. An other possibility is that Windows
Mozilla didn't run into the 16K limit as often for some reason; perhaps it
processed the data quickly enough so that the data didn't backup - I don't know.
beast is pretty damn fast, so yeah... unless we have some windows numbers for a
slower box, i wouldn't discount the possibility that this also helped windows
(in general).
Bienvenu, concerning your comment #33: this issue hasn't been fixed yet, right?
Is there a follow-up bug? I have a testcase at hand (the one from comment #15)
and am willing to test a fix when available.

The problem is still visible - when I visit the mentioned folder in threaded
mode, Mozilla consumes all CPU for several seconds. It's on a Pentium III
666MHz, with 512 MB RAM and the folder only contains 318 messages, but all of
them have the same subject.
slow threading when all messages have the same subject is a different problem
that is not fixed. I think there's a separate bug filed for it, but if not, I'll
file a new one and note it here.
David, bug 159660 seems to cover this.
Adam, no, that's a different issue, I think. That has to do with listing the
contents of a thread and displaying them. The slowness threading messages with
the same subject has to do with the algorithm used to add new messages to a
thread when we're threading by subject.
David, I've opened bug 226730 for the threading problem.
I don't know if this info will be useful (I have a stripped binary), but here
are the backtraces from next-stepping through the running mozilla-bin process
(the one that consumes all the CPU):


0x4285f900 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmsgdb.so
(gdb) bt
#0  0x4285f900 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmsgdb.so
#1  0x40a8bf96 in PL_DHashTableRawRemove () from /usr/lib/mozilla-1.4/libxpcom.so
(gdb) next
Single stepping until exit from function NSGetModule, 
which has no line number information.
0x40a8bf96 in PL_DHashTableRawRemove () from /usr/lib/mozilla-1.4/libxpcom.so
(gdb) bt
#0  0x40a8bf96 in PL_DHashTableRawRemove () from /usr/lib/mozilla-1.4/libxpcom.so
#1  0x09cebd28 in ?? ()
(gdb) next
Single stepping until exit from function PL_DHashTableRawRemove, 
which has no line number information.
0x40a8bf0a in PL_DHashTableOperate () from /usr/lib/mozilla-1.4/libxpcom.so
(gdb) bt
#0  0x40a8bf0a in PL_DHashTableOperate () from /usr/lib/mozilla-1.4/libxpcom.so
#1  0x09cebd28 in ?? ()
(gdb) next
Single stepping until exit from function PL_DHashTableOperate, 
which has no line number information.
0x4285faf5 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmsgdb.so
(gdb) bt
#0  0x4285faf5 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmsgdb.so
#1  0x09cebd28 in ?? ()
(gdb) next
Single stepping until exit from function NSGetModule, 
which has no line number information.
0x40a8c063 in PL_DHashTableEnumerate () from /usr/lib/mozilla-1.4/libxpcom.so
(gdb) bt
#0  0x40a8c063 in PL_DHashTableEnumerate () from /usr/lib/mozilla-1.4/libxpcom.so
#1  0x09cbb498 in ?? ()
(gdb) next
Single stepping until exit from function PL_DHashTableEnumerate, 
which has no line number information.
0x4285f550 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmsgdb.so
(gdb) bt
#0  0x4285f550 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmsgdb.so
#1  0x40a8c063 in PL_DHashTableEnumerate () from /usr/lib/mozilla-1.4/libxpcom.so
(gdb) next
Single stepping until exit from function NSGetModule, 
which has no line number information.
0x424d9de0 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmork.so
(gdb) bt
#0  0x424d9de0 in NSGetModule () from /usr/lib/mozilla-1.4/components/libmork.so
#1  0x4286e64a in NSGetModule () from /usr/lib/mozilla-1.4/components/libmsgdb.so

thanks, I know exactly what's going on, but it's hard to fix - turning off
threading by subject will speed it up for you.

user_pref("mail.thread_without_re", false);
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: