228383 - Runaway memory on folder compact (after installing RAM)

Status: VERIFIED FIXED

(MailNews Core :: Database, defect)

Description

[Mike Holderness]

User-Agent:       Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.4) Gecko/20030624
Build Identifier: Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.4) Gecko/20030624

Yesterday I upped my system RAM from 64M to 192M (largely because I got fed up
with Mozilla taking forever to swap after visiting a few pages - solution: avoid
broken MS VM). I reduced the swap file from 100M to 10M.

Since then, three times Mozilla has hung with RAM allocation maxed out
(according to Win NT Task Manager).

The last time was clearly during folder compaction on startup.

My largest mail folder has 1157 msgs in 11M. 

This may be related to Bug 73790 - reporeted fixed in 2001 - or to Tunderbird
bug 216535 - but have filed separately because it seems much more specific.


Reproducible: Sometimes

Steps to Reproduce:
1. Reboot machine. Start Mozilla;
2. Say "yes" to "compact folders?"
3. Cross fingers...

Actual Results:  
RAM usage maxes out, Mozilla hangs. (Once, Task Manager hung too!)

Expected Results:  
Compacted folders nicely and got on with life.

Comment 1

[Robert Accettura [:raccettura]]

Confirming, I've seen this on 1.6b+

I've seen this to.  And afterwords, the memory isn't so bad as in the middle,
but we don't get it all back either.

This is a regression as it's never been this bad.  I've got 1GB of RAM in this
laptop.  I shouldn't be having 500MB's of it in use to compact my inbox.

Comment 2

[Hermann Schwab]

1.4 had some problems with memory, so I would suggest to upgrade to 1.4.1, if
you like these bugs (and others) fixed, 1.5, if you like new features and bugs,
or 1.6, when it comes.
1.4 is too old, to report bugs, so please retry with 1.4.1, or 1.6b, or better a
current nightly.

Robert, why did you change OS from WindowsNT to All?
Do you use All as OS on your notebook, and if so, which version of All?

Comment 3

[Robert Accettura [:raccettura]]

I've seen this happen on OS X, and Windows, hence All.  There is no indication
that this is OS specific.

Comment 4

[David :Bienvenu]

I have noticed that compacting folders leaks some memory, but that shouldn't
cause Mozilla to hang. 

It could be that folder compaction is leaving db's open - I'll look.

Comment 5

[Robert Accettura [:raccettura]]

I haven't had a hang... but then again, I have 1GB of memory, and not all of it
was occupied yet.  If I had 512MB, I don't know what would happen.  I guess my
computer would have froze up for a while at least.

Reporter:  Can you specify your specs (more curiousity).  RAM/CPU and how much
mail you have.

Comment 6

[David :Bienvenu]

yeah, compact all folders doesn't close the folder db's afterwards. I'm not sure
this is a regression...

Comment 7

[Mike Holderness]

Need to note a caveat that an hour later my NT4 machine did something similar
while not running Mozilla. So I'm not in a good position to produce a clean test
case. 

Specs: 
Tosh Satellite 4300, Celeron 600MHz
192M RAM + 10M swap 
  (since yesterday, problem not seen before w/ 64M + 100M swap)
NT4 SP 6a + 2 security fixes since
Total in mail folders 52M (54M including .msf files)
Total in news folders 0.

Other processes running at the time: Netgear MA401 Wireless PC Card WiFi
monitor; MightyFax "print" driver; ZoneAlarm; CardWizard; MS Task Manager.

Plus Tosh standards: Power Saver; S3 Display; Battery Meter; Yamaha sound
manager; Country Selection Utility; MouseWare.

Comment 8

[Robert Accettura [:raccettura]]

Removed regression keyword for now.

Blocking 1.6?
Since it doesn't look like a pre-christmas release is likely.  Can we hold 1.6
for this as well in that case?

It's not pretty to experience.  It feels enough like a crash for uses to
consider it a crash.

David, could this memory issue have been why I experienced some oddness viewing
my folders the other day?

Comment 9

[David :Bienvenu]

I can't account for 500MB of memory getting used, unless you have folders with
hundreds of thousands of messages.

Comment 10

[Robert Accettura [:raccettura]]

Largest is 26,933 emails as of this post... a few attachments in there, but most
just email.

Comment 11

[David :Bienvenu]

Attachment: close local folder db's after compact

this fix closes local folder db's after compact (and breaks out the code to do
that into a helper routine). It also removes a couple extra unneeded db commits
because ending the batch notification now does a commit.

Comment 12

[David :Bienvenu]

Comment on attachment 137457 [details] [diff] [review]
close local folder db's after compact

this also has a fix for creating intermediate local directories for local mail
folders...I can remove any or all of the other fixes if either of you guys
think I should.

Comment 13

[(not reading, please use [e-mail] instead)]

Comment on attachment 137457 [details] [diff] [review]
close local folder db's after compact

r/a=sspitzer for 1.6 final

Comment 14

[David :Bienvenu]

fix checked in. This should fix some bloat. 

Robert, can you try tomorrow's build? I'm still sceptical that the db's are
accounting for all the bloat, but it will be interesting to see what it looks
like tomorrow.

Comment 15

[Robert Accettura [:raccettura]]

Can I assume the nightly available now has this patch in it?

Comment 16

[David :Bienvenu]

yes, it should.

Comment 17

[Robert Accettura [:raccettura]]

VERIFIED

per IM conversation with David.