421439 - Thunderbird loses memory, grows and eventually hits memory ulimit

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Yuri]

User-Agent:       Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.12) Gecko/20080229 Firefox/2.0.0.12
Build Identifier: 2.0.0.12

My Thunerbird grows spontaneously over time to over 1GB and crahes after memory is exhausted.

I did memory profiling with gooogle-perftools and am attaching here the resultimg memory profile.

Seems like NSGetModule looses ~96.6% of the memory.

Please see profile attached. 

I use version 2.0.0.12 but the same memory loss was happening already for a long while.



Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

[Yuri]

Attachment: profile converted to image showing the problem

Comment 2

[timeless]

Comment on attachment 307866 [details]
profile converted to image showing the problem

inn the future, it might be a good idea to profile builds for which you have symbols....

Comment 3

[timeless]

--enable-debugger-info-modules

Comment 4

[Yuri]

FYI: I tried to configure with --enable-debugger-info-modules but all modules are still being stripped. So I had to manually remove strip commands from makefiles.

Comment 5

[Yuri]

I would like to reopen this bug since I've ran new experiment with debugging info.

From the profile I see that all lost memory is being allocated from the line:
orkinHeap.cpp:187: void* block = ::operator new(inSize);

Comment 6

[Yuri]

Attachment: profile converted to image showing the problem (with debug info and line information)

Comment 7

[timeless]

so, i don't know why you think we're "losing" memory. obviously crashing isn't a good idea, but w/o a stack trace, there's nothing we can do.

afaik we don't default to --enable-strip, so I'm assuming you have some other odd configure flags (please include your configure flags somewhere).

also note that investigating problems in a branch version of thunderbird is not very helpful. please use a trunk version.

Comment 8

[Yuri]

> so, i don't know why you think we're "losing" memory. obviously crashing isn't
> a good idea, but w/o a stack trace, there's nothing we can do.

I think thunerbird is losing memory because profile shows this. And also when
I leave it alone it goes to 1.5GB and crashes. There is no need for mail client
to use 1.5GB.

Stack trace of what? Crash resulting in a memory loss can be triggered by
*any* malloc not necessarily the one losing memory. Also stack trace can be
inferred from the profile that I attached. All arrows mean function calls.

> afaik we don't default to --enable-strip, so I'm assuming you have some other
> odd configure flags (please include your configure flags somewhere).

The only configure flag I used (except for --prefix) is --enable-debugger-info-modules.
But I used FreeBSD-patched port version. This might have caused stripping.

I am rerunning with the trunk version and will post results here once I am done.

Comment 9

[Yuri]

I've got build error (below) while compiling tru8nk from CVS.
====error log=====
cd FreeBSD7.0_OPT.OBJ ; sh /yuri-huge/thunderbird/mozilla/security/nss/cmd/shlibsign/./sign.sh /yuri-huge/thunderbird/mozilla/dist \
        /yuri-huge/thunderbird/mozilla/security/nss/cmd/shlibsign/FreeBSD7.0_OPT.OBJ FreeBSD \
        /yuri-huge/thunderbird/mozilla/dist/lib /yuri-huge/thunderbird/mozilla/dist/lib/libsoftokn3.so
/yuri-huge/thunderbird/mozilla/security/nss/cmd/shlibsign/FreeBSD7.0_OPT.OBJ/shlibsign -v -i /yuri-huge/thunderbird/mozilla/dist/lib/libsoftokn3.so
Fatal error 'Spinlock called when not threaded.' at line 78 in file /usr/src/lib/libthr/thread/thr_spinlock.c (errno = 2)
Abort trap

Comment 10

[timeless]

sounds like the freebsd ports people have some random patches that they haven't sent us, can you please figure out which they are and file bugs to integrate them?

"memory loss" is something one would find w/ profiling tools (which we document on our developer sites). btw, profiles as pictures are not truly appreciated. and please don't expect us to regenerate stack traces from pictures, that's not our jobs. as for crashes, i want a stack trace from mdb or gdb or whatever showing a crash.

Comment 11

[Yuri]

Ok, I will submit FreeBSD patches as a separate PR.

> "memory loss" is something one would find w/ profiling tools 

That's what I am doing.
> btw, profiles as pictures are not truly appreciated.
What kind of profiles do you truly appreciate? perftools profile can only be viewed as picture that I've attached.

When I talk about the crash here I mean that OS kills thunderbird as a runaway process. Because on my system 1.5GB is a limit for the process. So stacktrace wouldn't be of any significance.

This picture shows who called the allocator when lost memory was allocated.
Please let me know what information is missing so I can obtain it and attach it here.

Comment 12

[timeless]

well um. typically when you use ulimit (or an equivalent), a call to malloc() will return 0, and our code either handles it (good) or fails to handle it (crash). If you crash because we dereferenced a null pointer, then i want a stack trace. if your kernel is evil like the linux kernel in its default configuration and kill()s our process. Then I can't handle the memory case.

We can try to use heuristics to determine maximum memory load, but that's a bit of a mess. And I'd rather not go there. How big is your mailbox? AFAIK the current codebase tries to maintain a complete in memory reference of a mail folder's index. This is of course not the best approach for a folder with >1million messages, but oh well.

Comment 13

[Yuri]

> well um. typically when you use ulimit (or an equivalent), a call to malloc()
> will return 0, and our code either handles it (good) or fails to handle it

See, this is essentially a different issue how ulimit-style limitation is handled. I am not sure but I think depending on circumstances either malloc returns 0 (as a result of failure to the call to brk-like syscall) or OS will send a signal.

But the main issue in this PR is why memory grows so much.

> How big is your mailbox?

My 'Sent' file on disk is gigantic: ~1GB (with about 5k messages)
My 'Inbox' on disk is ~317MB (with about 5k messages)
These are the biggest two.

So does thunderbird try to load them in memory? This might be the cause of a problem.

Comment 14

[Joshua Cranmer [:jcranmer]]

(In reply to comment #13)

> My 'Sent' file on disk is gigantic: ~1GB (with about 5k messages)
> My 'Inbox' on disk is ~317MB (with about 5k messages)
> These are the biggest two.
> 
> So does thunderbird try to load them in memory? This might be the cause of a
> problem.

We're not /that/ stupid. However, the mork code loads the entire msf file (Inbox.msf, Sent.msf) into memory.

It looks like everything is merely a mork problem. I am currently working on ridding mailnews of mork (see bug 11050 for more details); if all goes well, trunk should be rid of it by the end of this summer.

Comment 15

[Yuri]

Problem doesn't exist for me in 2.0.0.17.
Since I seem to be the only one who have experienced it I am closing it.