Closed Bug 369255 Opened 17 years ago Closed 12 years ago

Thunderbird is using insane amounts of memory during normal usage, perhaps due to unclosed/rebuilding msf files (leak in junk filter?)

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: BenB, Unassigned)

Details

(Keywords: memory-leak, perf, Whiteboard: [needs testcase/reproduction]) 

I've seen this since years, and coped with it by restarting or buying new RAM, but it's really getting out of hand. I currently have 500 MB RAM usage after 5 days.
top says VIRT 750 MB, RES 477 MB, SHR 30 MB, TIME 135 hours.

Sorry for the vage report, I know memory leaks and excessive RAM usage are a whole *class* of bugs, and can root at almost any line in the code, and can be either a single forgotten dereference, or a lot of them, or a coding practice problem, or a general XPCOM design problem (the latter is not worth discussing here). I am just too lazy to run tools (Purify, valgrind, dbaron's leakmon extension, tracerefcnt) to diagnose the problem, sorry.

If I am the only one seeing the problem is such extreme amount, probably the amount of spam I get (2-3000 per day), all filtered by Thunderbird's Bayesian Junk filter, not on the server, is probably what differs me most, so that would be a place to look. Also, I'm running on Linux. Self-built 2.0 beta from Jan 15, but I've seen this since years.

If the developers don't consider this bug report useful or worth to investigate, it's fine to close it INVALID for lack of detail.
FWIW, the mem usage starts around 100 MB after a few mins running (which is already bad, but not subject of this bug) and seems to grow linearly with time, so this is probably a plain old leak.
Summary: Thunderbird is using insane amounts of memory during normal usage (junk filter to blame?) → Thunderbird is using insane amounts of memory during normal usage (leak in junk filter?)
There is some evidence that getting lots of Spam is implicated in the memory bloat/leak, though it's hard to know if it's Spam or just getting lots of mail.

Do you know if your spam is mostly html or plain text or a mixture?
> Do you know if your spam is mostly html or plain text or a mixture?

I have no idea, but I can surely give you a sample of a few millions or so (no kidding).

Seriously, I did on my newest Spam folder:
grep Content-Type *|grep <type>|wc -l
And the values for <type> are:
multipart/alternative: 7701
multipart/related: 4766
text/html: 17097
text/plain: 12292

These are 16375 mails.

FWIW, that "newest Spam folder" nicely coincides almost with the reported runtime of the app, it was created roughly at the time the app was started. So, modulo the 1.5 days, the app used up the reported 500 MB while filtering exactly these 16500 mails (or 12500 of them).
 PID  USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4226 ben       15   0  792m 481m  34m S  0.7 23.9 291:13.53 thunderbird-bin

For comparison:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4196 ben       15   0  578m 345m  29m S 13.3 17.2   1270:03 firefox-bin
(580/350MB in 52 days)
Question to rule out problem caused by extension.
Does your problem occur even when no extension is installed?

> If I am the only one seeing the problem is such extreme amount
You are not the only one. See dependency tree for meta Bug 320915.
But no one has found on what situation problem occurs, except "when keeped running".

I have very few extensions, and I don't think any would cause this. IIRC I saw this when I had no extensions installed.

I can only mention two things which seem to have the most effect:
- 64bit (increases RAM usage two-fold)
- Lots of spam. The spam-filter seems to use lots of RAM. I get 3000/day. Esp. when new mail is coming in and filtered, RAM usage spikes by several hundred megs IIRC.

Even when idle, it's large, though. Right now, I have it running since only a few days, and on 32bit (in part because of Thunderbird), and Thunderbird is idle, but still uses 438 MB RAM.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
32021 ben       20   0  576m 438m  23m S  1.3 22.0  87:06.82 thunderbird-bin
29087 ben       20   0  445m 208m  16m S  2.0 10.5 474:00.40 firefox-bin

Do you use RSS? If yes, read Bug 378077 and see Bug 385178(fixed on trunk only), please.
> Do you use RSS?

No.
Status: NEW → ASSIGNED
Correction: I didn't *use* RSS, but I had one feed configured. The newest article in there was from November 2007, although the site updates regularly, so I guess the feed was not fetched. I removed it now and restart Thunderbird.
"3000_spams/day * several_days" means delete of many mails from Inbox.
Similar phenomenon to Bug 398684 is involved?
The phenomenon in Bug 398684 is one observed during duplicate test of CPU hog issues while delete of many many mails.
TB 2.x is in security maintenance mode, so this is not going to get resolved for 2.x. Therefore clearing dependency on Gecko 1.8 bug.
No longer blocks: mlk1.8
Version: 2.0 → Trunk
Assignee: mscott → nobody
Status: ASSIGNED → NEW
For me on a Mac OSX 10.5.6 this is a big problem. I have been running TB 3beta which was nearly unusable, and now run Shredder nightlies, which seem slightly better.  

After a few minutes memory usage goes up to about 800mb Real Memory, and over 2GB Virtual Memory and stays around there.

I am running without extensions (I was using experimental toolbar and gloda address book but they are too buggy so now I am running vanilla install). I get 500 spam per day, and I have 10 or so folders with 15 to 20 thousand messages each. Turning off Message Indexing seems to help a bit, but it still goes up over time. Any tips? 

Thanks, Dan - Macbook Pro 17, 2.16ghz dual core, 2GB Ram, 320gb HD.
Dan, if you see this problem in shredder but not in thunderbird 3, then your problem is not likely to be this bug.  You might choose from this list of memory bugs with recent activity - http://tinyurl.com/8493nh - or file a new bug.
Dan, there are a couple bugs that leave the .msf files open for news groups and imap folders, and this can cause enormous memory bloat. I fixed one in the nightlies, I think, and I have a reviewed fix for the news group issue, and I have a patch for the imap folders that fixes the issue, but I haven't been able to get it reviewed yet. If you have imap folders, then turning off imap auto sync will most likely fix the bloat issues.
Thanks - I think that Experimental Toolbar and the internal Message Indexer were the major culprits. After completely removing Experimental toolbar and turning off "Enable Message Indexer" in Advanced things are running much better. I think the indexer was constantly running, eating up gobs of RAM. This memory problem was happening in both Thunderbird and Shredder. Now it's well behaved, around 70mb Real Memory and 1gb Virtual Memory, much better!
Dan, I checked in some fixes yesterday that are in today's build that should have helped quite a bit as well.
Thanks David, I'm running the nightlies, and I've turned off Experimental Toolbar and the Message Indexer. Shredder is now running better at 77mb real memory, which grows to 120mb real memory over an hour or so of usage. Everything else is the same as listed in my earlier posts. These numbers are usable with my system, much better than before, but even more efficiency would be great. Thanks, and have a great holiday season everyone!
Keywords: mlk
Ben, still see this with current trunk?
No:
 4297 ben       20   0  440m 227m  26m S    0  5.7  60:54.45 thunderbird-bin
(virtual memory 440 MB, resident RAM 227 MB - 32bit Linux).

I still believe that this is due to the .msf problems (.msf files constantly rebuilt after every start/crash) that I had at the same time when I saw this.
several bugs have been fixed on trunk in this area - mostly by bienvenu - but I can't say if your specific issue is addressed. Still, if you don't see it...
closable then?
No, it's definitely not closeable, and I don't think it has been fixed.
It's merely that - I guess - there are specific circumstances which lead to havoc in the .msf files, which causes them to be rebuilt on every start and also cause a lot of crashes (causing a lot of rebuilds), which causes this bug and several other bugs that I saw during this time and not before and after.
BTW: Another factor possible factor: I am now using server-side mail filtering, which means my TB sees 10 times less mail.
actually, I think the odds are at least even that one of the half dozen or more .msf bugs of the past 6+ months fixed this - though I understand not having the crash is a factor. I think the crashes are more symptomatic, or an inducing factor, not the direct cause.  But, can you approximate and test by forcing a few crashes?  

Alternatively, perhaps rkent can link your symptoms back to a specific .msf bug (I'm not up to going through them)
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Whiteboard: dupeme
ultimately (at the moment) we have no demonstrable testcase in this bug. Not even enough information to accurately do a dup.  Suggest close incomplete and reopen if you see the issue again.  If it's any consolation, there are bugs open where people *are* seeing memory issues.
Whiteboard: dupeme → [needs testcase/reproduction]
Do not close. I have spent lots of time on this bug, to provide lots of information which hits at the problem areas (.msf files) and what triggers the bug (repeated crashes).
Keywords: perf
(In reply to Ben Bucksch (:BenB) from comment #21)
> No:
>  4297 ben       20   0  440m 227m  26m S    0  5.7  60:54.45 thunderbird-bin
> (virtual memory 440 MB, resident RAM 227 MB - 32bit Linux).
> 
> I still believe that this is due to the .msf problems (.msf files constantly
> rebuilt after every start/crash) that I had at the same time when I saw this.

Ben, 

Were you using mail.check_all_imap_folders_for_new at the time of your report? If so, bug 540214 probably fixed your issue, shortly after comment 27. 

Except for 540214, in recent years we're not seeing credible reports of massive memory usage - per https://bugzilla.mozilla.org/buglist.cgi?bug_id=369255%2C540214%2C381992%2C518918%2C542234%2C185634%2C650508%2C583365%2C366457%2C538378&bug_id_type=anyexact&query_format=advanced&list_id=2095509 (derived from http://tinyurl.com/6vv5c5q). In addition, I've never seen reports of high mem usage connected to bayesian filtering - and we users who have had large amounts of junk mail.

I suggest it is now, reasonable to close this bug
Summary: Thunderbird is using insane amounts of memory during normal usage (leak in junk filter?) → Thunderbird is using insane amounts of memory during normal usage, perhaps due to unclosed/rebuilding msf files (leak in junk filter?)
wsmwk, can you please stop asking to close this bug or asking me questions? Please just leave it alone, unless *you* can add substantial information yourself that helps fix it.

Just because I don't currently see it doesn't mean it's gone. I see this regularly whenever I have some msf file corruption. Just because I didn't have one recently doesn't mean that Thunderbird now recovers from that situation without running into this bug.
Wayne is just trying to help. I think closing this bug as INCOMPLETE is reasonable because we don't have a reproducible case or really any actual information as to what the cause is, just theories. Turning on MSGDB logging would be the first thing to do if this starts happening again so we can see what databases are opened and closed when this happens.
> Turning on MSGDB logging would be the first thing to do if this starts happening again

Yes, I'd gladly do that, once I run into it again.

Can you please give me detailed instructions as to what exactly I should do when this happens again, how to investigate it?
I.e. which what do I put in the NSPR_LOGGING env var? What else can I look at to investigate the bug?

The problem is that when I run into it, I need to recover quickly, because I am completely dependent on email for my work and contacts, so I cannot wait for your to wake up (timezone) and respond and instruct me. That's why I ask now.

I would like to keep it open, because it is a serious bug, even if rare.
(In reply to Ben Bucksch (:BenB) from comment #31)
> > Turning on MSGDB logging would be the first thing to do if this starts happening again
> 
> Yes, I'd gladly do that, once I run into it again.
> 
> Can you please give me detailed instructions as to what exactly I should do
> when this happens again, how to investigate it?
> I.e. which what do I put in the NSPR_LOGGING env var?

MSGDB:4 or MSGDB:5 (the former shows all db opens and closes; the latter adds a list of the open db's after every open/close, so open db bloat becomes obvious).

Seeing what's in that log would be the starting point, to see what errors if any we're getting opening db's, and how we handle it.
Actually, re-reading this bug, it's not what I thought it was. I had a specific bug in mind when I had corrupted files and Thunderbird eats 1 GB half an hour after start. I thought I had described this better and with more information, but this bug is indeed useless.

So, yes, we can close this bug. But:

bienvenu, thanks for answering. Do you have any other ideas how I could investigate the bug? As I said, we h ave only once chance in 2 years, there's no "second step", because I need to recover immediately, I cannot preserve the broken state to investigate further later.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → INCOMPLETE
(In reply to Ben Bucksch (:BenB) from comment #33)

> bienvenu, thanks for answering. Do you have any other ideas how I could
> investigate the bug? As I said, we h ave only once chance in 2 years,
> there's no "second step", because I need to recover immediately, I cannot
> preserve the broken state to investigate further later.

W/ so little information, it's hard to say. My best guess would be to look at the log when it's happening, and save off any .msf files that the log says it's getting an error opening.
> W/ so little information, it's hard to say.

That's the same problem I had when I filed the bug :).

Thanks, I'll try the log next time, and maybe save the broken profile, in case that helps.
You need to log in before you can comment on or make changes to this bug.