Closed Bug 796989 Opened 12 years ago Closed 11 years ago

Memory leak in TB 15 (2GB) in safe mode

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Version:
15 Branch
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: jberkus, Unassigned)

References

Details

(Keywords: memory-leak, regression, Whiteboard: [regression:TB15?][on Mac]) 

Thunderbird 15 (on Mac) apparently has a creeping memory leak.   This includes 15.0.1, which did not cause any change in behavior in this regard.

1. I have multiple accounts with thousands of emails.  This means that on startup TB needs about 250MB.
2. I tend to leave TB on all the time.
3. Over the course of 16-24 hours, TB will gradually use more and more memory on the Mac.
4. Eventually it peaks at around 1.8GB real, 6GB virtual on a laptop with 2GB RAM.
5. At that point, the system becomes unresponsive and TB needs to be killed.

Note that I did not observe this issue with 14.0.
Can you install the gecko profiler extension and submit a repoprt so we can analyse what's going on ?
Keywords: mlk, regression, regressionwindow-wanted
Sure, link?
Unfortunately the profiler doesn't work on 15.x; it's only supported on 16.x and forward. Best results would be if you could reproduce using a Nightly build, but failing that you could test Beta.

The profiler extension is available in the Add-Ons library; go to Tools / Add-ons / Get Add-ons and search for "Gecko Profiler"

Also, if you could check Help / Troubleshooting Information / about:memory to see if it identifies what sort of memory is leaking.
see also http://mikeconley.ca/blog/2012/06/15/gecko-profiler-now-works-in-thunderbird-daily/

I have maybe 3 people reporting memory issues in the past month. But not been able to narrow it down. And in one case for a startup issue a msgdb log revealed no databases causing the high memory

I agree with irving, using daily build or aurora would be a plus.
Severity: normal → major
linux reports 
http://forums.mozillazine.org/viewtopic.php?f=39&t=2543045
http://forums.mozillazine.org/viewtopic.php?f=39&t=2548781
bug 793792
bug 791581

windows reports 
https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_15_0_1_memory_usage_leakage

bug 795590 is windows, but doesn't seem the same class as this bug.

plus several reports have been sourced to lightning and google contacts.
Here's about:memory after several hours of running, and apparently memory leak up to 800MB:

Main Process

Explicit Allocations
608.09 MB (100.0%) -- explicit
├──446.09 MB (73.36%) ── heap-unclassified
├───87.66 MB (14.41%) -- js
│   ├──61.43 MB (10.10%) ++ (298 tiny)
│   ├──19.25 MB (03.16%) -- compartment([System Principal], about:blank)
│   │  ├──11.37 MB (01.87%) ++ gc-heap
│   │  └───7.87 MB (01.29%) ++ (9 tiny)
│   └───6.98 MB (01.15%) ++ runtime
├───31.54 MB (05.19%) -- storage
│   ├──31.53 MB (05.19%) -- sqlite
│   │  ├──24.30 MB (04.00%) -- global-messages-db.sqlite
│   │  │  ├──24.17 MB (03.97%) ── cache-used
│   │  │  └───0.13 MB (00.02%) ++ (2 tiny)
│   │  └───7.24 MB (01.19%) ++ (11 tiny)
│   └───0.00 MB (00.00%) ── prefixset/all
├───18.09 MB (02.97%) -- window-objects
│   ├──11.87 MB (01.95%) ++ (17 tiny)
│   └───6.22 MB (01.02%) ++ top(chrome://messenger/content/messenger.xul, id=61)/active
├───13.89 MB (02.28%) -- dom/workers()
│   ├───6.95 MB (01.14%) ++ worker(resource:///modules/attachmentChecker.js, 0x14fc07800)
│   ├───6.95 MB (01.14%) ++ worker(resource:///modules/attachmentChecker.js, 0x167528000)
│   └───0.00 MB (00.00%) ++ (3 tiny)
└───10.83 MB (01.78%) ++ (10 tiny)

Other Measurements
    0.00 MB ── canvas-2d-pixel-bytes
  627.11 MB ── explicit
    0.59 MB ── gfx-surface-image
          0 ── ghost-windows
  534.98 MB ── heap-allocated
  551.03 MB ── heap-committed
   15.90 MB ── heap-committed-unused
      2.97% ── heap-committed-unused-ratio
    2.79 MB ── heap-dirty
  218.88 MB ── heap-unused
    0.00 MB ── images-content-used-uncompressed
        331 ── js-compartments-system
         14 ── js-compartments-user
   54.00 MB ── js-gc-heap
    2.13 MB ── js-main-runtime-analysis-temporary
   28.09 MB ── js-main-runtime-gc-heap-allocated
   23.35 MB ── js-main-runtime-gc-heap-arena-unused
    0.00 MB ── js-main-runtime-gc-heap-chunk-clean-unused
    0.00 MB ── js-main-runtime-gc-heap-chunk-dirty-unused
   51.44 MB ── js-main-runtime-gc-heap-committed
   23.35 MB ── js-main-runtime-gc-heap-committed-unused
     83.13% ── js-main-runtime-gc-heap-committed-unused-ratio
    2.56 MB ── js-main-runtime-gc-heap-decommitted
    0.34 MB ── js-main-runtime-mjit
   12.36 MB ── js-main-runtime-objects
   14.00 MB ── js-main-runtime-scripts
   14.58 MB ── js-main-runtime-shapes
    4.36 MB ── js-main-runtime-strings
    0.50 MB ── js-main-runtime-type-inference
      9,051 ── page-faults-hard
  3,095,080 ── page-faults-soft
  587.05 MB ── resident
   31.53 MB ── storage-sqlite
4,262.83 MB ── vsize
    2.43 MB ── window-objects-dom
    5.42 MB ── window-objects-layout-arenas
    0.22 MB ── window-objects-layout-pres-contexts
    4.56 MB ── window-objects-layout-style-sets
    0.03 MB ── window-objects-layout-text-runs
    5.42 MB ── window-objects-style-sheets
This is my work desktop, so I don't feel that comfortable running it on aurora.  Therefore I'll try reverting to 14.0 next, instead.

Note that this will take a while, because my profile is quite large, so don't expect to see anything from me for a few days.
I don't understand what will take a while. You know you don't have to redownloaded messages, right?
my profile is ~~ 8GB, and I don't have the space to spare on this laptop, so I need to find a drive with space on it.
Josh, thanks for the about:memory (and sorry about the crash from leaving about:memory open; did that send in a crash report? If so, can you follow up with the link from Help / Troubleshooting / about:crashes?)

What type of accounts are you using? Is most of your mail in POP or local folders, or is it in IMAP? If IMAP, do you have offline use enabled?
I've never once had TB fire breakpad on this machine.  I've discussed before that breakpad-for-TB-on-mac doesn't seem to work.

And IMAP with offline use enabled, correct.  Hence the huge profile storage size.
BTW, I'm "jberkus" on irc.mozilla.org
jberkus, do you have a profiler results file?
Flags: needinfo?(josh)
Wayne,

I've upgraded to 16.0.2.  Let me make sure the memory leak is still happening.
Flags: needinfo?(josh)
Hmmm.  Ok, leak is still there but it's slower.  

I left Thunderbird running for 3 days.  over that time, memory usage crept up from:

390MB real, 396MB virtual
to: 1.53GB real, 2.07GB virtual.

I'll proceed with taking memory profiler data both before and after memory bloat.  Note that I'm rather an extreme case; I have four active IMAP accounts, with 50+ folders total, and receive ~~ 300 emails/day.  OSS developer, what can I tell you?

Is attempting to downgrade to 14.0 still a worthwhile test excercise as well?
Wayne,

I've installed the profiler, but I have no idea where the results file is, and the profiler docs don't say.  Hint?
Wayne,

Ignore the last question.  I found the profiler, did "Upload Full Profile", but now I have no idea where the profile is so that I can link you to it.
Mike ?
(In reply to [:jberkus] Josh Berkus from comment #19)
> Wayne,
> 
> Ignore the last question.  I found the profiler, did "Upload Full Profile",
> but now I have no idea where the profile is so that I can link you to it.

After you upload the profile, just below the "Share" heading and above the "Upload Full Profile" button you should see the line "Success! Use this link"; the word "link" is a hyperlink to the URL where the profile was uploaded.
Nope.  Tried it 3 times, no link.
(In reply to [:jberkus] Josh Berkus from comment #22)
> Nope.  Tried it 3 times, no link.

Which version of the add-on is this?
I have no way to tell.

However, I downloaded it last week.
(In reply to [:jberkus] Josh Berkus from comment #24)
> I have no way to tell.
> 
> However, I downloaded it last week.

If you go into your Add-ons Manager, it should tell you what version of the profiler that you're running.
Ah, I see.  It's an Extension, not an Add-on.

0.9.8
Feh, sorry: 1.9.8
Strange. I have the same version of the add-on, and I'm able to upload a profile using the following steps on Windows 7:

http://i.imgur.com/doN7F.png
So, here's a "before" profile.  What I did to get it was:

Shut down TB.
Restarted TB after a few minutes.
Enabled the profiler.
Waited 3 minutes.
Uploaded a profile.

http://people.mozilla.com/~bgirard/cleopatra/?1352403914629#report=09b1eeaec8f90e37d76471d262172d9df28301ce

Note that having the profiler running make TB eat memory much faster than it did without the profiler running; it went from ~~400MB to 670MB in just the 10 minutes it was running with the profiler on.  Normally, it would take a few hours to build up to that amount of RAM.

Tommorrow, I will take a profile after TB has been running for 24 hours.
So, looking at this bug a bit closer, I don't think the Gecko Profiler is going to help us here.

The Gecko Profiler allows us to visualize and drill into performance problems caused by inefficient code / patterns. In particular, it helps us to see where Thunderbird is spending most of its time while it r uns.

It's not really designed to help us drill into what is allocating memory and not de-allocating it.

We might have other tools for this, but again, I don't think the Gecko Profiler is what we want to use to solve this particular problem.
jberkus, does this memory loss occur if you start thunderbird in safe mode?
Summary: Memory leak in TB 15 → Memory leak in TB 15 (2GB)
Whiteboard: [regression:TB15][needs profile]
I'll try to find out.

I need to run TB for 24-36 hours to really see the leak, so be patient.
Josh ?
Ludovic,

Sorry, holidays, you know?  

Will try running it now.
Ludovic,

OK, this looks like it's not an add-on issue.  Twice I ran Thunderbird for 24 hours with add-ons disabled (via Help-->restart with add-ons disabled).  In both cases, Real Mem for Thunderbird went up by 450MB over the course of 24 hours, from ~~ 400mb to ~~ 850MB (according to top, 660MB of that is pinned and not virtual).  This is consistent with the leakage rate I've been seeing since 16.0 (which was half the leakage rate of 15.0).

This is currently 17.0.1.

NOTE: I am shortly going to deprecate my Mac in favor of a new, Linux laptop.  As such, I won't be doing anything on TB for Mac except to follow-up on existing bugs.
Josh, obtain a log so we can see if folders are being closed.
see https://wiki.mozilla.org/MailNews:Logging#Environment_Variables_to_set
and use 
 msgdb:1
attach log to bug or email to one of us, and comment in the bug please
Flags: needinfo?(josh)
Whiteboard: [regression:TB15][needs profile] → [closeme 2013-02-28][regression:TB15]
Josh, if you ever get a crash that doesn't fire breakpad, please file a bug and make it block meta bug 482756.

Did you ever get TB14 fired up to see if it reproduces there?
Summary: Memory leak in TB 15 (2GB) → Memory leak in TB 15 (2GB) in safe mode
Wayne,

Sorry, I didn't know that we still wanted to try reverting me to TB14. I thought that approach had been dropped.

Note that I am no longer using the MacBook as my work machine, per comments above -- I've moved back to Linux, where I don't see the memory leak issue.  So I'd need to set up a test where I leave the macbook running with TB on for a couple days, which I can do once I get it back from my wife.

Note that this issue has never crashed TB.  It just eventually eats all the RAM on the system, which then has to be hard rebooted.
Flags: needinfo?(josh)
(In reply to [:jberkus] Josh Berkus from comment #38)
> Wayne,
> 
> Sorry, I didn't know that we still wanted to try reverting me to TB14. I
> thought that approach had been dropped.

It is sometimes useful to (re)confirm - sometimes we find interesting things :)

If it indeed started in TB15, then, it will help to narrow the regression range to where it started in the TB15 development cycle.  I'd start by testing TB15.0b1 at https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/  If b1 does NOT fail, then try b3. If b3 doesn't fail try b2. If b3 does not fail, try b4

> So I'd need to set up a test where I leave the macbook running with TB on
> for a couple days, which I can do once I get it back from my wife.

to save us some time, please get a msgdb:5 log in your next testings of TB17 and TB14. details at https://wiki.mozilla.org/MailNews:Logging
Whiteboard: [closeme 2013-02-28][regression:TB15] → [closeme 2013-02-28][regression:TB15?]
Depends on: 480843
Whiteboard: [closeme 2013-02-28][regression:TB15?] → [regression:TB15?]
Wayne,

As it turns out, I'm seeing a memory leak on Linux too.  So I'm in a better position to test this now.
Josh, you could test http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/archaeopteryx@coole-files.de-a63ca6826d5a/ which has a memory related fix, and report your results
(In reply to Wayne Mery (:wsmwk) from comment #41)
> Josh, you could test
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/
> archaeopteryx@coole-files.de-a63ca6826d5a/ which has a memory related fix,
> and report your results
Flags: needinfo?(josh)
If you still have the Mac we'd be interested in an update (until then, closing incomplete)...

using version 24 Earlybird from http://www.mozilla.org/en-US/thunderbird/channel/

Please see  https://wiki.mozilla.org/MailNews:Logging  and turn on msgdb:1,timestamp   log setting 

- attach msgdb log file
- help | troubleshooting information
- attach file with results from help | troubleshooting | about:memory | measure and save
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INCOMPLETE
Whiteboard: [regression:TB15?] → [regression:TB15?][on Mac]
Understood.  I just haven't had time to follow-up on this and retest.

I'll reopen it if I can repro the issue and collect all data requested.
Flags: needinfo?(josh)
I am running TB24 now and I started seeing this issue. Within 24 hours its killing my 12GB of RAM, half of which is typically free, so, TB is eating north of 5GB real memory. Earlier with TB17, I did not have system come down like this, although I have seen it eat upto 1.5GB of real (RES in Linux 'top') RAM after few days of uptime.

Is anybody making any effort in resolving this? Has any developer tried to valgrind the thing? Something drastically changed in 24 compared to 17.
(In reply to Sunil from comment #45)
> I am running TB24 now and I started seeing this issue. Within 24 hours its
> killing my 12GB of RAM, half of which is typically free, so, TB is eating
> north of 5GB real memory. Earlier with TB17, I did not have system come down
> like this, although I have seen it eat upto 1.5GB of real (RES in Linux
> 'top') RAM after few days of uptime.

Sunil, a disproportionate number of linux users report memory usage issues (as compared to windows users) and they almost always end up not being caused by Thunderbird. If you see this problem when thunderbird is started in safe mode, then please make a new report, but be prepared to answer many questions.
You need to log in before you can comment on or make changes to this bug.