1322233 - thunderbird 45 leaks memory over a few days in safe mode until OOM. One imap account, 38 folders, no rss feeds. (linux:fedora) Better in version 52.

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Pierre Ossman]

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20161114215630

Steps to reproduce:

I rarely close Thunderbird except for reboots, so general use over a few days.


Actual results:

Thunderbird keeps consuming memory until it gets killed by the OOM reaper, or I kill it manually because something isn't working. E.g. right now I cannot open any link because it cannot allocate memory to execute a sub process.

Right now it is sitting at 9 GiB RSS.


Expected results:

Thunderbird should remain at sensible memory levels, similar to those it has when it just started.

Comment 1

[Pierre Ossman]

For reference, I have no plugins and these extensions:

 - Deselect on delete
 - Toggle Word Wrap

Comment 2

[Wayne Mery (:wsmwk)]

unfortunately, linux users disproportionately report memory issues. i.e. there's something not quite right with some distros.

please see if you can narrow it down using https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems

see also bug 988967

Comment 3

[Pierre Ossman]

There's a lot of things on that list unfortunately, and it takes a few days for the problem to appear. No built-in diagnostic tool to see where allocations are I guess?

Comment 4

[Wayne Mery (:wsmwk)]

(In reply to Pierre Ossman from comment #3)
> No built-in diagnostic tool to see where allocations are I guess?
nothing that's useful

> There's a lot of things on that list unfortunately, and it takes a few days
> for the problem to appear. 

roughly a quarter of the list is specific to Windows. that makes the list much smaller.
track your memory usage every half day or couple hours - see if it is linear or tied to a specific action. 
newsgroups and rss can also have adverse effect as noted in the list.

Comment 5

[Pierre Ossman]

Attachment: Memory growth

No changes yet, just a graph of RSS during the night. I.e. it is leaking memory even without user interaction.

We can also see that the growth is mainly linear. Samples were taken every 30 seconds.

Comment 6

[Pierre Ossman]

More testing:

> 1. Are you using the newest update of Thunderbird?

Yes

> 2. Start Thunderbird in Safe Mode (instructions).

Didn't help

> 3. If you are running an MS-Windows OS, ...

Nope

> 4. If you have gmail accounts, ...

Nope

> 5. If you have newsgroup accounts, ...

Nope

> 6. Are any of these indexing methods enabled: ...

Nope

> 7. On Windows, ...

Nope

> 8. ... perhaps one or more folders (internally a database) are not being closed ...

Nope. Log shows the number of open folders going up and back down to 0.

> 9. Does problem resolve by disabling global indexing?

Already disabled

> 10. ... is global-messages-db.sqlite corrupt?

Will test

> 11. Are any folders 2-4GB in size?

No. Largest is 947 MiB, and second largest is 658 MiB.

> 12. Does disabling interval checks for new mail help?

Will test

> 13. Are any non-Inbox IMAP folders configured to check for new messages ...

Yes, will test

> 14. Do you have thousands of contacts in your address book?

No

> 15. Is panacea.dat cache file in your profile too big?

No, 84 KiB

> 16. If error message indicates a script problem ...

Nope

> 17. If using high CPU, ...

Nope

> 18. Is your Thunderbird profile or any part of Thunderbird on a non-local hard drive, ...

Yes, NFS. Will test.

> 19. If you suspect slow or high disk activity, ...

Nope

> 20. If you have RSS news reader account, ...

Nope

> 21. Common resource requirements/maintenance steps:

Rather not try all of these

> 22. If you are running an old version of linux, ...

Nope

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to Pierre Ossman from comment #5)
> Created attachment 8817137 [details]
> Memory growth
> 
> No changes yet, just a graph of RSS during the night. I.e. it is leaking
> memory even without user interaction.
> 
> We can also see that the growth is mainly linear. Samples were taken every
> 30 seconds.

The X scale is labeled in hours?
What happens to memory use rate if you set Thunderbird offline while are way from your desk? 
  File | Offline | Work Offline

> > 21. Common resource requirements/maintenance steps:
> Rather not try all of these

irrc, those all questions for you to answer - not things you are being asked to do at this time

Comment 8

[Pierre Ossman]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #7)
> 
> The X scale is labeled in hours?

Sample index.

> What happens to memory use rate if you set Thunderbird offline while are way
> from your desk? 
>   File | Offline | Work Offline
> 

I can give it a try.

Comment 9

[Pierre Ossman]

Offline did not have an effect. Memory usage grew overnight at about the same rate.

Comment 10

[Wayne Mery (:wsmwk)]

From where do you currently get your Thunderbird build?
Does memory issue persist if you use the beta from http://www.mozilla.org/en-US/thunderbird/channel/ ?

Comment 11

[Pierre Ossman]

I disabled all the settings related to updating folders, and the leak is still there.

My build is from Fedora right now. I'll give the beta a test.

Comment 12

[Pierre Ossman]

Same issue with the beta, I'm afraid. :/

I'm not sure what else I can test?

Comment 13

[Wayne Mery (:wsmwk)]

Please post your accounts and addons listed at help | troubleshooting.

Also, if you have only email accounts, does it happen if you create a new profile?
 see https://support.mozilla.org/en-US/kb/using-multiple-profiles

Comment 14

[Pierre Ossman]

Attachment: Troubleshooting Information

Here's the info from the troubleshooting page.

I'll create a new profile and see if that helps.

Comment 15

[Pierre Ossman]

Alright, so a new profile helped. Memory useage has stayed flat over the night.

What's next?

Comment 16

[Wayne Mery (:wsmwk)]

Still flat?  You have just one imap account. It's not gmail, correct?

You previously mentioned "13. Are any non-Inbox IMAP folders configured to check for new messages ... Yes, will test".
Do you have non-inbox folders configured to check for new messages in the folder properties?  How many?

Comment 17

[Pierre Ossman]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #16)
> Still flat?  You have just one imap account. It's not gmail, correct?
> 

Just one IMAP account, yes. It is not gmail, but our company server running dovecot.

> You previously mentioned "13. Are any non-Inbox IMAP folders configured to
> check for new messages ... Yes, will test".
> Do you have non-inbox folders configured to check for new messages in the
> folder properties?  How many?

38 folders and I used the hidden setting to check all folders. I tried disabling that setting, but it did not make the leak go away.

Comment 18

[Wayne Mery (:wsmwk)]

Hmm. These memory issues often are related to .msf files. Can you keep the original profile around for a while and we revisit this in the new year?

Comment 19

[Pierre Ossman]

Sure. Just let me know when you want to try something more.

Comment 20

[Wayne Mery (:wsmwk)]

Can you try a nightly build from https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/thunderbird-53.0a1.en-US.linux-x86_64.tar.bz2 ?

Comment 21

[Wayne Mery (:wsmwk)]

Sorry, wrong version in comment 20. Please use https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/thunderbird-54.0a1.en-US.linux-x86_64.tar.bz2

Comment 22

[Wayne Mery (:wsmwk)]

With your old profile :)
And, which flavor of linux are you running?

Comment 23

[Pierre Ossman]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #20)
> Can you try a nightly build from
> https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/
> thunderbird-53.0a1.en-US.linux-x86_64.tar.bz2 ?

Installed and testing. I should point out that it reset my folder settings though, which may or may not affect the results.

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #22)
> And, which flavor of linux are you running?

Fedora 25 (it was 24 when the bug was first filed).

Comment 24

[Pierre Ossman]

So the daily build doesn't seem to leak any memory. But again, it did reset a lot of my settings for some reason.

Comment 25

[Pierre Ossman]

Argh. The reset happened now that I restored my backup copy and started the old version again. It seems something about moving the profile directory around triggers this. :/

Ah well, this'll be a good test to see if the reset stops the memory leak then.

Comment 26

[Wayne Mery (:wsmwk)]

reset, meaning the column headers changed for your folder message list? 
All folders, or just inbox?
Who is your mail ISP

Comment 27

[Pierre Ossman]

The folder list jumped around for a bit before settling on the proper layout. After that all my folder had lost their folder specific settings (columns, threading).

This is our own mail server running dovecot.

Comment 28

[Pierre Ossman]

The leak seems to have disappeared as a result of this. I'm running the old version now, and it has grown less than 100 MiB since startup.

My profile folder is also much, much smaller. Previously it was 2.2 GiB. Now it's just 300 MiB.

Comment 29

[Wayne Mery (:wsmwk)]

Since we do not know what caused the problem, let's close this incomplete.
Perhaps was the typical bad .msf file

Comment 30

[Pierre Ossman]

The problem is unfortunately back. I'm seeing a lot of this at my machine at home, and a bit at my machine at work. I'm also running a newer version: thunderbird-52.6.0-1.fc27.x86_64.

I can't reopen or change the version on the bug, so if someone could please help out. :)

Comment 31

[Pierre Ossman]

Nvm, you guys just have a bugzilla interface I'm not used to. :)

Comment 32

[Wayne Mery (:wsmwk)]

You first reported this in version 45.  Have you tried to reproduce with version 38?  (Assuming you had been using the previous version)  You can get the mozilla version from https://archive.mozilla.org/pub/thunderbird/releases/

Comment 33

[Pierre Ossman]

I can give it a try. Can I safely just downgrade thunderbird and it won't choke on my profile from 52?

Comment 34

[Wayne Mery (:wsmwk)]

(In reply to Pierre Ossman from comment #33)
> I can give it a try. Can I safely just downgrade thunderbird and it won't
> choke on my profile from 52?

yes

Comment 35

[Wayne Mery (:wsmwk)]

(In reply to Pierre Ossman from comment #33)
> I can give it a try. ... downgrade thunderbird 

Pierre, did you have good results??

Because you have a very reproducible case, can you run the beta from http://www.mozilla.org/en-US/thunderbird/channel/ settings per bug 1240722 comment 7 and MSGDB logging level 1 enabled per https://wiki.mozilla.org/MailNews:Logging

Comment 36

[Pierre Ossman]

Sorry, things are a bit chaotic here so I have not had time to have a look at this. It is still present on my todo list though so I plan to get to it once I get other things to calm down a bit. :)

Comment 37

[Wayne Mery (:wsmwk)]

(In reply to Pierre Ossman from comment #6)
> ...
> > 11. Are any folders 2-4GB in size?
> 
> No. Largest is 947 MiB, and second largest is 658 MiB.

What is size of you Inbox folders and .msf  ON DISK?
And also your (other) largest active folder (<folder> and <folder>.msf>)?


> > 13. Are any non-Inbox IMAP folders configured to check for new messages ...
> 
> Yes, will test

How many folders are configured to check for new messages?
Are you using server side filters?

Comment 38

[Pierre Ossman]

(In reply to Wayne Mery (:wsmwk) from comment #37)
> 
> What is size of you Inbox folders and .msf  ON DISK?

I assume you mean:

-rw-rw-r--. 1 ossman ossman 5.5M Jul  4 09:34 INBOX.msf
drwxr-xr-x. 8 ossman ossman 4.0K Jun 11 14:11 INBOX.sbd
5.5M	INBOX.msf
790M	INBOX.sbd

I also have some stale stuff in there:

-rw-------. 1 ossman ossman 111K Apr  4  2017 INBOX
drwx------. 5 ossman ossman 4.0K Feb 22  2017 INBOX-1.sbd
-rw-r--r--. 1 ossman ossman   25 Jun  9  2016 msgFilterRules.dat

> And also your (other) largest active folder (<folder> and <folder>.msf>)?
> 

These are the two largest:

379M	Tracker
18M	Tracker.msf
4.0K	Tracker.sbd

113M	Spamassassined
1.3M	Spamassassined.msf

> 
> How many folders are configured to check for new messages?

As mentioned on comment #17 I check all folders. Currently there are 36 of them.

> Are you using server side filters?

Yes. And I have no filters at all client side.

Comment 39

[Craig Arno]

Just getting around to trying Linux beta 60 now.  I'm not actively developing on Linux at the moment.  I'll leave beta 60 running on my Ubuntu 16.04 server for a few days then check on 7 IMAP account with big INBOX memory usage.

Comment 40

[Craig Arno]

A preliminary look, not exhaustive, says Linux Thunderbird Beta 60.0b10 looks much improved operating under Ubuntu 16.04 LTS x64.  After letting Beta 60.0b10 run mostly unattended for 5 days I see only 4% of 16GB memory used.  In earlier versions I'd see most of all 16GB used even when left unattended.  Again, much improved!

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
22740 craig     20   0 3190996 648652 101996 R  14.9  4.0 469:28.70 thunderbird

root@pluto:~# ps aux|grep thun
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
craig    22740  6.0  4.0 3198044 664152 pts/19 Sl   Aug15 471:31 ./thunderbird

INBOX (arno.com)
-rw-------  1 craig     users  903335626 Aug 20 20:16 craig

Mail and News Accounts
ID 	Incoming server 	Outgoing servers
Name 	Connection security 	Authentication method 	Name 	Connection security 	Authentication method 	Default?
account1	(imap) arno.com:143	STARTTLS	Normal password	smtp.frontier.com:465	SSL/TLS	Normal password	true
account2	(none) Local Folders	None	Normal password
account3	(imap) imap.gmail.com:993	SSL/TLS	OAuth2	smtp.gmail.com:465	SSL/TLS	OAuth2	true
account4	(imap) imap.gmail.com:993	SSL/TLS	OAuth2	smtp.gmail.com:465	SSL/TLS	OAuth2	true
account5	(imap) imap.gmail.com:993	SSL/TLS	OAuth2	smtp.gmail.com:465	SSL/TLS	OAuth2	true
account7	(rss) Feeds-2	None	Normal password
account8	(imap) imap.mail.yahoo.com:143	STARTTLS	OAuth2	smtp.mail.yahoo.com:465	SSL/TLS	Normal password	true
account9	(imap) imap.mail.yahoo.com:993	SSL/TLS	Normal password	smtp.mail.yahoo.com:465	SSL/TLS	Normal password	true
account10	(imap) imap.gmail.com:993	SSL/TLS	OAuth2	smtp.gmail.com:465	SSL/TLS	OAuth2	true


These extensions are loaded:
Extensions
Name 	Version 	Enabled 	ID
CardBook	30.5	true	[e-mail]
Enigmail	2.0.8	true	{847b3a00-7ab1-11d4-8f02-006008948af5}
Lightning	6.2	true	{e2fda1a4-762b-4020-b5ad-a41df1933103}
LookOut (fix version)	1.2.21	true	lookout@s3_fix_version
Provider for Google Calendar	4.4.1	true	{a62ef8ec-5fdc-40c2-873c-223b8a6925cc}
Classic Password Editor	1.0.3	false	[e-mail]
FoxClocks	4.3.2	false	{d37dc5d0-431d-44e5-8c91-49419370caa1}
HabuL	1.21.0	false	{41e52bd9-defc-4a0f-a542-cf643a0776d1}
Inverse SOGo Connector	24.0.4	false	[e-mail]
Saved Password Editor	2.10.4	false	[e-mail]
Select Addressbook Text	1.5	false	[e-mail]

Compared to prior several years of testing, memory and speed performance in this version is impressive.  Can't wait for an official release for Ubuntu 16.04 x64.

====
I'm not sure Troubleshooting Memory numbers can be trusted.  Some of these numbers are quite large and might indicate a problem, if they can be trusted:
Main Process

WARNING: the following values are negative or unreasonably large.

    explicit/maildb
    explicit/maildb/database(imap://[e-mail]/INBOX)
    explicit/(21 tiny)
    explicit/(21 tiny)/heap-unclassified 

This indicates a defect in one or more memory reporters. The invalid values are highlighted.
Explicit Allocations

[-->8,754.42 MB<-- is highlighted]
[-->8,698.33 MB<-- is highlighted]

543.92 MB (100.0%) -- explicit
├──8,754.42 MB (1609.50%) -- maildb [?!]
│  ├──8,698.33 MB (1599.18%) ── database(imap://[e-mail]/INBOX) [?!]
│  ├─────26.15 MB (04.81%) ── database(imap://[e-mail]/Trash)
│  ├─────12.54 MB (02.31%) ++ (9 tiny)
│  ├──────9.34 MB (01.72%) ── database(imap://[e-mail]/INBOX)
│  └──────8.07 MB (01.48%) ── database(mailbox:///home/craig/.thunderbird/25odqldu.default/Mail/Feeds-2/Arn%C3%B8%20History)
├───76.51 MB (14.07%) -- js-non-window
 ... (rest deleted)

Comment 41

[Pierre Ossman]

I am running thunderbird-52.9.1-1.fc28.x86_64 now, and I'm seeing improvement on that version as well. I started getting a question about automatic compacting of folders, which might be related.

I also see less disk usage:

4.1M	INBOX.msf
278M	INBOX.sbd

about:memory shows weird stuff here as well though:



WARNING: the following values are negative or unreasonably large.

    explicit/maildb
    explicit/maildb/database(imap://[e-mail]/INBOX/Tracker)
    explicit/maildb/database(imap://[e-mail]/INBOX)
    explicit/maildb/database(imap://[e-mail]/INBOX/TigerVNC)
    explicit/maildb/database(imap://[e-mail]/INBOX/Bugzilla)
    explicit/maildb/database(imap://[e-mail]/INBOX/Trash)
    explicit/maildb/database(imap://[e-mail]/INBOX/Junk)
    explicit/(21 tiny)
    explicit/(21 tiny)/heap-unclassified 

This indicates a defect in one or more memory reporters. The invalid values are highlighted.
Explicit Allocations

504.10 MB (100.0%) -- explicit
├──57,543.36 MB (11415.17%) -- maildb [?!]
│  ├──36,835.34 MB (7307.21%) ── database(imap://[e-mail]/INBOX/Tracker) [?!]
│  ├──14,620.26 MB (2900.30%) ── database(imap://[e-mail]/INBOX) [?!]
│  ├───2,757.48 MB (547.02%) ── database(imap://[e-mail]/INBOX/TigerVNC) [?!]
│  ├───1,833.73 MB (363.77%) ── database(imap://[e-mail]/INBOX/Bugzilla) [?!]
│  ├─────730.41 MB (144.90%) ── database(imap://[e-mail]/INBOX/Trash) [?!]
│  ├─────658.26 MB (130.58%) ── database(imap://[e-mail]/INBOX/Junk) [?!]
│  ├──────56.52 MB (11.21%) ── database(imap://[e-mail]/INBOX/noVNC)
│  ├──────28.73 MB (05.70%) ── database(imap://[e-mail]/INBOX/Drift)
│  ├──────11.37 MB (02.26%) ── database(imap://[e-mail]/INBOX/Sent)
│  └──────11.25 MB (02.23%) ++ (16 tiny)
├───57.52 MB (11.41%) -- js-non-window
 ...


I will check my other problematic machine as well to see if it is also working better with this version.

Comment 42

[Craig Arno]

With this much success under Linux, I upgraded my Win 10 "production workstation" platform to Thunderbird 60.0 (from v52.something).  After an extended period (~½ hour) where Win Thunderbird was mostly unresponsive (I couldn't even get to Help->About to check version info) and continued to give me "Script not responding" messages (not informing me why it wouldn't respond), it seems to have finally settled out, is running well, and is only using 280MB of memory with 12 accounts (a mixture of Local Dovecot IMAP, Gmail, Yahoo, and Frontier FIOS connections).

First impressions, fast, and being a good citizen with regard to memory and processor usage.  This system is Win 10 Pro i7-6600U 4-Core 2.8GHz, 16GB RAM, 1TB SSD, 4K monitor workstation.  Since this is where I'm spending most of my time, Thunderbird 60.0 should get a much heavier workout (lots of filters, folders, junk mail handling, etc) than just letting it run mostly idle on my Linux server.

Looking good, and I'm finally hopeful Thunderbird's former appetite for consuming all 16GB of memory and crashing is under control.  This must have been a big effort in code review and cleanup.

Comment 43

[Craig Arno]

The script message which keeps popping up at startup is:

"A script on this page may be busy, or it may have stopped responding. You can stop the script now, or you can continue to see if the script will complete.

Script: resource://calendar/modules/utils/calUnifinderUtils.jsm:111"

This could be a Lightning issue.  I use Calendar heavily (and CardBook).  After about 1/2 hour of telling the script to continue, this message stopped showing and everything "Thunderbird" became quite fast and lean.


Thunderbird Windows becomes responsive and memory consumption actually drops to about half of startup values  ~330MB -> ~180MB, in about 1 hour.  This is amazing to me after surviving so many years of memory issues... to watch memory usage actually decline, CPU stays near 0% (bouncing up to 4% occasionally and briefly), and Thunderbird stays responsive!

On a second cold startup under Windows, Thunderbird 60.0 came up much faster (~2 minutes), and is now very responsive.

My Linux server does a lot of the caching work with IMAP/email services, DNS caching, Gigabit Ethernet, Internet QoS/routing, 8-core CPU, local mail handling, etc. to make things easier/faster for Windows machines.  (a home server setup)  Thunderbird 60.0 Windows is working well in this environment.

Comment 44

[Pierre Ossman]

(In reply to Pierre Ossman from comment #41)
> 
> I will check my other problematic machine as well to see if it is also
> working better with this version.

The other machine is also well behaved now.

Comment 45

[Wayne Mery (:wsmwk)]

Pierre, thanks for the detailed report.  

Similar to what I mentioned in bug 998755, I cannot explain your improved memory usage. And the reduced size of Inbox is only a few MB, so I wouldn't think that helped.  But, hopefully the issue won't come back.

(FWIW, there is a disk speed improvement in version 52 via Bug 1242050 - C-C TB: use a default output stream buffer size of 16KB instead of 4KB)

Comment 46

[Richard Leger]

(In reply to Craig Arno from comment #43)
> The script message which keeps popping up at startup is:
> 
> "A script on this page may be busy, or it may have stopped responding. You
> can stop the script now, or you can continue to see if the script will
> complete.
> 
> Script: resource://calendar/modules/utils/calUnifinderUtils.jsm:111"
> 
> This could be a Lightning issue.  I use Calendar heavily (and CardBook). 
> After about 1/2 hour of telling the script to continue, this message stopped
> showing and everything "Thunderbird" became quite fast and lean.

This script issue may be related to Bug 1502923