988967 - Memory leak on IMAP accounts with no quota

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Nathan Bell]

Attachment: thunderbird.memory.lizn_6.csv

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0 (Beta/Release)
Build ID: 20130917102605

Steps to reproduce:

Remove the quota from an IMAP account (I used Cyrus IMAP), then create a Mail Account in Thunderbird (it persisted in 24.4.0, and has existed for a long time).


Actual results:

The longer you have Thunderbird open, the more memory it consumes. Several users had almost 4GB of memory used after a day or two. Adding a quota back to the account did not fix the issue, but adding the quota and deleting their account and recreating it did.

I've attached a csv file show the results of leaving thunderbird running for days on a "bad" account (starts off very high, then thunderbird was restarted a couple of times after running for a day or so) then being fixed to be a "good" account. This is the command I used to gather the data:

while sleep 30 ; do echo -n "$(date) " ; ps aufx | grep -v grep | grep thunderbird/thunderbird | grep lizn | head -n1 | awk '{print $1,$2,$5,$6}' ; done | tee -a thunderbird.memory.lizn_6

I massaged the file (removed lines with no data for when thunderbird was not running, quoted the date) to create the attached csv file.


Expected results:

For users with quotas, or whose quotas were removed after their Thunderbird account was created, the memory usage stays below 500MB even after weeks of staying open.

Comment 1

[Nathan Bell]

Attachment: prefs.js file for bad account

Not sure if it's useful, but here is the prefs.js from the same account that was experiencing problems. No preferences were changed before and the account was fixed; I just removed and recreated the IMAP account.

Comment 2

[Wayne Mery (:wsmwk)]

We're talking memory here, not disk space, right?

Please post info from help | Troubleshooting

Comment 3

[Nathan Bell]

Yes, ram usage, not disk usage.

Comment 4

[Wayne Mery (:wsmwk)]

Does it happen if you start aThunderbird in safe mode?
see https://support.mozilla.org/en-US/kb/safe-mode

what are the exact, full steps you are using to change quota?
And with which ISP?

Comment 5

[Nathan Bell]

The problem doesn't persist in safe mode, but also doesn't persist with the user quota turned off. The preferences set are in the attached prefs.js file. I'm sure it's a specific preference that was set interacting with the quota checking mechanism when there is no quota. I was unable to figure out exactly which preference it was.

The ISP in question is us, using a self-hosted cyrus-imapd installation. Specifically using cyrus version 2.3.11.

We removed the quota with the cyradm command "setquota user/$username" and recreated it with "setquota user/$username STORAGE 4000000"

The memory leak persisted even after recreating the quota until the user account was deleted and recreated.

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Nathan Bell from comment #5)
> ...
> The memory leak persisted even after recreating the quota until the user
> account was deleted and recreated.

So you are now working, and not able to reproduce the problem?

Comment 7

[Nathan Bell]

No; I can recreate the problem by removing the quota again. It takes a couple of days of use for the memory to get out of hand (quicker if Thunderbird is used extensively), but removing the quota does indeed recreate the problem. I was trying to say that replacing the quota doesn't immediately fix the problem; the account also needs to be deleted and recreated.

Comment 8

[Wayne Mery (:wsmwk)]

re-reading this, it sounds retarded. But interesting if true.
I have an account I can test...

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to Nathan Bell from comment #5)
> The problem doesn't persist in safe mode, 

Nathan, what addons do you have installed?
Does high memory go away if you have any removed?

Comment 10

[Wayne Mery (:wsmwk)]

(Nathan's email fails. But it's a large rate of leaking if still occuring, so not closing without testing

Comment 11

[Wayne Mery (:wsmwk)]

Richard, 
Would you be able to test the scenario of this cyrus issue?
(comment 5 and comment 7)

Comment 12

[Richard Leger]

Hi Wayne, which version of Thunderbird do you want me to test this in? I can setup a test account and setup Thunderbird on one of our VM to let it run for a couple of days with the account opened and see what happens perhaps... does the IMAP server has to be on the local network as the client or over Internet fine?

Comment 13

[Wayne Mery (:wsmwk)]

Network wise, either scenario will do.
Best if you can test version 52.

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #13)
> Network wise, either scenario will do.
> Best if you can test version 52.

Comment 15

[Wayne Mery (:wsmwk)]

I had asked a year ago if addons are installed, but no response. And Nathan's account is no longer active.

Comment 16

[Wayne Mery (:wsmwk)]

Gene, have you encountered anything that would explain comment 5?

I'm not finding any previously reported bugs in this area. https://mzl.la/2yjbMAv I also can't imagine this would have been linux only.

Comment 17

[gene smith]

I've been looking at quota stuff a lot recently. I don't see how it would immediately affect memory usage or leakage in tb. It's mostly just passive information that is returned by the server for reporting purposes (the folder properties quota tab). Tb doesn't adjust any memory allocation or structure sizes based on this information.

Comment 18

[Richard Leger]

(In reply to Wayne Mery (:wsmwk) from comment #13)

Network wise, either scenario will do.
Best if you can test version 52.

Hi Wayne,

Is this IMAP memory leak test still required?
If Yes would the protocol below fit what you expect for testing and results?

• Install Thunderbird 52 (32bits) on Windows
• Setup my IMAP(Cyrus)/SMTP mailbox (without quota)
• Keep default preference settings
• Let it load folders/email as per default settings
• Let it seat for two days
• Take a memory report
• Exit Thunderbird
• Set a quota on my cyrus user mailbox
• Start Thunderbird (with existing profile)
• Let it sync and seat opened for two days
• Take a memory report again
• File those two reports in this bug

Does that sound like what you may need to identify further the issue?
If I missed something, please clarify what else may need to be added to the protocol

Hope that help.

Regards,

Comment 19

[gene smith]

(In reply to Richard Leger from comment #18)

(In reply to Wayne Mery (:wsmwk) from comment #13)

Network wise, either scenario will do.
Best if you can test version 52.

Hi Wayne,

Is this IMAP memory leak test still required?
If Yes would the protocol below fit what you expect for testing and results?

• Install Thunderbird 52 (32bits) on Windows

Probably better to use the latest release, 68.x.

• Setup my IMAP(Cyrus)/SMTP mailbox (without quota)
• Keep default preference settings
• Let it load folders/email as per default settings

Should record a memory report here.

• Let it seat for two days
• Take a memory report

Running without quota, reporter says you should see an increase in memory use (the leak).

• Exit Thunderbird
• Set a quota on my cyrus user mailbox
• Start Thunderbird (with existing profile)

Reporter says you will still see leaking unless you also delete and re-create the account here.

• Let it sync and seat opened for two days
• Take a memory report again

Reporter says you should see no leak at this point with fresh account.

• File those two reports in this bug

Does that sound like what you may need to identify further the issue?
If I missed something, please clarify what else may need to be added to the protocol

Hope that help.

Regards,

As I wrote in comment 17, I have my doubts that quota is what is causing this. But who knows...
Hopefully Wayne agrees with my comments. Thanks for looking into this.

Comment 20

[Richard Leger]

Attachment: memory-report-01-noquota-at-startup.json.gz

(In reply to gene smith from comment #19)

(In reply to Richard Leger from comment #18)

(In reply to Wayne Mery (:wsmwk) from comment #13)

• Install Thunderbird 52 (32bits) on Windows

Probably better to use the latest release, 68.x.

• Setup my IMAP(Cyrus)/SMTP mailbox (without quota)
• Keep default preference settings
• Let it load folders/email as per default settings

Should record a memory report here.

Report 01

See attached the first report recording memory usage at startup, immediately after successful setup of the IMAP (Cyrus)/SMTP account while loading folders...

Client Environment

Windows 7 Pro SP1 (64bits)
HP ProDesk 405 G1 MT
4GB RAM - 300GB+ HDD

Server Environment

name : Cyrus IMAPD
version : 2.5.10-Debian-2.5.10-3ubuntu1 be9a1799 2016-10-18
vendor : Project Cyrus
support-url: http://www.cyrusimap.org
os : Linux
os-version : 4.15.0-101-generic
environment: Built w/Cyrus SASL 2.1.27
Running w/Cyrus SASL 2.1.27
Built w/Berkeley DB 5.3.28: (September 9, 2013)
Running w/Berkeley DB 5.3.28: (September 9, 2013)
Built w/OpenSSL 1.1.0g 2 Nov 2017
Running w/OpenSSL 1.1.1 11 Sep 2018
Built w/zlib 1.2.11
Running w/zlib 1.2.11
CMU Sieve 2.4
TCP Wrappers
NET-SNMP
mmap = shared
lock = fcntl
nonblock = fcntl
idle = idled

Comment 21

[Richard Leger]

Attachment: memory-report-02-noquota-after-3days15hours40minutes.json.gz

(In reply to gene smith from comment #19)

(In reply to Richard Leger from comment #18)

(In reply to Wayne Mery (:wsmwk) from comment #13)

• Install Thunderbird 52 (32bits) on Windows
  Probably better to use the latest release, 68.x.
• Let it seat for two days
• Take a memory report
  Running without quota, reporter says you should see an increase in memory use (the leak).

Report 02

Find attached second report of memory usage capture after few days of Thunderbird running non-stop with the same IMAP/SMTP mailbox setup and environment as per previous report/comment...

FYI, mailbox is about ~15GB in content size :-)

Comment 22

[gene smith]

Lots of data here and not sure what it all means. But when I compare the two .json files in kdiff3 it does look like most of the "amount" items have increased when running without quota in the server.

I guess you are doing the two reports again with a fresh profile/account and with quota in the server? The reporter says you should see no big change in the memory usage ("amount") between the two reports with quota set.

Just wondering which tool you are using to produce the reports? Is it only showing TB memory usage?
Edit: Never mind. I see it is the about:memory built-in tool.
Thanks.

Comment 23

[Richard Leger]

(In reply to gene smith from comment #22)

Just wondering which tool you are using to produce the reports?

I use the built-in Memory Use tool available in Thunderbird Menu > Help > Troubleshooting Information > about:memory

It allows to generate and save (anonynimised or not) Memory Use report as thoses published here.

Lots of data here and not sure what it all means.

That is the data generated by the tool used. So you may want to refer to the tool itself to understand the report.

But when I compare the two .json files in kdiff3 it does look like most of the "amount" items have increased when running without quota in the server.

The same tools also allow to load saved reports to view it on screen as well as compare two saved reports...

Might be a better way to read/compare reports than use third-party tools such as kdiff3... in this case...

I guess you are doing the two reports again with a fresh profile/account and with quota in the server?

Yes. The second phase of test is currently running. Results will be published on Friday (in two days).

Comment 24

[Richard Leger]

Attachment: memory-report-03-withquota-at-startup.json.gz

(In reply to gene smith from comment #19)

(In reply to Richard Leger from comment #18)

(In reply to Wayne Mery (:wsmwk) from comment #13)

• Exit Thunderbird
• Set a quota on my cyrus user mailbox
• Start Thunderbird (with existing profile)

Reporter says you will still see leaking unless you also delete and re-create the account here.

Report 03

See attached the third report recording memory usage at startup, immediately after successful setup of the IMAP (Cyrus)/SMTP account set with quota while loading folders...

Comment 25

[Richard Leger]

Attachment: memory-report-04-withquota-after-3ays0hours17minutes.json.gz

(In reply to gene smith from comment #19)

(In reply to Richard Leger from comment #18)

(In reply to Wayne Mery (:wsmwk) from comment #13)

• Let it sync and seat opened for two days
• Take a memory report again

Reporter says you should see no leak at this point with fresh account.

Report 04

Find attached fourth report of memory usage capture after few days of Thunderbird running non-stop with the same IMAP/SMTP mailbox setup set with quota and environment as per previous report/comment...

Comment 26

[gene smith]

Tried to use the Load and Diff button but the results don't look like a diff. So just used the Load button and looked at "explicit" for both sets. In both cases the report after 3 days reports some errors in the explicit category so not sure how accurate it is. Anyhow when I look at the explicit deltas it looks like without quota uses 113M more after 3 days and with quota uses 135M more after 3 days. So maybe worse with quota which is opposite of the reporter's observation. Not sure if statistically significant or if I'm even looking that the right thing in the reports.

Comment 27

[Wayne Mery (:wsmwk)]

Thanks gene for looking at this. I agree with your approach.

So we conclude there is nothing showing up that can be actioned?

Comment 28

[Worcester12345]

Has anyone had this issue with 64 bit Thunderbird?

Comment 29

[Richard Leger]

Attachment: 78.0b4.64-bit.memory-report.json.gz

I seems to be encountering this issue on my production Windows 10 Pro (64bits) machine currently running TB 78.0b4 (64-bit).

Find attached the memory report when I noticed Thunderbird starting to slow down after few days of use, and not responding intermittently without much reason to :-)

Sometime it is due to this Bug 1642292 but sometime not as it also happens when calendar is not syncing... at the moment the memory foot print of TB has increased to ~1.5GB!

Comment 30

[gene smith]

I really don't know how to analyze the memory report you attached. When you say "a few days of use" does this include any sleep or hibernate events occurring during this time? I have noticed recently, but only with a personal trunk build, that TB goes full tilt with the MainThread running at near 100% cpu and causing my laptop fan to go full speed. This is usually after several hibernates and/or sleeps and seemed to occur only in the evening. But haven't noticed it in the last few days. Probably not related to what you report.

Comment 31

[Richard Leger]

The memory report about:memory feature available in TB.

Yes I put computer to sleep in evening no hybernate though. Don't encounter issue with any other app such as Firefox (many tabs open) or Chrome (many tabs opened) only happens in Thunderbird.

Comment 32

[Wayne Mery (:wsmwk)]

Even if some leakage is detected over a period of days, I don't see this make progress because we don't have something that can be replicated. It would be more useful to do a shorter time frame with a controlled, specific list of steps, perhaps repeated multiple times. (such as moving the same 1000 messages). I think you'd need to "warm up" thunderbird first, for example with some actions which open the folders being used.

Comment 33

[Wayne Mery (:wsmwk)]

We have no other significant leak bug reports with activity in the last 3 years citing imap involvement https://mzl.la/3TBXyRY.
And the last comments suggest there may be no leakage related to imap quota.
So closing this bug.