1937218 - Thunderbird memory usage is more than 2.5GB (sometimes 8GB) after hours or days, with 600k messages, and intermittent hanging for 10 to 30 secs

Status: UNCONFIRMED

(MailNews Core :: General, defect)

Description

[mdaal]

Attachment: MSGDB:5,timestamp log

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15

Steps to reproduce:

When TB is left open for hours to days, I notice that we eats up a lot of system memory. I am doing nothing but occasionally check mail and sending mail.

Note: version is 128.5.2esr (64-bit). I don't know what this means. particularly 'esr'

Actual results:

system runs out of memory.

5 min after TB startup and it has already consumed 2.5GB, and raising

Expected results:

TB should use less than 2 GB of system memory.

Comment 1

[Wayne Mery (:wsmwk)]

esr = extended support release, which is the normal Thunderbird release channel.

Same problem after Help > Troubleshoot Mode?

Comment 2

[mdaal]

Attachment: Process Sample of Thunderbird.txt

Comment 3

[mdaal]

(In reply to Wayne Mery (:wsmwk) from comment #1)

esr = extended support release, which is the normal Thunderbird release channel.

Same problem after Help > Troubleshoot Mode?

Hi Wayne,

Thanks for looking into this.

Today, without making any changes to to TB settings, the memory usage is lower.

I open TB, wait 10 min, memory usage is 2.92GB.
I turn on Troubleshoot mode, re-open TB, wait 2 min, memory usage is 3.65G. Then at 7min its 1.99 GB. Then at 14 min its 3.03 GB.

I do not know if this is a related symptom (though it certainly another problem), every several minutes I get a spinning beach ball in the application. This happens whenever TB is in view, e.g. while reading message, composing message, etc. I will now see if I still get spinning beach ball in Troubleshoot mode.

Here is process sample taken prior to Troubleshoot mode, "Sample of Thunderbird.txt"

Comment 4

[Wayne Mery (:wsmwk)]

Please capture a minute of https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance
Thanks

Comment 5

[mdaal]

(In reply to Wayne Mery (:wsmwk) from comment #4)

Please capture a minute of https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance
Thanks

I captured 1 min. And uploaded the file. Here is the link
https://share.firefox.dev/3ZZyhVl

Sincerely
Miguel

Comment 6

[Geoff Lankow (:darktrojan) [PTO, back 2 Nov]]

You have a lot of mail. I think I can see some folders with over 100,000 messages. Not that there's anything wrong with that.

But it does mean a lot of memory is needed and some things take a while. Your profile has a couple of periods where Thunderbird is opening a very large message database and this takes a few seconds because of the size of the database. This'll be when you see the beach-balling.

Unfortunately the profiler tool doesn't capture memory allocation in release versions, so I can't see if there's anything in particular that's unexpected here. I don't think there is, and you mentioned memory use going down as well as up, which means Thunderbird is freeing up large chunks of memory when it has finished using them. That is working as it should.

Comment 7

[mdaal]

(In reply to Geoff Lankow (:darktrojan) from comment #6)

You have a lot of mail. I think I can see some folders with over 100,000 messages. Not that there's anything wrong with that.

But it does mean a lot of memory is needed and some things take a while. Your profile has a couple of periods where Thunderbird is opening a very large message database and this takes a few seconds because of the size of the database. This'll be when you see the beach-balling.

Unfortunately the profiler tool doesn't capture memory allocation in release versions, so I can't see if there's anything in particular that's unexpected here. I don't think there is, and you mentioned memory use going down as well as up, which means Thunderbird is freeing up large chunks of memory when it has finished using them. That is working as it should.

The high memory usage is due to the large amount of messages - Alright
TB seems to be coping because the TB is able to free up memory too - Alright
It is to be expected for TB to beach ball when opening a large message DB - Alright

But why do I get a beach ball intermittently when composing and reading ? (I don't think Im open or closing a DB then)
And why does the memory usage fluctuate so wildly? Sometimes its 2GB, right now its 5.2 GB, I have seen it at 8GB.

Comment 8

[Wayne Mery (:wsmwk)]

But why do I get a beach ball intermittently when composing and reading ? (I don't think Im open or closing a DB then)
And why does the memory usage fluctuate so wildly? Sometimes its 2GB, right now its 5.2 GB, I have seen it at 8GB.

Activity in the background.

What does Mac Activity Monitor show for swap used and physical memory?

Comment 9

[mdaal]

(In reply to Wayne Mery (:wsmwk) from comment #8)

But why do I get a beach ball intermittently when composing and reading ? (I don't think Im open or closing a DB then)
And why does the memory usage fluctuate so wildly? Sometimes its 2GB, right now its 5.2 GB, I have seen it at 8GB.

Activity in the background.

What does Mac Activity Monitor show for swap used and physical memory?

Currently, it's reporting 7.99 GB Memory, 2.10 GB swap and 16.00 GB physics memory. I attach a screen shot

Comment 10

[mdaal]

Attachment: Activity Monitor Screenshot 2024-12-21 at 7.15.14 AM.png

Comment 11

[Wayne Mery (:wsmwk)]

Thanks. Way too much memory.

Can you check the list ideas at https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems ?

Comment 12

[mdaal]

(In reply to Wayne Mery (:wsmwk) from comment #11)

Thanks. Way too much memory.

Can you check the list ideas at https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems ?

To be clear, are you saying I need to go through the 22 diagnostic steps? Are you asking for the information the each step yields?

Comment 13

[Wayne Mery (:wsmwk)]

We don't have an automated system to check each of those items in a user's environment, and you won't want us jumping on your machine, so we often must rely on what users are able to check on their own.

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #11)

Thanks. Way too much memory.

Can you check the list ideas at https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems ?

Result with 128.8.0 ?

Comment 15

[mdaal]

Attachment: Activity monitor for 128_8_0.png

I updated when I got your message one day ago. Since then TB had been open. It's using ~4GB now. Here is what I see on the activity monitor

My bigger problem now is that the program keeps hanging for 10 to 30 secs intermittently. Driving me crazy.

Comment 16

[mdaal]

Attachment: Activity monitor for 128.7 from yesterday before 128.8 upgrade

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to Geoff Lankow (:darktrojan) from comment #6)

You have a lot of mail. I think I can see some folders with over 100,000 messages. Not that there's anything wrong with that.

Ignoring speed for the moment, in the previously provided log I see max of 25 open dbs. Even if 25 dbs were open with 100k messages, that shouldn't be much more than 2.5gb (just for the dbs). So even a peak of 4-8GB Thunderbird total memory usage seems high.

But it does mean a lot of memory is needed and some things take a while. Your profile has a couple of periods where Thunderbird is opening a very large message database and this takes a few seconds because of the size of the database. This'll be when you see the beach-balling.

mdaal,

How many messages in your largest active Inbox and Archive folder?

Do you actually use the Unified Inbox? And the tags folder mode? How is memory usage with those modes disabled?

Comment 18

[mdaal]

Attachment: Screenshot of largest Folder. may not be largest active folder

I updated when I got your message one day ago. Since then TB had been open. It's using ~4GB now. Here is what I see on the activity monitor

My bigger problem now is that the program keeps hanging for 10 to 30 secs intermittently. Driving me crazy. (In reply to Wayne Mery (:wsmwk) from comment #17)

(In reply to Geoff Lankow (:darktrojan) from comment #6)

You have a lot of mail. I think I can see some folders with over 100,000 messages. Not that there's anything wrong with that.

Ignoring speed for the moment, in the previously provided log I see max of 25 open dbs. Even if 25 dbs were open with 100k messages, that shouldn't be much more than 2.5gb (just for the dbs). So even a peak of 4-8GB Thunderbird total memory usage seems high.

But it does mean a lot of memory is needed and some things take a while. Your profile has a couple of periods where Thunderbird is opening a very large message database and this takes a few seconds because of the size of the database. This'll be when you see the beach-balling.

mdaal,

How many messages in your largest active Inbox and Archive folder?

Do you actually use the Unified Inbox? And the tags folder mode? How is memory usage with those modes disabled?

1 - It depends on what you mean by 'Active'. I have been using TB for 20 years and all my mail stored in it. I had an account in grad school with >200,000 messages and when the account for retired by the school I transferred all that mail to my main personal [e-mail] IMAP account as a sub-folder. It is not an archive folder. That sub-folder gets no new messages. (It's not supposed to at least.) Screenshot attached.

I have another personal account, [e-mail], which gets all communications from commercial sources. It is active in the sense that it gets new mail daily. in that sense it has my largest active inbox.

I have not been using archive folders. I would need to look up what an archive folder is for and how archived messages within it are treated and their accessibility.

2 - I do not use the unified inbox feature

3 - I did not know there is a tags folder mode. Im guessing this creates special folders into which all messages with a common tag are placed. Or do you mean View > Folders >Tags (unchecked)?

4 - To answer "How is memory usage with those modes disabled?" Can you clarify if you mean View > Folders >Tags (unchecked)? Presently it is checked. And I don't use Unified Folders (checked), so the memory info sent is assuming that.

Comment 19

[Wayne Mery (:wsmwk)]

Sorry for the long delay in replying.

(In reply to mdaal from comment #18)

Created attachment 9472492 [details]
Screenshot of largest Folder. may not be largest active folder

I updated when I got your message one day ago. Since then TB had been open. It's using ~4GB now. Here is what I see on the activity monitor

My bigger problem now is that the program keeps hanging for 10 to 30 secs intermittently. Driving me crazy.

How is version 140?

(In reply to Wayne Mery (:wsmwk) from comment #17)

(In reply to Geoff Lankow (:darktrojan) from comment #6)

You have a lot of mail. I think I can see some folders with over 100,000 messages. Not that there's anything wrong with that.

Ignoring speed for the moment, in the previously provided log I see max of 25 open dbs. Even if 25 dbs were open with 100k messages, that shouldn't be much more than 2.5gb (just for the dbs). So even a peak of 4-8GB Thunderbird total memory usage seems high.

But it does mean a lot of memory is needed and some things take a while. Your profile has a couple of periods where Thunderbird is opening a very large message database and this takes a few seconds because of the size of the database. This'll be when you see the beach-balling.

mdaal,

How many messages in your largest active Inbox and Archive folder?

Do you actually use the Unified Inbox? And the tags folder mode? How is memory usage with those modes disabled?

1 - It depends on what you mean by 'Active'. I have been using TB for 20 years and all my mail stored in it. I had an account in grad school with >200,000 messages and when the account for retired by the school I transferred all that mail to my main personal [e-mail] IMAP account as a sub-folder. It is not an archive folder. That sub-folder gets no new messages. (It's not supposed to at least.) Screenshot attached.

I have another personal account, [e-mail], which gets all communications from commercial sources. It is active in the sense that it gets new mail daily. in that sense it has my largest active inbox.

I have not been using archive folders. I would need to look up what an archive folder is for and how archived messages within it are treated and their accessibility.

Description at https://support.mozilla.org/en-US/kb/archived-messages

2 - I do not use the unified inbox feature

3 - I did not know there is a tags folder mode. Im guessing this creates special folders into which all messages with a common tag are placed. Or do you mean View > Folders >Tags (unchecked)?

These are the same.

4 - To answer "How is memory usage with those modes disabled?" Can you clarify if you mean View > Folders >Tags (unchecked)? Presently it is checked. And I don't use Unified Folders (checked), so the memory info sent is assuming that.

Try with unchecked.

Comment 20

[mdaal]

Thanks for responding. I am currently using 140.2.1esr.
There is no improvement on any of the issues. Thunderbird is using ~4GB or more. The application still hangs for 10 - 30 seconds. I still have the majority of my mail unarchived.

Comment 21

[Wayne Mery (:wsmwk)]

mdaal,

How many fonts are installed per "Font Book"?
There are no calendars, nor add-ons?

Can you install https://support.mozilla.org/en-US/kb/thunderbird-daily and configure the two imap accounts, so that you can capture memory usage in a new performance profiler?

Comment 22

[Wayne Mery (:wsmwk)]

Miguel ?

Version 140 would have had the fix for Bug 1933104 (v134) - Message list tree can waste a significant amount of time creating rows that are never seen, which I think is the last major message list fix.

(In reply to Wayne Mery (:wsmwk) from comment #21)

mdaal,

How many fonts are installed per "Font Book"?
There are no calendars, nor add-ons?

Can you install https://support.mozilla.org/en-US/kb/thunderbird-daily and configure the two imap accounts, so that you can capture memory usage in a new performance profiler?

Also, how many messages in your largest folder, and what is the size of the .msf file ??

Comment 23

[Wayne Mery (:wsmwk)]

Miguel seems to be gone. Still needs more information, so => incomplete

Comment 24

[mdaal]

(In reply to Wayne Mery (:wsmwk) from comment #23)

Miguel seems to be gone. Still needs more information, so => incomplete

Hi Wayne,

Very Sorry for the lack of response! (I replied to your message on LinkedIn though!)

This problem has not improved. TB uses >2GB memory and it hangs all the time. I have been bad at at responding because I took a new job and its really taking up all my time.

Sincerely
Miguel

Comment 25

[Wayne Mery (:wsmwk)]

Thanks for the update.

This problem has not improved. TB uses >2GB memory and it hangs all the time

With version 140? Or 148?
Can you get a new profile?

Comment 26

[mdaal]

Hi Wayne,

I am using 140.8.1esr.
I have not tried creating a new profile since the last time we did that, as a suggestion from you. I dont even remember how do do that.

Thank you for trying to help!

Sincerely
Miguel

Comment 27

[Jonathan Kamens]

Thunderbird beta is regularly using so much memory that it causes my (Linux) computers to OOM and kill TB. This happens on at least two different computers for me regularly.

On one computer, this happened on March 11, March 19, and April 1.

On the other one, it happened on March 11, March 16, March 19, March 24, March 27, March 29 (twice!), and April 4.

As such, while this isn't reproducible "on demand," it happens frequently that if there's useful information or logs I can collect to help debug this, please let me know and within a few days I can probably get them.

Comment 28

[Wayne Mery (:wsmwk)]

(In reply to Jonathan Kamens from comment #27)

Thunderbird beta is regularly using so much memory that it causes my (Linux) computers to OOM and kill TB. This happens on at least two different computers for me regularly.

Please create a new bug report and use an MZLA build to ensure that the profile has symbols. Daily builds get the best memory details. Limit the profile to about one minute duration.

Comment 29

[Wayne Mery (:wsmwk)]

(In reply to mdaal from comment #26)

I have not tried creating a new profile since the last time we did that, as a suggestion from you. I dont even remember how do do that.

https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Comment 30

[Jonathan Kamens]

Your instructions seem to assume that this happens when I am sitting at the computer and I know how to reproduce it. Neither of these is true. Pretty much every time TB has OOM'd for me it has happened overnight, not when I was actively using TB. GIven this, it seems unlikely that the memory increases are gradually and steady; rather, it seems like something is specifically happening overnight when I am not actively using TB that is causing a sudden increase in memory. I don't think the memory profiling instructions you sent, which say to do profiling for a short period of time while reproducing the issue, can be followed in these circumstances. Please advise.

Comment 31

[Jonathan Kamens]

Attachment: memory_usage.png

This is certainly interesting. After the recent exchange here I decided to collect more data by running thunderbird on all of my Linux computers inside pidstat and logging memory usage once per minute. The attached graph, memory_usage.png, shows the start of one of those runs. The data points in this graph are a minute apart, starting with thunderbird launches. Right from the start memory usage starts careening crazily, fluctuating between less than 10% of device memory to as high as over 50%. This presumably means that I could capture one of these swings entirely if I profiled Thunderbird (using the previously provided instructions) for ten minutes. Is that worth doing, or would that be too much data?

Comment 32

[Wayne Mery (:wsmwk)]

I could capture one of these swings entirely if I profiled Thunderbird for ten minutes.

You must use an MZLA provided install, preferably daily build. Do all the steps under "Configuring profiler settings". Save the profile locally. Submit also the Mail and News Accounts section of Help > Troubleshooting Information.

Comment 33

[Jonathan Kamens]

I do not get a connection prompt when I start the developer tools, but this doesn't seem to prevent the profiler from capturing data. Perhaps the instructions page you sent is out-of-date and that connection prompt no longer happens?

The first time I attempted to capture profiling TB hung and I had to kill it.

The second time it displayed "Failed to get the active browserId whiel starting the profiler" in the error console when I started profiling, but it seems to have successfully collected profiling data anyway.

I am not sure what you mean when you say "Save the profile locally". Do you mean I should click the "Upload Local Profile" button and then click "Download" in the pop-up, or something else?

If I am capturing ten minutes of data, how often should I do sampling? If 1ms is the appropriate frequency for up to a minute of data, does that mean I can do 10ms sampling for 10 minutes of data without worrying about the profile data getting too obscenely large?

Sorry about all the questions here, just want to make sure I'm doing the right thing.

I'm going to let nightly run overnight tonight to confirm that it is reproducing the issue and then capture a profile based on your answers to the questions above.

Comment 34

[Jonathan Kamens]

sigh apparently I can't gather profiling data because TB nightly is consistently hanging a few seconds after I start profiling.

The fact that it's not displaying a connection prompt when I open the developer tools may or may not be relevant. :shrug:

Comment 35

[Wayne Mery (:wsmwk)]

Sorry, correct wording is indeed download. There may also be recent changes regrading connection

If nightly isn't feasible, beta is next best thing. Also, oauth prompts can complicate things.

Comment 36

[Jonathan Kamens]

Unfortunately, beta is also hanging a few seconds after I start profiling.