1814335 - Memory Leak using Filters & Templates

Status: RESOLVED INCOMPLETE

(Thunderbird :: Untriaged, defect)

Description

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

102.6.0 on Debian 11.

1. Create a few ( <5) message filters. Create a template.
2. One filter takes messages directed to another email address (a copy is forwarded to my account by Zoho) and puts them in a special folder with a todo tag
3. I grab the messages and verify the validity of the attachment
4. Then I reply using a template, giving the sender a promo code to get a deal we offer.
5. Finally I tag the message DONE.

Actual results:

When I handle the messages in that folder, memory usage starts growing like crazy. Memory in use will shoot from 500M (-ish) to 800M (-ish) from processing two messages as described.

Memory grows steadily anyway but more slowly. I have to restart T-Bird at least once a day to keep it under 1GB. After one incident about a month ago where it took over my entire 128G workstation, I put a monitor on my panel just to keep an eye on it.

Expected results:

Memory usage should remain relatively constant? I would THINK?

Comment 1

[Kahomono]

Should note - workstation is Debian 11 but running backport kernel 6.0
full uname -a is:
Linux clever-machine-name 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux

Comment 2

[Kahomono]

I use birdtray, but quitting it and starting T-Bird directly didn't change this behavior. Neither did starting T-Bird in --safe-mode.

Comment 3

[Wayne Mery (:wsmwk)]

For clarity, I have numbered your steps to reproduce.

Are steps 1 and 2 really required to reproduce the memory loss? Because you write "When I handle the messages in that folder, memory usage starts growing like crazy." which is step 3 only.

Also, I'm not finding any bug reports about memory loss related to filters, other than incomplete bug 1619703 - Can’t add new filters, results in high CPU and memory until OOM

Comment 4

[Kahomono]

(In reply to Wayne Mery (:wsmwk) from comment #3)

For clarity, I have numbered your steps to reproduce.

Are steps 1 and 2 really required to reproduce the memory loss? Because you write "When I handle the messages in that folder, memory usage starts growing like crazy." which is step 3 only.

Memory starts growing even if I do nothing but allow new email to trickle in as it does. But, it's slower.

Also, I'm not finding any bug reports about memory loss related to filters, other than incomplete bug 1619703 - Can’t add new filters, results in high CPU and memory until OOM

I cannot tell if it's because of the filter or because of using the template to reply. My monitor on it samples every 30sec. I will try to see if I can get something more real-time on the case.

Comment 5

[Kahomono]

OK - I wrote a script that grabs the memory size of T-Bird at 1sec intervals and outputs it if changed.

#! /bin/bash
Prev=0 
while true; do
  Now=`ps -C thunderbird -o size --noheaders`
  if [[ $Prev -ne $Now ]]; then
    echo `date +%R:%S` $Now
    Prev=$Now
  fi
  sleep 1
done 

Then I processed one of the emails. Here is the sequence of events:

11:28:37 498144
11:28:39 499160 < entered folder with "deal" emails
11:28:46 576968 < opened a PDF attachment
11:28:47 560584
11:28:48 576968
11:28:50 581064 < Reading the PDF to see if it meets requirements
11:28:51 579032
11:28:52 554304
11:28:54 543784
11:28:55 579616
11:29:00 580632 < saved PDF attachment
11:29:04 572316
11:29:05 595164
11:29:11 596180 < copied name & email address from message
11:29:13 599236
11:29:16 597188
11:29:17 620760 < switch to Templates folder
11:29:20 803720 < Write New Message from Template
11:29:24 822196 < Paste in name & email
11:29:25 821180 < Send
11:29:29 818132
11:29:30 874488 < Sent
11:29:35 685640
11:29:41 687680
11:29:42 688704
11:29:45 680432 < Tag deal email "Done"
11:29:46 692732
11:29:55 693756 < Untag from "ToDo"
11:29:57 694772
11:30:02 695788
11:30:09 692732
11:30:16 693244

And now, without doing ANYTHING more, the memory has crept up to 708852

Comment 6

[Wayne Mery (:wsmwk)]

Mike, can you reproduce comment 0?

Comment 7

[Mike Cloaked]

My filter setup is somewhat different to that used by the author of the bug - I have been tracking the memory used by Thunderbird
(currently 111.0b4) when running for a while, and I also deliberately manually re-ran my filters that include parsing all emails adding a tag that depends on checking each mail for dkim and dmarc pass, and memory usage has been pretty constant at 5.3% of available memory. The Thunderbird version running for the original reporter at the first comment is 102 ,so it is not clear that a direct comparison can be made since I am running a very recent beta version. However I have not seen any memory increases for any of the recent beta versions of Thunderbird on several systems here, each with different, though similar, profile directories and files, and all running in Arch Linux. Comparison with a much older version of Thunderbird running in Debian may not be a valuable comparison though. So the simple answer is that I have not seen any rise in memory used by Thunderbird - but the specific set up for this bug report has different filters and templates involved.

Comment 8

[Arthur K. (he/him)]

(In reply to Kahomono from comment #1)

Should note - workstation is Debian 11 but running backport kernel 6.0
full uname -a is:
Linux clever-machine-name 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux

I would think this is not a kernel issue but something wonky happening with your rather fancy filter setup and it perhaps not releasing RAM or something but the PDF opening seems to be culprit our of the gate:

11:28:46 576968 < opened a PDF attachment

RAM goes from 500MB to 580MB, some other interop actions happen, then we see a big RAM spike from the Template usage going from 580MB to around 803MB

11:29:20 803720 < Write New Message from Template

Finally, a small bit of RAM gets reclaimed after sending and we decrease down to 685MB:

11:29:35 685640

And we finally settle down to around 693MB RAM use:

11:30:16 693244

I surmise you've already bumped up to 102.8.0 and this still happens? Just staring at the breakdown of RAM usage in comment 5, it's making me think it's not freeing RAM after the PDF opening and template application action.

Something that might give insights into this and what's happening under the hood would be to get a perf profile. Can you try this for me please?

1. Ctrl-shift-i to open Developer Toolbox
2. Look for a blinking Incoming Connection window to appear (or be hiding behind a window) and click OK when it's asking to "Allow Connection?"
3. In the Browser Toolbox Window, press F1 and from the Default Developer Tools at top left, click the checkbox for "Performance"
4. You'll see the Performance tab appear, click on it
5. You'll see a Start Recording Button and a Settings menu drop down. Do Custom for the drop-down menu and enable these menu options:

Threads: GeckoMain, Compositor, DOM Worker, Renderer, RenderBackend, Timer, StyleThread, Socket Thread, StreamTrans, ImgDecoder, DNS Resolver, TaskController

Features (Recommended on by default):
Native Stacks, JavaScript, CPU Utilization

Features:
Main Thread File IO, Profiled Threads File IO, All File IO, Preference Read, IPC Messages, JS Allocations

Experimental:
CPU Utilization - All Threads, Markers - All Threads, Process CPU Utilization, Power Use

6. If you are able to replicate this fairly easily, set up TB to be ready to process one of these actions and click the Start Recording Button from step 5 and let your filter do it's thing
7. Once a filter run completes its task and you think TBs is done and has chewed up some RAM, click Capture Recording
8. You should in short order see TB show you a profile capture screen of everything that just happened from start to finish. At the top right, click the Upload Local Profile and check all the boxes presented and click the Upload button. It'll send it anonymized to the Mozilla and then present you with a link in the form of https://share.firefox.dev/<some alphanumeric string> that we can all look at and examine what's happening from A-Z.

Paste that share.firefox.dev link in this bug.

I'm going to give the STR a try on Windows in a bit.

Comment 9

[Arthur K. (he/him)]

I wasn't able to closely replicate this as it's quite a complicated setup with the PDF parsing and template ingestion. I am curious how much of an outlier setup this is because I've never heard of one this complicated before. I wasn't aware TB even had the ability to parse a PDF. Kahomono, how exactly does TB parse/read the PDF to extract it's info (per your filters) and ingest into a template? I struggle to conceive of how to set up TB to do that.

Comment 10

[Wayne Mery (:wsmwk)]

Further about getting data about memory usage with profiling - see https://profiler.firefox.com/docs/#/./memory-allocations and configure for "native allocations". See also https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Comment 11

[Arthur K. (he/him)]

Has TB 102.9.0 been offered as an update yet? Can you try with it once it's offered as well as please try the suggestion in comment 8?

Comment 12

[Wayne Mery (:wsmwk)]

Does this also reproduce when using version 115 started in Help > Troubleshoot Mode?
If it does, and you have not already done so, please list complete steps to reproduce.