1768765 - High memory use, causing systemd-oomd to kill Firefox nearly daily

Status: NEW

(Core :: Widget: Gtk, defect)

Description

[Asif Youssuff]

Attachment: memory-report.json.gz

I recently moved to Fedora, which uses systemd-oomd to help keep systems snappy: https://fedoraproject.org/wiki/Changes/EnableSystemdOomd and I very quickly found that Firefox was now being killed quite often, especially when I had other apps open.

I have a large session and have session restore enabled in Nightly, and this used to work for me when I was still using Ubuntu - I rarely saw lockups, for example.

Hopefully the attached about:memory dump is useful. Happy to troubleshoot further.

Comment 1

[Asif Youssuff]

Attachment: about:support

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Performance' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Asif Youssuff]

Okay, it looks like I had dom.ipc.processCount.webIsolated set to 100, which might explain some of this. I'll leave this open in case there is something interesting here. Ideally, Firefox would do something before the systemd-oomd killer came along and closed Firefox - that is really pretty distressing.

Comment 4

[Markus Stange [:mstange]]

Haik, do you know if we have working low memory detection on Linux? What would be an appropriate component for this bug?

Comment 5

[Haik Aftandilian [:haik]]

(In reply to Markus Stange [:mstange] from comment #4)

Haik, do you know if we have working low memory detection on Linux? What would be an appropriate component for this bug?

Yes, we do have low memory detection on Linux starting with Firefox 96 (bug 1532955). I'll move this over to the same category that was used for low memory detection.

@Asif, could you revert dom.ipc.processCount.webIsolated to the default value (4) and let us know if you continue to run into this problem?

From the memory report, the large number of processes is probably the cause.

Comment 6

[Asif Youssuff]

Attachment: memory-report.json.gz with default setting for dom.ipc.processCount.webIsolated

Haik, I went ahead and set dom.ipc.processCount.webIsolated to the default when I realized that this could be a part of the problem, and unfortunately, I still see the issue.

I have attached a new memory report.

I am also seeing reports on the web that the issue is more pervasive than I had previously assumed:

• https://www.reddit.com/r/linux/comments/uyl572/ubuntu_2204s_new_oom_killing_system_is_killing/
• https://www.reddit.com/r/Ubuntu/comments/uyl4i6/ubuntu_2204s_new_oom_killing_system_is_killing/

That part of this is likely external software causing a Firefox issue, but there is also a piece of this that is Firefox consuming a lot of memory. Not sure how you want to investigate and track this.

Comment 7

[Haik Aftandilian [:haik]]

@Asif, thanks. We are looking into this.

Adding the link to the Ubuntu bug report here too https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

Thanks for the info.

A few questions - it looks like you have ~110 extensions, correct? It certainly seems possible that one or more extensions might be bloating the memory use of processes (such as reddit, which are generally the largest processes, and you have several reddit-specific extensions).

When was the memory report taken? Right after restarting, or shortly before it got killed, or just randomly?

If you want to know what's really using memory when it kills firefox, run in a shell something like "top -d 30 -b -o "+%MEM" >/tmp/memory.txt". After firefox is killed, you can easily look back and see what process was using how much memory (roughly). For example, my top shows these as the largest:

2220214 jesup 20 0 19.9g 4.1g 859916 S 5.0 6.6 1766:16 firefox-bin
548842 jesup 20 0 5964084 3.6g 1.3g S 0.0 5.8 0:57.06 gdb
2123607 root 20 0 3213744 1.9g 20484 S 0.0 3.0 34:08.35 packagekitd
2220660 jesup 20 0 4268748 1.4g 112484 S 0.0 2.2 48:46.98 liveuamap.com
3086476 jesup 20 0 4318556 1.3g 120912 S 0.0 2.0 32:17.89 google.com
3007787 jesup 20 0 4271600 1.2g 114388 S 0.0 2.0 23:34.84 google.com
1919929 jesup 20 0 7894004 1.1g 113836 S 5.0 1.8 9213:56 gnome-shell
2811789 jesup 20 0 3718284 806020 120268 S 10.0 1.2 1028:08 cnn.com

Note: I have ~6900 tabs in ~30 windows... though only 91 tabs loaded at the moment.

Comment 9

[Asif Youssuff]

Randell,

it looks like you have ~110 extensions, correct?

Most are disabled - I count 30, but that includes the search extensions. I'll attach my about:support to this bug, since that may help a bit.

When was the memory report taken? Right after restarting, or shortly before it got killed, or just randomly?

A few minutes before it was killed - I could have likely made it limp along longer, but I opened some other apps and systemd-oomd kicked in.

If you want to know what's really using memory when it kills firefox, run in a shell something like "top -d 30 -b -o "+%MEM" >/tmp/memory.txt". After firefox is killed, you can easily look back and see what process was using how much memory (roughly).

Sure, happy to do so!

Comment 10

[Asif Youssuff]

Attachment: about:support

Comment 11

[Asif Youssuff]

Attachment: top-memory-may-29

Randell,

I went ahead and ran top -d 30 -bc -w512 -o "+%MEM" >/tmp/memory.txt - I have attached the last run as the attached file "top-memory-may-29".

I don't know how you got domains in your top output, so my output looks more like:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 112696 asif      20   0   21.6g   3.4g 399552 S   9.4  17.6 124:01.60 /home/asif/.local/bin/firefox-nightly/firefox-bin
 116416 asif      20   0 3514368 555896  99656 S   1.9   2.8   8:31.53 /home/asif/.local/bin/firefox-nightly/firefox-bin -contentproc -childID 48 -isForBrowser -prefsLen 44860 -prefMapSize 233057 -jsInitLen 280328 -parentBuildID 20220529090310 -appDir /home/asif/.local/bin/firefox-nightly/browser 112696 true tab
 116564 asif      20   0 3427828 540380 103576 S   1.3   2.7   8:37.28 /home/asif/.local/bin/firefox-nightly/firefox-bin -contentproc -childID 51 -isForBrowser -prefsLen 44860 -prefMapSize 233057 -jsInitLen 280328 -parentBuildID 20220529090310 -appDir /home/asif/.local/bin/firefox-nightly/browser 112696 true tab
 113384 asif      20   0 3478640 517528 101572 S   1.1   2.6   6:05.24 /home/asif/.local/bin/firefox-nightly/firefox-bin -contentproc -childID 14 -isForBrowser -prefsLen 44680 -prefMapSize 233057 -jsInitLen 280328 -parentBuildID 20220529090310 -appDir /home/asif/.local/bin/firefox-nightly/browser 112696 true tab
 116319 asif      20   0 3366868 498020  97380 S   2.0   2.5   6:58.49 /home/asif/.local/bin/firefox-nightly/firefox-bin -contentproc -childID 47 -isForBrowser -prefsLen 44860 -prefMapSize 233057 -jsInitLen 280328 -parentBuildID 20220529090310 -appDir /home/asif/.local/bin/firefox-nightly/browser 112696 true tab
   2185 asif      20   0 7064060 420576 225192 S   3.6   2.1  49:04.18 /usr/bin/gnome-shell
 112837 asif      20   0   27.4g 302512  88892 S   0.6   1.5  21:00.01 /home/asif/.local/bin/firefox-nightly/firefox-bin -contentproc -childID 1 -isForBrowser -prefsLen 38911 -prefMapSize 233057 -jsInitLen 280328 -parentBuildID 20220529090310 -appDir /home/asif/.local/bin/firefox-nightly/browser 112696 true tab
 112973 asif      20   0   49.0g 293588  87640 S   0.4   1.5   5:18.30 /home/asif/.local/bin/firefox-nightly/firefox-bin -contentproc -childID 3 -isForBrowser -prefsLen 39510 -prefMapSize 233057 -jsInitLen 280328 -parentBuildID 20220529090310 -appDir /home/asif/.local/bin/firefox-nightly/browser 112696 true tab
 128237 asif      20   0 3048148 263184  99028 S   0.1   1.3   0:41.03 /home/asif/.local/bin/firefox-nightly/firefox-bin -contentproc -childID 187 -isForBrowser -prefsLen 45250 -prefMapSize 233057 -jsInitLen 280328 -parentBuildID 20220529090310 -appDir /home/asif/.local/bin/firefox-nightly/browser 112696 true tab
  75145 asif      20   0   24.9g 245680  87000 S   2.6   1.2  16:27.04 /usr/lib64/signal-desktop/signal-desktop --type=renderer --enable-crashpad --enable-crash-reporter=e727a0ac-12ac-4792-a175-38d568c62704,no_channel --user-data-dir=/home/asif/.config/Signal --app-path=/usr/lib64/signal-desktop/resources/app.asar --no-sandbox --no-zygote --disable-seccomp-filter-sandbox --lang=en-US --num-raster-threads=4 --enable-main-frame-before-activation --renderer-client-id=4 --launch-time-ticks=20480751495 --share+

Hope there is something interesting in there.

Thanks!

Comment 12

[Gabriele Svelto [:gsvelto]]

Asif could you check the value reported in the files /proc/<pid>/oom_score for the PIDs corresponding to the processes in Firefox? Given the size of the main process I guess its score should be very high and we might need to tweak it (and the child process scores) to make sure that the OOM killer doesn't take it down.

Comment 13

[Asif Youssuff]

Gabriele, I wasn't entirely sure if you wanted me to capture the oom_score for each of the processes, so I took just a sample.

The main process score is 855 and content processes ranged between 804 and 808.

Thunderbird's parent process had a score of 823.

Comment 14

[Gabriele Svelto [:gsvelto]]

That's exactly what I needed, thank you! Given there's almost a 50 points difference between the main process and the content ones we'll probably need and adjustment in the 75-100 range.

Comment 15

[Michael De La Rue]

According to the documentation I found, systemd-oomd will ignore oom_adjust parameters. In Open Bug 1984223 I put a proposal for how to integrate with systemd-oomd so that this problem should be properly solved.