Open Bug 1974582 Opened 7 months ago Updated 9 hours ago

Firefox freezes Xorg when low on RAM

Categories

(Core :: Widget: Gtk, defect, P3)

Product:
Core
Component:
Widget: Gtk
Version:
Firefox 128
Platform:
Desktop
Linux
Type:
defect

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: falling_failing_falling, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

System info
56.76 KB, text/plain
Details

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36

Steps to reproduce:

  1. Running Firefox on Gentoo Linux (XFCE) with a lot of tabs opened with either:
    a) No RAM limit.
    b) RAM limit set through libcgroup's cgconfig and cgred.
    c) RAM limit set through libcgroup's cgconfig and cgred, SWAP limited to 0 Gb.

Actual results:

If no RAM limit is set ( case a ) :

  1. Firefox freezes.
  2. Xorg freezes.
  3. The whole system freezes but is recoverable after a lot of time.

If the RAM limit is set ( case b ) :

  1. Firefox freezes.
  2. Xorg freezes.

If the RAM limit is set ( case b ) and the swap is disabled (still by libcgroup) :

  1. Firefox freezes.
  2. Firefox freezes at startup when it tries to load the pages of the last session. So basically, Firefox becomes totally unusable.

Note I got the same result on those two computers:

  1. My 2021 laptop. Previously 16 Gb RAM, but now 32 Gb RAM. AMD Ryzen 7 4800H. GTX 1660 Ti dGPU (4 Gb model) + Radeon Vega Mobile iGPU.
  2. My 2019 tower. 32 Gb RAM. Intel Core i7 10700k. Radeon RX 5700 XT (8 Gb model).

Note that there's the exact same behaviour in Thunderbird.

Note that I was trying to set the RAM and SWAP limit of Firefox to either:
*) 2 Gb RAM + 2 Gb SWAP
*) 4 Gb RAM + 4 Gb SWAP
*) 6 Gb RAM + 6 Gb SWAP

None of them is enough since Firefox tries to eat the whole available RAM.

I'm on Firefox 128.0 ESR but there was the same behaviour in previous versions, and there's still probably the same behaviour on latest versions too.

Expected results:

Firefox should behave like Chromium by automatically unloading the pages without closing their tab instead of trying to eat the whole available RAM and SWAP.
I don't think there's a feature for doing what I want but it would greatly enhance the user experience.

Note that the steps to reproduce can be reproduced on any other Linux distributions.

Note that the Chromium feature that allows unloading tabs without closing them makes it able to be perfectly usable with 200+ tabs open with a RAM limit of 2 Gb.

The Bugbug bot thinks this bug should belong to the 'Core::Widget: Gtk' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Widget: Gtk
Product: Firefox → Core
Component: Widget: Gtk → Performance
OS: Unspecified → Linux
Product: Core → Firefox
Hardware: Unspecified → Desktop

Thanks for filing this bug. In this case I think the BugBot was correct about the component, so moving this back.

Your situation sounds sort of similar to [1], and that thread suggests earlyoom[2] as a potential fix. Curious if you've tried that--you'd likely wind up with crashed tabs instead of a frozen system, which seems like a step in the right direction.

In terms of Firefox diagnostics, have you looked at about:memory[3] or about:processes[4] to evaluate per-tab memory usage? How about using about:unloads[5] to unload tabs you're not actively using--does this reduce crash frequency? It would also be useful to get some system info from about:support and attach it to the bug.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1597028
[2] https://github.com/rfjakob/earlyoom, also available on gentoo directly
[3] https://firefox-source-docs.mozilla.org/performance/memory/about_colon_memory.html
[4] https://support.mozilla.org/en-US/kb/task-manager-tabs-or-extensions-are-slowing-firefox
[5] https://support.mozilla.org/en-US/kb/unload-inactive-tabs-save-system-memory-firefox#firefox:linux:fx128

Component: Performance → Widget: Gtk
Flags: needinfo?(falling_failing_falling)
Product: Firefox → Core

So far, firefox get memory usage from /proc/meminfo that is a global view that is not cgroups friendly. That means even firefox reach the limitation enforced by cgroup, it may found a lot of available memory from /proc/meminfo. We should consider to read memory statistics from memory.stat under /sys/fs/cgroup/ if there is.

Priority: -- → P3

Sorry if it took me 3 days to answer, I didn't know I received an answer.

In terms of Firefox diagnostics, have you looked at about:memory[3]

Yes, of course, and none of the buttons I clicked on there helped solving the issue.

or about:processes[4] to evaluate per-tab memory usage?

Yes, I did that too. I saw some tabs using more than 500 mb of RAM for some reasons.

How about using about:unloads[5] to unload tabs you're not actively using--does this reduce crash frequency?

Not really, and anyway, now Firefox crashes on startup when loading pages.

It would also be useful to get some system info from about:support and attach it to the bug.

I'll try it, ... hmm can't even reach it since nothing it clickable.
After starting Firefox, I can't even open tabs nor clicking on the menu items. I can't even close windows.
I had to disable cgroup temporally to get access to the about:support page.
I added an attachment with what you asked me but it's kinda in french sorry.

Flags: needinfo?(falling_failing_falling)
Attached file System info —

Your situation sounds sort of similar to [1], and that thread suggests earlyoom[2] as a potential fix. Curious if you've tried that--you'd likely wind up with crashed tabs instead of a frozen system, which seems like a step in the right direction.

I don't think earlyoom is a good solution, it would make tabs crash, but it might not be the right ones.
By the right ones, I mean there should be a "crashing" priority, so the oldest tabs should crash first, and only enough of them.
And it make close them completely instead of just making crash them.
So in my opinion, Firefox, should add the same feature that Chromium has.

After some weeks of usage with a CGroup RAM limit of 8 Gb for Firefox, I can say it is currently not enough since Firefox keep freezing after some hours of usage, at least it doesn't freeze on startup anymore.

Severity: -- → S3

An alternative to detect if we're running out of memory is by using the preallocated process as a canary to detect low memory conditions without polling or having to parse proc files. When we spawn new preallocated processes we set their oom_score_adj to ensure that the OOM killer will pick them up before any other process. If we observe one of the preallocated processes being killed with SIGKILL we can assume that memory was tight and the OOM killer reaped it from under us. Using that as a signal for low memory we can fire a memory-pressure event to shrink caches, run the GC and unload tabs.

Duplicate of this bug: 1979627
No longer duplicate of this bug: 1979627

Coming to this bug from #1979627 I've previously experimented with the earlyoomd, configuring it to target Isolated Web Component processes with some beneficial results.

One thing, though, is that I don't agree with @FellFromTheSky that it's the oldest processes that should always be unloaded. I see that groups of pages from the same site become extremely problematic simultaneously. Manually killing background tabs that are generating high CPU (thus thrashing memory??) seems to be very beneficial in stopping problems whilst killing older quiet ones doesn't seem to help.

My system has systemd-oomd installed by default though and that is reported to be incompatible with the earlyoomd so I'm not convinced of that as a solution.

I think though that, in general that seems to be too late. By the time the various oom killers are getting involved bad things are happening on the system. If instead tabs could be unloaded based on Thinker Li [:sinker]'s cgroup suggestion that might work better?

RAM usage of Firefox before killing it (8 hours of not so intensive usage), and RAM usage of Firefox after killing it.
As you can see

tux ~ # 
./calc_group_total.bash 
Per CGroup:
|    CGroup    |  mem.current  |   mem.high   |    mem.max    |  swap.current  |    swap.max    |
|    cgTeams    |      0       |     1,9G     |     2,2G     |      0       |      0       |
|  cgFirefox   |     17G      |     17G      |     18G      |      0       |      0       |
|   cgPaleMoon    |      0       |     1,9G     |     2,2G     |      0       |      0       |
| cgThunderbird  |     656M     |     1,9G     |     2,2G     |      0       |      0       |
|  cgChromium  |      0       |     1,9G     |     2,2G     |      0       |      0       |

TOTAL
|      All       |     18G      |     25G      |     26G      |      0       |      0       |
tux ~ # 
pkill firefox
tux ~ # 
./calc_group_total.bash 
Per CGroup:
|    CGroup    |  mem.current  |   mem.high   |    mem.max    |  swap.current  |    swap.max    |
|    cgTeams    |      0       |     1,9G     |     2,2G     |      0       |      0       |
|  cgFirefox   |     4,8G     |     17G      |     18G      |      0       |      0       |
|   cgPaleMoon    |      0       |     1,9G     |     2,2G     |      0       |      0       |
| cgThunderbird  |     656M     |     1,9G     |     2,2G     |      0       |      0       |
|  cgChromium  |      0       |     1,9G     |     2,2G     |      0       |      0       |

TOTAL
|      All       |     5,4G     |     25G      |     26G      |      0       |      0       |
tux ~ #
Blocks: 1979627

You may want to check and try following preference.

  • browser.tabs.unloadOnLowMemory
  • browser.tabs.min_inactive_duration_before_unload
  • browser.low_commit_space_threshold_percent
  • browser.low_commit_space_threshold_mb

I just realized that browser.tabs.unloadOnLowMemory is false by default. Turning it on may help.
browser.tabs.unloadOnLowMemory is how long before a tab being able to be unloaded.
browser.low_commit_space_threshold_percent is the percentage of memory being used to kick unloading.
browser.low_commit_space_threshold_mb is the megabytes of memory left to kick unloading.

On my side, turning "browser.tabs.unloadOnLowMemory" isn't helping at all.
I have it turned on since 3+ months, and I still got all the freezes I described above.

It doesn't respect cgroup as I mentioned before.
So, it may only work for case a. And, you may want to tune other preferences in order to work better with your case.

I also like you know the following patch that base on Linux PSI. You may want to tune the pref browser.memory_PSI_threshold_percent with this patch.

https://hg.mozilla.org/try/rev/f14c4d572dfc526b27f1a1a4bf8454978e62b163

I think that this enhancement proposal might help with this bug. https://bugzilla.mozilla.org/show_bug.cgi?id=1984223

On a different bug, we started discussing early-oomd, but it's more on topic in this bug

@FellFromTheSky - if the oomkiller kills a tab process, the tab itself remains and normally, currently, shows the crashed state, which allows it to be recovered. As I see it, that's not much different from firefox tab unloading. The idea, then, is that memory used by having tabs open can be reclaimed by the operating system when it needs that memory.

Currently I believe firefox is deliberately configuring the oom_adjust on tabs so that the behavior that you want - in other words it prioritizes background tabs, not the ones you are currently using, is more or less what should happen. Also, if the oomkiller kills a tab process, the tab itself remains and normally, currently, shows the crashed state, which allows it to be recovered. As I see it, that's not much different from firefox tab unloading. The idea, then, is that memory used by having tabs open can be reclaimed by the operating system when it needs that memory.

Maybe it would be worth turning on early-oom in Gentoo and seeing if it fixes anything?

Maybe it would be worth turning on early-oom in Gentoo and seeing if it fixes anything?

If the tab won't be removed, but only its process killed as you said, then early-oomd and bustd would only help if they can handle killing the tabs when the cgroup limit is reach, and I doubt it is currently supported.

See Also: → 1871576

@FellFromTheSky - earlyoom isn't cgroup aware, but you can set memory limits lower so that it starts killing before the browser's memory limit is reached. You might want to look at nohang (https://github.com/hakavlad/nohang ) which can at least match on cgroups and has much more flexibility than bustd or early-oomd.

I have tried turning on the earlyoom daemon with the following arguments

    --prefer "(^|/)(Isolated Web Co)$"  --avoid  "(^|/)(codium|emacs|chromium)$"

this caused tab process killing which meant that the system never locked up but was still inconvenient. I've then gone back and tried adjusting the tab unloading parameters, making them much more aggessive, e.g. setting browser.low_commit_space_threshold_mb to 8000 That seems to have stopped the killing of browser tab processes. I also think it's made everything much more responsive.

You might want to look at nohang (https://github.com/hakavlad/nohang ) which can at least match on cgroups and has much more flexibility than bustd or early-oomd.

I guess for now I should use it yes (or better alternatives if any), but in my opinion, Firefox should be aware of the CGroup limits by itself, and kill the tabs when OOM ...

Also, since some weeks, it's unrelated to memory issues, I replaced jackd and pulseaudio by pipewire's own implementations of jack and pulseaudio, result is, that when pipewire crashes, anything relying on audio becomes unplayable, and Firefox often freeze when trying to load some pages like Cloudflare.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: