1743351 - Memory spike when closing 200-300 tabs at once

Status: REOPENED

(Core :: Performance: General, defect, P3)

Description

[Triffid Hunter]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0

Steps to reproduce:

Close multiple tabs → Close tabs to the right, with many tabs to be closed - 200-300 in my typical case.

This is repeatable for me, I have tried it several times with similar results each time.

Actual results:

The tabs closed, then Firefox's memory usage climbed to over 40GB before snapping back to normal (3-5GB) about half a minute later.

This has caused firefox to oom-crash for me several times, I had to add extra swap to determine whether the memory usage grew without bounds or reached a maximum and recovered.

Expected results:

Firefox should not require dozens of gigabytes of extra RAM to close some tabs.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Performance' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Andrew McCreight [:mccr8]]

Unfortunately it does take some memory to clean up tabs. Paul Bone has done some work recently to reduce the impact from garbage collection in a bunch of processes at once, but there are probably other clean up routines that can run.

Our work on process separation (known as Fission), which is starting to roll out to users in Firefox 94, might help this situation because we'll kill off content processes without doing as much work.

Comment 3

[Triffid Hunter]

Some memory, sure - but over a hundred megabytes more than the tabs were using initially per tab, and cumulative in a way that looks like a severe memory leak (~1GB/sec or so) until the last closing tab is handled or firefox oom-crashes?

Fwiw, Firefox will restore the ostensibly closed tabs at next startup if it oom-crashes before completion, so users who don't think to add swap will have to either close them one by one, or be stuck with them forever.

Well, I'm glad this is now on your list of things to check at least ;)

Comment 4

[Olli Pettay [:smaug][[e-mail]]]

pbone, remind me, how many CCs can we run simultaneously.

Comment 5

[Triffid Hunter]

I uploaded a video of my system monitor tracking anomalous memory usage at https://triffid-hunter.no-ip.info/firefox-memory.mp4 if anyone's curious, it starts almost immediately after I select "close tabs to the right"

Also note that the freshly revealed tab doesn't begin to load for over two minutes, and takes dozens of seconds to load due to (presumably) GC churning

Comment 6

[Paul Bone [:pbone]]

(In reply to Olli Pettay [:smaug] from comment #4)

pbone, remind me, how many CCs can we run simultaneously.

This is almost certainly CCs, because CCs have no control in this situation unlike GCs, and I'm pretty sure CCs require temporary memory to be allocated to clean them up.

This increases the priority of Bug 1693712.

Comment 7

[Triffid Hunter]

Hi all,

My package manager updated me to Firefox-97 recently, and ff97 does not exhibit the behaviour described in this bug.

There's a very mild and entirely manageable RAM bump when closing a large number of tabs, and it seems to complete in at most a few seconds without stalling the browser.

Since I can no longer reproduce the problematic behaviour in this updated version, I think we can close this bug.

I can't find a way to mark it as RESOLVED→FIXED though

Comment 8

[Markus Stange [:mstange]]

Great, thanks for the update! Resolving the bug.

Comment 9

[Triffid Hunter]

Hi all,

Seems there's a regression on this one, I'm currently running 103.0.1 and closing a couple hundred tabs caused a ~20GB spike in memory usage.

Comment 10

[Triffid Hunter]

And gone again in 103.0.2.. How strange

Comment 11

[Triffid Hunter]

Definitely regressed in 104.0.1.

After closing a window with maybe 30 tabs: https://i.imgur.com/lg05UC5.png

After closing around 100 tabs "to the right" without closing the window: https://i.imgur.com/DCIQ18e.png

Note: graph scale is 32GB of RAM and 64GB of swap - so those spikes on tab/window closure are exceeding 16GB above baseline (while firefox itself is only using maybe 4GB), and almost completely locks the browser for almost 10 seconds.

No other applications are doing anything during these events, those memory spikes are 100% Firefox.

Should I make a new bug about the regression since this one is closed?

Comment 12

[Andrew McCreight [:mccr8]]

No code landed in this bug, so I think reopening it is fine.

Comment 13

[Markus Stange [:mstange]]

Could you help us in making this easy to reproduce on our machines? For example, could you write down a few URLs of your 200 tabs, and then check if the same problem happens if you only load those URLs in a hundred tabs or so?

Comment 14

[Triffid Hunter]

I tried opening reddit a bunch of times then closing them all once the CPU usage settled, but it didn't do the thing - I'll keep trying to find a repeatable way to produce the issue and report back when I find one 👌

Comment 15

[Triffid Hunter]

Hmm, just had firefox-104.0.2 CTD after closing 250-ish tabs - starting to wonder if the tabs need to be 'unloaded' by the "tab suspender" addon to trigger the issue

Comment 16

[Triffid Hunter]

Tried closing 336 tabs with the profiler running after I noticed a bump of a few GB after closing just 3 tabs - but Firefox CTD again and seems to have lost the profiler log.

Last screenshot before it crashed ( https://i.imgur.com/DA8kfhu.png ) showed a memory usage of 35GB above baseline (~10GB for whole system), as well as a huge amount of system CPU and iowait (blue & orange in top graph, green is user CPU) which suggests that Firefox is thrashing the utter heck out of my NVMe (2B2QEXM7, claims 3.3GB/s write and 420k IOPS) while it's choking

Ironically, after reopening firefox, all the 'closed' tabs have been restored - but the memory spike was much smaller when I closed them all after a browser restart, so it seems to be an issue that accumulates with usage over time somehow.

I grabbed this second run with the profiler, but since the issue didn't occur due to the browser restart I guess the log won't be too useful - but it's at https://share.firefox.dev/3BSNqe2 if you want to peruse it anyway.

Comment 17

[Triffid Hunter]

Attachment: memory-report-2.json.gz

Apparently the issue also occurs if I close tabs one by one, I grabbed a memory report from about:memory while it was happening.

This is from firefox 106.0.5 by the way.

I observed via a system monitor that 'WebExtensions' (pid 7322) was chewing multiple gigabytes while the issue was occurring, even after only closing 5-10 tabs - then dropping back to 170MB or so about 10-30s later.

I guess the tail of zcat memory-report-2.json.gz | jq -r ".reports[] | select(.process == "WebExtensions (pid 7322)") | (.amount | tostring) + ", " + .description" | sort -g may be interesting :-

14620080, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
14769952, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
14780304, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
14788448, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
14844256, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
15098992, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
18726912, GC arenas in non-empty chunks that is decommitted, i.e. it takes up address space but no physical memory or swap space.
21659168, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
21661696, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
21708208, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
21715616, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
21743680, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
21770752, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
21797840, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
21908352, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
22020592, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
22091504, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
28873504, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
29036448, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
29039440, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
29231456, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
32984272, Objects, including fixed slots.
32984272, Used object cells.
86438736, Committed bytes which the heap allocator uses for internal data structures.
123545096, The sum of 'explicit/heap-overhead/*'.
127307323, The number of soft page faults (also known as 'minor page faults') that have occurred since the process started. A soft page fault occurs when the process tries to access a page which is present in physical memory but is not mapped into theprocess's address space. For instance, a process might observe soft page faults when it loads a shared library which is already present in physical memory. A process may experience many thousands of soft page faults even when the machine has plenty of available physical memory, and because the OS services a soft page fault without accessing the disk, they impact performance much less than hard page faults.
583470352, Non-inline Latin1 string characters. Sometimes over-allocated to simplify string concatenation.
6751058424, Memory mapped by the heap allocator that is currently allocated to the application. This may exceed the amount of memory requested by the application because the allocator regularly rounds up request sizes. (The exact amount requested is not recorded.)
6751058424, The same as 'heap-committed/allocated'.
6899089408, Memory mapped by the process that is present in physical memory and not shared with any other processes. This is also known as the process's unique set size (USS). This is the amount of RAM we'd expect to be freed if we closed this process.
6982930432, Memory mapped by the process that is present in physical memory, also known as the resident set size (RSS). This is the best single figure to use when considering the memory resources used by the process, but it depends both on other processes being run and details of the OS kernel and so is best used for comparing the memory usage of a single process at different points in time.
7199522816, Amount of memory currently mapped. Includes memory that is uncommitted, i.e. neither in physical memory nor paged to disk.
11154886656, The peak 'resident' value for the lifetime of the process.
17309040640, Guard pages mapped after the end of wasm memories, reserved for optimization tricks, but not committed and thus never contributing to RSS, only vsize.
27156295680, Memory mapped by the process, including code and data segments, the heap, thread stacks, memory explicitly mapped by the process via mmap and similar operations, and memory shared with other processes. This is the vsize figure as reported by 'top' and 'ps'. This figure is of limited use on Mac, where processes share huge amounts of memory with one another. But even on other operating systems, 'resident' is a much better measure of the memory resources used by the process.

Comment 18

[Triffid Hunter]

Interestingly, the memory usage of the parent firefox process seems to follow WebExtensions in consumption too, although drops back to baseline on a somewhat longer schedule

Also, both seem to consume less memory and settle to baseline faster when the window has fewer total tabs.

I'll have to check if opening several hundred tabs can immediately reproduce the issue, or if they need to stew for a bit first and maybe unload from inactivity (via Tab Suspender addon)

Comment 19

[Triffid Hunter]

This still occurs in Firefox-110.0, but it seems to be slightly more manageable - closing a few hundred tabs only spiked from ~1GB to ~23GB before dropping back to baseline a couple minutes later

Comment 20

[Triffid Hunter]

Just encountered this again in Firefox-120.0, https://i.imgur.com/OhjDJ6k.png shows a ~24GB spike (blue is RAM, total 32GB, orange is swap, total 16GB) followed by firefox crashing after closing 495 tabs.

After reopening firefox, the tabs were back, but this time it only had a small 2-3GB spike when I closed them.

It seems like something is accumulating the longer I have Firefox open that inflates the memory consumption of closing tabs.

Comment 21

[Triffid Hunter]

I've currently got Firefox-127.0 in a state where closing a single tab will cause an 8GB memory spike for 45 seconds, as well as causing short UI stalls for all windows.

See https://i.imgur.com/3Z825OD.png

Comment 22

[Triffid Hunter]

Hi, this issue doesn't seem to be occurring in Firefox-133.0 - I closed ~300 tabs a couple of times and the memory usage didn't jump at all, and I didn't notice any UI stalls either.

Comment 23

[Triffid Hunter]

Welp, this bug back in the 140 series with a vengeance, currently on 145 and just had a CTD after closing only a dozen tabs, although I did have 1095 tabs remaining.

Also had several crashes from closing multiple tabs with 143 and 144.

Seems like there's a long UX stall on all windows while it decides if it's gonna crash or recover, and sometimes it seems like attempting to send UI events to the windows increases the chances of a crash although this is anecdotal and difficult for me to instrument.

Comment 24

[Triffid Hunter]

Also seems like this bug only triggers after Firefox has been open for a while, after reopening from a crash I can close the same set of tabs (or even a much larger set) no problem - the UI stall remains though.

Comment 25

[Triffid Hunter]

https://i.imgur.com/yRTTSrC.png - ~18GB memory spike just before Firefox-148.0.2 crashed about 20 seconds after closing 641 tabs with "close tabs to right"