1615988 - Very high virtual memory usage in the WebExtensions process on Linux

Status: RESOLVED FIXED

(Core :: JavaScript: WebAssembly, defect, P3)

Description

[Daniel Roi]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

As soon I start Firefox v73 64bit on my Linux system, the htop utility reports 32.4 GB VIRTUAL memory usage and 221 RES. After a couple of seconds, usage is "lowered" to 26.4 GB VIRT and 158 MB RES.

This is with no tabs open, but I have a dozen extensions installed, each of which only uses between 0.4 to 1.4 mb according to about:performance

This issue has been reported years ago by other users at https://bugzilla.redhat.com/show_bug.cgi?id=1288888 but no solution has been given.

Also note, as reported above, that not all instances report high virtual memory usage (only those with the "-childID 2" parameter, but not other childIDs).

Comment 1

[Daniel Roi]

(Please disregard the User Agent reported above, I'm using a different pc for this bug report.)

Comment 2

[Andre Klapper]

Please enter the address
about:memory?verbose
in the address bar and attach (using the "Attach File" link above) the output here.

Comment 3

[Daniel Roi]

Attachment: memory-report.json.gz

Comment 4

[Daniel Roi]

Let me know if you need other information.

Comment 5

[Clara Guerrero ( Need Info Brindusa Tot Please)]

Hi Daniel,

You mention you can reproduce this issue even with no tabs opened (but mention you have several add-ons installed), have you tried safe mode? (add-ons disabled). Here is a link that can help you do that:
https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode

Also, try this on the latest version of nightly. You can download it from here: https://nightly.mozilla.org/

I will move this over to a component so developers can take a look over it. If is not the correct component please feel free to change it to an appropriate one.

Thanks for the report.

Best regards, Clara.

Comment 6

[Daniel Roi]

Ok, with safe mode (latest Firefox version) I have much better values: 2589 MB VIRT, 236 MB RES, 120 MB SHR. I have also tried disabling (and reenabling) my addons one by one but VIRT stays around 26-30 GB.

Comment 7

[Andrew Overholt [:overholt]]

Maybe an extensions bug?

Comment 8

[Clara Guerrero ( Need Info Brindusa Tot Please)]

Daniel did you start to have this problem with Firefox v73 onwards?

You can capture a performance profile using the Cleopatra add-on. You can get more info on how to install and use the Cleopatra add-on (that helps you get the performance profile) by going to:

https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler
https://perf-html.io/
Please also note that this add-on works only on FF Nightly.

Best,
Clara

Comment 9

[Daniel Roi]

Clara, I can't remember when I first noticed this, but I still have this issue with Firefox v75 (on Linux). Judging from the other bug report I linked in my first comment, the problem is there at least from 2015.

I'll see if i can capture a performance profile with my current Firefox, as I don't have time for a new install and readding the extensions.

Comment 10

[Tomislav Jovanovic :zombie]

I think VIRT memory on linux is not actually "committed" memory, so this is not actually "using" anything outrageous. Andrew, can you please explain the numbers we're looking at here?

Comment 11

[Andrew McCreight [:mccr8]]

I'm seeing this in the memory report: 2,697.16 MB ── vsize

I think this includes a lot of memory that is shared between processes, but I don't know how much exactly it is on Linux. The resident numbers look pretty low.

Eric, do you know anything about super high vsize numbers on Linux and whether they're an issue or not?

Comment 12

[Eric Rahm [:erahm]]

(In reply to Andrew McCreight [:mccr8] from comment #11)

I'm seeing this in the memory report: 2,697.16 MB ── vsize

I think this includes a lot of memory that is shared between processes, but I don't know how much exactly it is on Linux. The resident numbers look pretty low.

Eric, do you know anything about super high vsize numbers on Linux and whether they're an issue or not?

2.7GB for the main process isn't outrageous. The main concern here is the 27GB in the web extension process:

27,051.25 MB ── vsize

My nightly on Ubuntu 18.04 is seeing similar behavior.

On a 64-bit build it's not a huge issue, but it probably indicates we're doing something bad in the web extension process. It could cause issues for anyone who is enforcing commit limits or trying to support suspend/resume. It could also indicate we're leaking mappings which will eventually cause ooms. A linux expert might know more about the expectations here and how to dig into this further. Maybe jld or kmag?

Comment 13

[Eric Rahm [:erahm]]

Looking locally I'm seeing 4 x 6GB anon mappings with no permissions:

erahm@shetland:/var/dev/erahm/mozilla-unified$ pmap 10493 | grep '62.*K'
00007fa3f3220000 6291328K -----   [ anon ]
00007fa57357e000 6289792K -----   [ anon ]
00007fa6f345f000 6291008K -----   [ anon ]
00007fa873510000 6290368K -----   [ anon ]

Seems likely these are wasm guard pages, it's odd they're not being reported. For example loading wasm by example I see:

6,441,402,368 B ── wasm-guard-pages

6,441,402,368 / 1024 = 6290432K which looks about right.

Comment 14

[Andrew McCreight [:mccr8]]

Nice! Maybe the extension process runs wasm at some point, then stops running it, and we end up with the pages still mapped, but whatever the memory reporter is has gone away, so nothing reports it?

Comment 15

[Andrew McCreight [:mccr8]]

It looks like the wasm guard page size is only reported if there are WASM ArrayBufferObjects.

Comment 16

[Tomislav Jovanovic :zombie]

(In reply to Andrew McCreight [:mccr8] from comment #14)

Nice! Maybe the extension process runs wasm at some point, then stops running it, and we end up with the pages still mapped, but whatever the memory reporter is has gone away, so nothing reports it?

Is that a leak then, or to be expected? And if so, is it memory reporter bug?

Comment 17

[Andrew McCreight [:mccr8]]

(In reply to Tomislav Jovanovic :zombie from comment #16)

Is that a leak then, or to be expected? And if so, is it memory reporter bug?

It seems like maybe the WASM ArrayBufferObjects are going away without removing the guard pages. I wouldn't think that would be the way it is supposed to work, but I don't know. Maybe we could find a WASM person to investigate this a bit.

Comment 18

[Tomislav Jovanovic :zombie]

This looks like a WASM issue, please forward if there's a more appropriate place.

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:lth, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 20

[Lars T Hansen [:lth]]

For something that dates to 2015, it predates most wasm work: js/src/wasm was created in November 2016, and experimental wasm implementations started appearing in browsers in 2016 according to WebAssembly.org (https://webassembly.org/roadmap/, bottom).

So more likely than not this is an asm.js problem. Of course, as we compile asm.js to wasm it's also a wasm problem, but there are some distinct code paths in the engine for asm.js still.

Comment 21

[smurfendrek123]

I'm having this "issue?" on firefox 79, 27.1g of virt memory use by WebExtensions. Fedora 32, kernel 5.7

Comment 22

[constantine.peresypkin]

(In reply to Eric Rahm [:erahm] from comment #13)

Looking locally I'm seeing 4 x 6GB anon mappings with no permissions:

erahm@shetland:/var/dev/erahm/mozilla-unified$ pmap 10493 | grep '62.*K'
00007fa3f3220000 6291328K -----   [ anon ]
00007fa57357e000 6289792K -----   [ anon ]
00007fa6f345f000 6291008K -----   [ anon ]
00007fa873510000 6290368K -----   [ anon ]

Seems likely these are wasm guard pages, it's odd they're not being reported. For example loading wasm by example I see:

6,441,402,368 B ── wasm-guard-pages

6,441,402,368 / 1024 = 6290432K which looks about right.

Yup, can confirm these anon wasm mappings. Here: Ubuntu 18.04, FF 83.0

Comment 23

[mozilla.org]

Similar, but not the same:
achim@data:$ pmap 914359| grep anon| wc
5437 32622 228355
achim@data:$

Mostly smaller lost of 1M, some 8M, ...

Debian unstable, firefox 88

Comment 24

[Ryan Hunt [:rhunt]]

Is there a specific extension that triggers this issue, or is it any extension? I can take a look at this, but I need steps to reproduce.

Comment 25

[laplandersan]

(In reply to Ryan Hunt [:rhunt] from comment #24)

Is there a specific extension that triggers this issue, or is it any extension? I can take a look at this, but I need steps to reproduce.

It's "Ublock origin" for me. Once I restart FF with 5 tabs open and enable it, extensions pid vsize jumps from 2,4G to 27G. The only second extension i have - "DownTheAll!" doesn't impact on extension process vsize when i separately enable it after restart.

Comment 26

[laplandersan]

(In reply to laplandersan from comment #25)

(In reply to Ryan Hunt [:rhunt] from comment #24)

Is there a specific extension that triggers this issue, or is it any extension? I can take a look at this, but I need steps to reproduce.

It's "Ublock origin" for me. Once I restart FF with 5 tabs open and enable it, extensions pid vsize jumps from 2,4G to 27G. The only second extension i have - "DownTheAll!" doesn't impact on extension process vsize when i separately enable it after restart.

FF 90.0.2 64bit Linux Mint

Comment 27

[Ryan Hunt [:rhunt]]

I can confirm that this is related to wasm guard pages (taking the uBlock issue as representative). Wasm guard pages are used for generating fast wasm code by reserving large areas of memory.

It looks like uBlock is allocating 4 wasm heaps, leading to about 24 GiB of virtual memory used (6GiB per-heap * 4). This virtual memory is only reserved, not committed, and so it doesn't actually consume much memory. We can consider finding ways to reduce vmem usage, but it's low priority because we're not actually committing the memory.

I'm going to attach a patch which fixes memory reporting so that this is apparent when people do memory reports, and call that it for this bug.

Comment 28

[Ryan Hunt [:rhunt]]

Attachment: Bug 1615988 - Fix memory reporting of wasm memory. r?sfink

This commit makes several tweaks to memory reporting for wasm memory.

• Add a field for shared wasm memory and track it in SAB.
  • Before this commit, shared wasm memory wouldn't report its guard pages but should.
• Track wasm guard pages in RuntimeSizes, not ClassInfo
  • We want wasm guard pages to show as a top-level memory report item, similar to vmem,
    and not under the owning array object. Displaying under the owning object bloats the
    tree of memory usage with large amounts of memory that's only reserved and not
    committed, which is confusing. Before this commit the class info reporter would
    try to make this happen, but this approach was broken and the simplest fix is to
    track this on RuntimeSizes and report the value from their.
• Only add wasm memory if the buffer is attached. Detached buffers may still be live
  but no longer own the wasm heap and shouldn't report their old size.

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1862f870e642
Fix memory reporting of wasm memory. r=sfink

Comment 30

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/1862f870e642

Comment 31

[psydvl]

Still have memory leak on 95.0 with WebExtensions process
htop reports 35.8G VIRT

NixOS 21.11, Linux 5.15.3-zen1, Wayland 1.19.0, GNOME 41.1, Firefox 95.0 (Wayland support enabled)
For WebExtensions associated process:

$ ps -F $pid
UID	PID	PPID	C	SZ	RSS	PSR	STIME	TTY	STAT	TIME	CMD
nixi	2352	2177	1	9189245	328476	3	dec13	?	Sl	23:34	/nix/store/ks5gs0rvnz896x519xfsv4r5i0xmi89f-firefox-95.0/lib/firefox/firefox -contentproc -childID 6 -isForBrowser -prefsLen 167 -prefMapSize 263431 -jsInitLen 279148 -parentBuildID 20211209203124 -appDir /nix/store/ks5gs0rvnz896x519xfsv4r5i0xmi89f-firefox-95.0/lib/firefox/browser 2177 tab

Comment 32

[fabien]

Here 27.2G VIRT in htop, Firefox 95.0 using uBlock Origin and other extensions. ArchLinux kernel 5.15.7, Wayland 1.20.0, Gnome Shell 41.2.

Comment 33

[Ryan Hunt [:rhunt]]

The uBlock Origin extension creates multiple wasm heaps. Each one takes a signficant amount of virtual memory (6GiB) that is reserved but not committed until needed. This means it takes virtual address translation space, but not physical memory.

The landed fix changes browser memory reporting to accurately report the amount of virtual memory reservations coming from wasm heaps. This can let you confirm if these large reservations are from wasm or from something else. If it is indeed from wasm heaps, then this is not a physical memory leak. In addition, the virtual memory reservations should go away when the extension is collected.