1615988 - Very high virtual memory usage in the WebExtensions process on Linux

Status: UNCONFIRMED

(Core :: Javascript: WebAssembly, defect, P3)

Description

[Daniel Roi]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

As soon I start Firefox v73 64bit on my Linux system, the htop utility reports 32.4 GB VIRTUAL memory usage and 221 RES. After a couple of seconds, usage is "lowered" to 26.4 GB VIRT and 158 MB RES.

This is with no tabs open, but I have a dozen extensions installed, each of which only uses between 0.4 to 1.4 mb according to about:performance

This issue has been reported years ago by other users at https://bugzilla.redhat.com/show_bug.cgi?id=1288888 but no solution has been given.

Also note, as reported above, that not all instances report high virtual memory usage (only those with the "-childID 2" parameter, but not other childIDs).

Comment 1

[Daniel Roi]

(Please disregard the User Agent reported above, I'm using a different pc for this bug report.)

Comment 2

[Andre Klapper]

Please enter the address
about:memory?verbose
in the address bar and attach (using the "Attach File" link above) the output here.

Comment 3

[Daniel Roi]

Attachment: memory-report.json.gz

Comment 4

[Daniel Roi]

Let me know if you need other information.

Comment 5

[Clara Guerrero]

Hi Daniel,

You mention you can reproduce this issue even with no tabs opened (but mention you have several add-ons installed), have you tried safe mode? (add-ons disabled). Here is a link that can help you do that:
https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode

Also, try this on the latest version of nightly. You can download it from here: https://nightly.mozilla.org/

I will move this over to a component so developers can take a look over it. If is not the correct component please feel free to change it to an appropriate one.

Thanks for the report.

Best regards, Clara.

Comment 6

[Daniel Roi]

Ok, with safe mode (latest Firefox version) I have much better values: 2589 MB VIRT, 236 MB RES, 120 MB SHR. I have also tried disabling (and reenabling) my addons one by one but VIRT stays around 26-30 GB.

Comment 7

[Andrew Overholt [:overholt]]

Maybe an extensions bug?

Comment 8

[Clara Guerrero]

Daniel did you start to have this problem with Firefox v73 onwards?

You can capture a performance profile using the Cleopatra add-on. You can get more info on how to install and use the Cleopatra add-on (that helps you get the performance profile) by going to:

https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler
https://perf-html.io/
Please also note that this add-on works only on FF Nightly.

Best,
Clara

Comment 9

[Daniel Roi]

Clara, I can't remember when I first noticed this, but I still have this issue with Firefox v75 (on Linux). Judging from the other bug report I linked in my first comment, the problem is there at least from 2015.

I'll see if i can capture a performance profile with my current Firefox, as I don't have time for a new install and readding the extensions.

Comment 10

[Tomislav Jovanovic :zombie]

I think VIRT memory on linux is not actually "committed" memory, so this is not actually "using" anything outrageous. Andrew, can you please explain the numbers we're looking at here?

Comment 11

[Andrew McCreight [:mccr8]]

I'm seeing this in the memory report: 2,697.16 MB ── vsize

I think this includes a lot of memory that is shared between processes, but I don't know how much exactly it is on Linux. The resident numbers look pretty low.

Eric, do you know anything about super high vsize numbers on Linux and whether they're an issue or not?

Comment 12

[Eric Rahm [:erahm]]

(In reply to Andrew McCreight [:mccr8] from comment #11)

I'm seeing this in the memory report: 2,697.16 MB ── vsize

I think this includes a lot of memory that is shared between processes, but I don't know how much exactly it is on Linux. The resident numbers look pretty low.

Eric, do you know anything about super high vsize numbers on Linux and whether they're an issue or not?

2.7GB for the main process isn't outrageous. The main concern here is the 27GB in the web extension process:

27,051.25 MB ── vsize

My nightly on Ubuntu 18.04 is seeing similar behavior.

On a 64-bit build it's not a huge issue, but it probably indicates we're doing something bad in the web extension process. It could cause issues for anyone who is enforcing commit limits or trying to support suspend/resume. It could also indicate we're leaking mappings which will eventually cause ooms. A linux expert might know more about the expectations here and how to dig into this further. Maybe jld or kmag?

Comment 13

[Eric Rahm [:erahm]]

Looking locally I'm seeing 4 x 6GB anon mappings with no permissions:

erahm@shetland:/var/dev/erahm/mozilla-unified$ pmap 10493 | grep '62.*K'
00007fa3f3220000 6291328K -----   [ anon ]
00007fa57357e000 6289792K -----   [ anon ]
00007fa6f345f000 6291008K -----   [ anon ]
00007fa873510000 6290368K -----   [ anon ]

Seems likely these are wasm guard pages, it's odd they're not being reported. For example loading wasm by example I see:

6,441,402,368 B ── wasm-guard-pages

6,441,402,368 / 1024 = 6290432K which looks about right.

Comment 14

[Andrew McCreight [:mccr8]]

Nice! Maybe the extension process runs wasm at some point, then stops running it, and we end up with the pages still mapped, but whatever the memory reporter is has gone away, so nothing reports it?

Comment 15

[Andrew McCreight [:mccr8]]

It looks like the wasm guard page size is only reported if there are WASM ArrayBufferObjects.

Comment 16

[Tomislav Jovanovic :zombie]

(In reply to Andrew McCreight [:mccr8] from comment #14)

Nice! Maybe the extension process runs wasm at some point, then stops running it, and we end up with the pages still mapped, but whatever the memory reporter is has gone away, so nothing reports it?

Is that a leak then, or to be expected? And if so, is it memory reporter bug?

Comment 17

[Andrew McCreight [:mccr8]]

(In reply to Tomislav Jovanovic :zombie from comment #16)

Is that a leak then, or to be expected? And if so, is it memory reporter bug?

It seems like maybe the WASM ArrayBufferObjects are going away without removing the guard pages. I wouldn't think that would be the way it is supposed to work, but I don't know. Maybe we could find a WASM person to investigate this a bit.

Comment 18

[Tomislav Jovanovic :zombie]

This looks like a WASM issue, please forward if there's a more appropriate place.

Comment 19

[Release mgmt bot [:sylvestre / :calixte / :marco for bugbug]]

The severity field is not set for this bug.
:lth, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 20

[Lars T Hansen [:lth]]

For something that dates to 2015, it predates most wasm work: js/src/wasm was created in November 2016, and experimental wasm implementations started appearing in browsers in 2016 according to WebAssembly.org (https://webassembly.org/roadmap/, bottom).

So more likely than not this is an asm.js problem. Of course, as we compile asm.js to wasm it's also a wasm problem, but there are some distinct code paths in the engine for asm.js still.

Comment 21

[smurfendrek123]

I'm having this "issue?" on firefox 79, 27.1g of virt memory use by WebExtensions. Fedora 32, kernel 5.7

Comment 22

[constantine.peresypkin]

(In reply to Eric Rahm [:erahm] from comment #13)

Looking locally I'm seeing 4 x 6GB anon mappings with no permissions:

erahm@shetland:/var/dev/erahm/mozilla-unified$ pmap 10493 | grep '62.*K'
00007fa3f3220000 6291328K -----   [ anon ]
00007fa57357e000 6289792K -----   [ anon ]
00007fa6f345f000 6291008K -----   [ anon ]
00007fa873510000 6290368K -----   [ anon ]

Seems likely these are wasm guard pages, it's odd they're not being reported. For example loading wasm by example I see:

6,441,402,368 B ── wasm-guard-pages

6,441,402,368 / 1024 = 6290432K which looks about right.

Yup, can confirm these anon wasm mappings. Here: Ubuntu 18.04, FF 83.0