1469278 - Memory reporting: Firefox using 45GB VSS on protonmail site

Status: RESOLVED DUPLICATE of bug 1615988

(Core :: JavaScript: WebAssembly, defect, P3)

Description

[benjbrandall]

Attachment: Screenshot from 2018-06-18 12-50-14.png

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Build ID: 20180607190419

Steps to reproduce:

Using Firefox as normal, ~6-7 tabs open.


Actual results:

FIrefox is listed as using 47GB of virtual memory. The laptop fan started going crazy, and I needed to shut down Firefox. (See attached htop screenshot)


Expected results:

Firefox should have used the same low amount of memory it usually does.

Comment 1

[Andre Klapper]

Please enter the address
   about:memory?verbose
in the address bar and attach (using the "Attach File" link above) the output here.

Comment 2

[benjbrandall]

Attachment: Memory output file

Comment 3

[Adrian Florinescu [:aflorinescu]]

I think this line shows the amout of virtual memory that it's used by the main process:
{
   "process": "Main Process (pid 9226)",
   "path": "resident",
   "kind": 2,
   "units": 0,
   "amount": 450338816,
   "description": "Memory mapped by the process that is present in physical memory, also known as the resident set size (RSS).  This is the best single figure to use when considering the memory resources used by the process, but it depends both on other processes being run and details of the OS kernel and so is best used for comparing the memory usage of a single process at different points in time."
  }

However, I can't make any other heads or tails from that memory report. It doesn't look to me that we could pinpoint what causes this from the memory report alone. Mike, any thoughts?

Also, [e-mail], is this something you can reproduce? A performance profile would probably be a bit more clear (https://perf-html.io/).

Comment 4

[Mike Conley (:mconley) (:⚙️)]

The memory report doesn't actually look that bad. Was this captured at the time when Firefox was using the 47GB of memory?

Comment 5

[benjbrandall]

Attachment: 2-memory-report.json.gz

I just captured another one at the same time as Firefox is reportedly using 44GB. Attached. Can't see the reference to any GBs in the report, though.

Comment 6

[benjbrandall]

Attachment: htop 44G firefox.png

Comment 7

[Mike Conley (:mconley) (:⚙️)]

Hm. I'm still only seeing memory usage on the order of hundreds of MB in the memory report, and not in terms of GB.

Hey erahm, is there something in this report that I'm not seeing?

Comment 8

[Eric Rahm [:erahm]]

(In reply to Mike Conley (:mconley) (:⚙️) (Catching up on needinfos / reviews) from comment #7)
> Hm. I'm still only seeing memory usage on the order of hundreds of MB in the
> memory report, and not in terms of GB.
> 
> Hey erahm, is there something in this report that I'm not seeing?

Resident looks fine, but the virtual memory usage is super high. In practice this often isn't a problem on 64-bit, but 45GB is pretty extreme:

> 45,083.70 MB ── vsize
> 12,287.00 MB ── wasm-guard-pages [2]

Note the WASM entries, I'd guess that's related. Maybe protonmail is doing something odd with WASM?

Comment 9

[Eric Rahm [:erahm]]

Luke, can you take a look at this?

Comment 10

[Luke Wagner [:luke]]

The wasm-guard-pages are VirtualAlloc(MEM_RESERVE)'d memory so this is just a reservation, not actual committed/accessible memory that consumes RAM.  Each wasm memory gets ~6gb of guard pages so the wasm-guard-pages of 12gb for 2 memories looks normal.  (Ideally the OS wouldn't report reserved-but-not-committed memory in task manager as normal virtual address space usage, but it does, which is why we have this special wasm-guard-pages entry in about:memory to explain .)

Comment 11

[Eric Rahm [:erahm]]

(In reply to Luke Wagner [:luke] from comment #10)
> The wasm-guard-pages are VirtualAlloc(MEM_RESERVE)'d memory so this is just
> a reservation, not actual committed/accessible memory that consumes RAM. 
> Each wasm memory gets ~6gb of guard pages so the wasm-guard-pages of 12gb
> for 2 memories looks normal.  (Ideally the OS wouldn't report
> reserved-but-not-committed memory in task manager as normal virtual address
> space usage, but it does, which is why we have this special wasm-guard-pages
> entry in about:memory to explain .)

Okay but what about the remaining 33GB of virtual memory?

Comment 12

[Luke Wagner [:luke]]

I don't know; is it all unclassified?

Comment 13

[Eric Rahm [:erahm]]

(In reply to Luke Wagner [:luke] from comment #12)
> I don't know; is it all unclassified?

Correct, it's unclassified. The only thing I've seen use virtual memory like this is either WASM or graphics leaks. With the existence of the WASM guard pages I'm guessing it's WASM related.

Comment 14

[Eric Rahm [:erahm]]

benjbrandall, one other thing that might help us out is to see if you can reproduce the issue in safe mode [1]. This will help us identify if extensions or hardware acceleration is involved.

[1] https://support.mozilla.org/kb/troubleshoot-firefox-issues-using-safe-mode

Comment 15

[Luke Wagner [:luke]]

Ah, I think the problem here is just that about:memory doesn't include the wasm-guard-pages stat from workers in the total process number.  Explicitly logging memory creation and visiting https://mail.protonmail.com/login (which fortunately loads everything immediately even without login), I can see 5 wasm memories being created: 2 on the main thread and 3 in a worker and only the two main-thread ones are reported.

Comment 16

[benjbrandall]

Attachment: safe-mode-3memory-report.json.gz

Here's the output while in safe mode, Firefox memory usage is normal(?) in htop (~2.4GB).

Protonmail is loaded fully, took many reloads and crashes though. Console says: TypeError: asm.js type error: Disabled by 'asmjs' runtime option (openpgp.min.js
)

Comment 17

[Eric Rahm [:erahm]]

(In reply to Luke Wagner [:luke] (PTO until July 22) from comment #15)
> Ah, I think the problem here is just that about:memory doesn't include the
> wasm-guard-pages stat from workers in the total process number.  Explicitly
> logging memory creation and visiting https://mail.protonmail.com/login
> (which fortunately loads everything immediately even without login), I can
> see 5 wasm memories being created: 2 on the main thread and 3 in a worker
> and only the two main-thread ones are reported.

Can we fix the memory reporting for the worker threads?

Comment 18

[Luke Wagner [:luke]]

Grepping takes me to the accumulation of wasmGuardPages in XPCJSRuntime.cpp.  I don't know how workers are reported differently or what is missing to properly merge workers into the main thread's JSRuntime wasmGuardPages.  Do you?

Comment 19

[Eric Rahm [:erahm]]

:baku, it looks like we're not reporting wasm guard pages for web workers, do you know who could look into that?

Comment 20

[Andrea Marchesini [:baku]]

We do report memory stats in workers calling:
https://searchfox.org/mozilla-central/rev/7944190ad1668a94223b950a19f1fffe8662d6b8/dom/workers/WorkerPrivate.cpp#1194

Wondering why wasm in workers are not correctly reported. Maybe njn is more familiar with this memory reporting.

Comment 21

[Nicholas Nethercote [inactive]]

The wasm guard pages got some special handling in bug 1307768. I bet the problem is in there somewhere.

Comment 22

[Nicholas Nethercote [inactive]]

The wasm guard pages are recorded in ClassInfo, which is totally the wrong spot for them. Luke, what is the right granularity for them -- is there at most one wasm guard page per JSRuntime?

Comment 23

[Luke Wagner [:luke]]

There is 1 per WebAssembly.Memory object.

Comment 24

[Ryan Hunt [:rhunt] (on leave until early May)]

It looks like protonmail uses wasm somewhere. Each WebAssembly.Memory object created reserves (not commits) a large amount (~6GiB) of virtual memory. As of bug 1615988, we now report this accurately in about:memory. This is not a memory leak, and once wasm code is freed this virtual memory will be unreserved. Reserved, but not committed, virtual memory generally doesn't take significant resources from the OS.

Marking this as a duplicate of bug 1615988.