1337585 - Page reload loop brings down the browser when console is open

Status: RESOLVED INACTIVE

(DevTools :: Console, defect, P2)

Description

[Jukka Jylänki]

STR:

1. Enable pref javascript.options.wasm;true
2. Visit https://s3.amazonaws.com/mozilla-games/tmp/2017-02-07-PlatformerGame-wasm/PlatformerGame-HTML5-Shipping-crash.html
3. After the page loads, open up the web browser console (ctrl-shift-k)
4. Wait and observe

It seems that opening the console window is crucial for the crash to manifest. Without the console window open, the browser memory usage stays solid around 300MB+200MB of RAM for the two Firefox processes. (main & content) After opening the console window, each page load starts to grow the process memory usage by about 50MB, up until getting a crash at around 1GB of memory usage. After around 20 page restarts or so, the crash occurs. Tested on a 64-bit Nightly on Windows 10 with 32GB of RAM.

When starting the test, my C:\ drive had 11.2GB of free space. When the crash occurs, C:\ had only 250MB of space left, and sometimes the Windows "free some space up" dialog shows up at bottom right of the screen. Not clicking that dialog, but waiting for a while F5 refreshing the Windows explorer window on My Computer view after the browser has died, available disk space pops back up to 11.2GB of free space a bit afterwards.

Refreshing the My Computer view on C:\ drive, it looks like each page reload reduces available disk space by about 1GB.

When it crashes, there seem to be three distinct types of crash behavior.

1. The most common is that the whole browser goes down without crash handler coming up, i.e. silent to desktop. Sometimes this is paired with the Windows desktop screen going black for a second or so.

2. On one test run only the tab crashed, here is a report:
 - https://crash-stats.mozilla.com/report/index/cb879e22-8c14-416b-a5e7-2ef952170207

3. On other test runs the whole browser crashed, but the crash handler did still show up, and generated these reports:
 - https://crash-stats.mozilla.com/report/index/329da182-7b79-441b-b336-e8a1e2170207
 - https://crash-stats.mozilla.com/report/index/9aa6f1d1-7c6b-4525-ae71-6ebbf2170207

Filing this under Networking: Cache due to the two reports above suggesting so.

Comment 1

[Jukka Jylänki]

See also bug 1337561, which is about OOMs occurring on the same page, although not sure if these have any relation.

Comment 2

[Jukka Jylänki]

Here is another crash signature from the page:
 - https://crash-stats.mozilla.com/report/index/9e20b569-f23f-49c7-9cc8-6c1162170208

Comment 3

[Honza Bambas (:mayhemer)]

The first crash doesn't have symbols, so it's unfortunately useless.  

The crash reports you provide are not isolated to http cache.  All are various crashes inside jemalloc.

Comment 4

[Luke Wagner [:luke]]

So using about:memory to snapshot at 3 progressive occasions in the content process, I see js-main-runtime-compartments/user grow from 7 -> 15 -> 25, so I think each reload is simply leaking the entire previous global and its contents.  Since each load makes a big memory commit, I think the OOM happens when some virtual address space quota gets exhausted and the disk usage is page file allocation backing committed memory.

Andrew: do you have any quick ways to diagnose what's keeping these old compartments alive?  I can confirm I only see the growth when the console is open.

Comment 5

[Andrew McCreight [:mccr8]]

This sounds like a dupe of bug 1084605. Bug 1084626 would fix it, but it also broke a ton of devtools stuff.

Comment 6

[Luke Wagner [:luke]]

Yes, that sounds like what we're seeing, thanks!

Comment 7

[Andrew McCreight [:mccr8]]

To answer your question more directly, basically the console seems to just leak every page when it is open.

Comment 9

[Jukka Jylänki]

Hey, great triaging. If it's all the same, I'd rather keep this report open until the root bug is fixed, since this one has a separate explicitly reproducible STR from bug 1084605, so should be easy to verify and close once the fix is done. (with the tiny >0% chance that fixing bug 1084605 won't directly fix this one)

Comment 10

[Jukka Jylänki]

Rechecked this today on Firefox Nightly 55.0a1 (2017-03-20) (64-bit), still reproducing. Looking at Task Manager, the memory usage keeps steadily growing until it's at about 3GB and the tab crashes.

Comment 11

[Jukka Jylänki]

Crash trace of one observed crash: https://crash-stats.mozilla.com/report/index/a099c8ac-61c2-4cb3-a9bb-778482170320

Comment 12

[Brian Grinstead [:bgrins]]

It's possible that https://hg.mozilla.org/mozilla-central/rev/9360c06128f8 will fix this, as it turns out that object actors were not being released properly when being removed from the console.

Comment 13

[Brian Grinstead [:bgrins]]

(In reply to Brian Grinstead [:bgrins] from comment #12)
> It's possible that https://hg.mozilla.org/mozilla-central/rev/9360c06128f8
> will fix this, as it turns out that object actors were not being released
> properly when being removed from the console.

If you have a chance can you please attempt to reproduce again since this change has landed?

Comment 14

[Jukka Jylänki]

Tried out on Windows 10 on Firefox Nightly 55.0a1 (2017-05-31) (64-bit).

The issue does still exist, I see that each page reload consumes both RAM and disk space, up until browser oom crashes to desktop.

The following crash report was generated: https://crash-stats.mozilla.com/report/index/1f44df27-ed0e-4a02-a42d-407440170601

Comment 16

[Nicolas Chevobbe [:nchevobbe]]

Hello Jukka,

There have been changes to the console code lately that might fix this.
Could you check if this issue still happens on Nightly ?

Comment 17

[Jukka Jylänki]

Clearing needinfo on old bugs. Looks like Mozilla is no longer hosting the PlatformerGame demo, so I am not able to test if the issue persists.