1287627 - [e10s] Mysterious continual rise in physical memory usage with twitch.tv open in a background tab and the browser window inactive

Status: RESOLVED FIXED

(Core :: Graphics: Layers, defect)

Description

[Emanuel Hoogeveen [:ehoogeveen]]

I've been getting some very odd OOMs recently, and finally figured out what was causing them. With e10s enabled, twitch.tv open in a background tab, and the browser *not* as the active window, system-wide memory usage starts growing out of control. This gobbles up all 32 gigs of RAM on my system until I get a low memory warning.

Steps to reproduce:
1) Starting from a fresh profile, ensure that e10s is enabled (which may require disabling accessibility, restarting the browser, resetting accessibility.loadedInLastSession and restarting the browser again), and ensure that the Flash plugin is installed and enabled.
2) Open a new tab and navigate to www.twitch.tv, letting the featured stream load (feel free to mute it).
3) Change back to the other tab.
4) Open up the Task Manager so that the Firefox window is no longer the active window, and ensure that the cursor is not hovering over the Firefox window (in Windows 10, this seems to send mouse events to the window).
5) Watch the percentage shown above the Memory column, and wait for 30-60 seconds.

Expected Result:
Memory usage should be approximately stable.

Actual Result:
The percentage of physical memory in use starts to increase, even though no particular process is shown as using more memory.

Making Firefox the active window or hovering the mouse over Firefox's window makes the memory usage drop back down. Keep waiting, however, and it'll use up all the memory on the system - even if the Firefox processes are 32-bit!

I ran a bisection and it pointed to bug 1252835 as the culprit:

Last Good build:
http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win64/1465988303/

First Bad build:
http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win64/1465990163/

Regression range:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=e1cac03485d9949c73d5ed6f703dac189422b913&tochange=5748a69df90b4f22108b67c01a1e331ef9018c29

Satoro, this problem is very strange, and considering the odd behavior in the Task Manager, it might be a bug in Windows 10 (especially considering 32-bit Firefox can end up using more physical memory than it even has address space for). But it *is* a regression, and although it looks like it requires a very specific setup, this is how I normally run Firefox on my system. I've seen this problem several times just using Firefox as I normally do.

A few other details:
I'm using the Direct3D 11 backend with Direct2D 1.1; my graphics card is a GeForce GTX 970 and I have the latest drivers. This bug requires e10s to reproduce - without e10s, the physical memory usage does not increase.
It also requires Flash - I tried doing the same thing with a YouTube video (HTML5), and the physical memory usage didn't increase either. Twitch are currently in the process of switching over entirely to HTML5, but they've only just started testing their new player for a select set of users, and Twitch may not be the only Flash app affected.

Comment 1

[Emanuel Hoogeveen [:ehoogeveen]]

/s/Satoro/Sotaro/, my bad.

Comment 2

[Emanuel Hoogeveen [:ehoogeveen]]

It turns out that measuring using about:memory from a separate window *does* work (thanks mccr8!), and it shows the following (with system memory usage at 50%):

14,236.81 MB ── d3d11-shared-textures

That's the only number out of the ordinary - everything else looks fine, both in about:memory and in the Details view of the Task Manager. But it's nice to know we do have some sort of metric for this!

Comment 3

[Sotaro Ikeda [:sotaro]]

:ehoogeveen, thanks for reporting the bug! I take a look.

Comment 4

[Sotaro Ikeda [:sotaro]]

I still failed to reproduce the problem, but the problem seemed to happen in the STR from the source code, since returning TextureHosts depends on Compositor's composition. If the Compositor does not do compositing, TextureHosts is not returned to client side and client side allocate more TextureClients.

Comment 5

[Sotaro Ikeda [:sotaro]]

Attachment: patch - Handle flush TextureHost without composition

Comment 6

[Sotaro Ikeda [:sotaro]]

There is alreayd Compositor::mCompositionTime, but it is set by CompositorBridgeParent and it could be different than actual composition time, therefore attachment 8772282 [details] [diff] [review] added mLastCompositionEndTime.

Comment 7

[Sotaro Ikeda [:sotaro]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f862922905c9&selectedJob=24100664

Comment 8

[Emanuel Hoogeveen [:ehoogeveen]]

Thanks! Sorry my STR didn't work for you, I'm not sure what's missing from them (Terrence wasn't able to reproduce either).

However, I can't reproduce the problem with the try build you linked, so it looks like you were right!

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3996c602754f
Handle flush TextureHost without composition r=nical

Comment 10

[Sotaro Ikeda [:sotaro]]

(In reply to Emanuel Hoogeveen [:ehoogeveen] from comment #8)
> Thanks! Sorry my STR didn't work for you, I'm not sure what's missing from
> them (Terrence wasn't able to reproduce either).
> 
> However, I can't reproduce the problem with the try build you linked, so it
> looks like you were right!

Good! Thanks for the confirmation!!

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/3996c602754f