1870488 - Web Worker Canvas Memory Leak

Status: RESOLVED FIXED

(Core :: Graphics: Canvas2D, defect)

Description

[Thomas Brierley]

Attachment: firefox-worker-canvas-raf-leak.html

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0

Steps to reproduce:

Set up a Web Worker, with an OffscreenCanvas, and RequestAnimationFrame loop, that uses the 2D context API to affect the canvas in any way, on every frame.

The issue has some kind of dependence on browser resource usage. For the simplified test case it was sufficient to use a very large canvas of 8k x 8k pixels for my machine. Beneath a certain size threshold the leak does not occur at all.

It's possible to reproduce the issue under realistic conditions through multiple small off screen canvases on a single page - which is how I discovered the issue. Even when all but one workers are 100% idle it's possible to trigger the leak through only painting to a single small canvas at a time. i.e the dependence on resource usage seems to be shared across off screen canvases on the same page regardless of active use.

Actual results:

The browser consumes all available RAM and crashes the tab. In my case 16 GiB is consumed in around 15 seconds.

Expected results:

No memory leaks.

Comment 1

[Thomas Brierley]

Online reduced test case for convenience:

https://jsfiddle.net/we0sqjtd/

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Graphics: Canvas2D' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Mayank Bansal]

Can you type "about:support" in your browser and copy-paste its contents to this bug?

Comment 4

[Thomas Brierley]

Attachment: Output of about:support

Comment 5

[Thomas Brierley]

I've also tried this with a fresh Firefox config, i.e defaults and no extensions, and found the behaviour to be slightly different: While the tab no leak occurs, but as soon as I close the tab, all RAM is quickly consumed and then the whole browser crashes instead of a tab.

When I restore my original Firefox config the behaviour reverts to what I initially described. But both configurations result in a leak and crash.

Comment 6

[Brad Werth [:bradwerth]]

I can't reproduce on macOS with either the attachment or the jsfiddle. We'll discuss in triage.

Comment 7

[Bob Hood [:bhood]]

Lee, can you repro?

Comment 8

[Andrew Osmond [:aosmond] (he/him)]

This seems more me than Lee. Did you reproduce this on release, or a recent nightly? We've been making a lot of changes in the past week.

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

Also, if on release, does disabling accelerated canvas help? Flip gfx.canvas.accelerated to false, restart and try to reproduce as normal.

Comment 10

[Mayank Bansal]

I found a version of leak on Windows with GPU-canvas, and filed bug 1871207.
Filed Bug 1871208 for d2d-canvas leak

Comment 11

[Thomas Brierley]

Andrew, I've tried setting gfx.canvas.accelerated to false and restarting, the leak still occurs for me.

Note, I do not know whether this is a regression or has been here all along, since I discovered it while developing something new.

Comment 12

[Thomas Brierley]

Andrew, this is on release, via Flatpack flathub:

Name     Application ID       Version  Branch  Arch    Origin   Ref                                Active commit
Firefox  org.mozilla.firefox  120.0.1  stable  x86_64  flathub  org.mozilla.firefox/x86_64/stable  f75dc4d15a98

Comment 13

[Thomas Brierley]

Noticed there was an update, can also reproduce on v121:

Name     Application ID       Version  Branch  Arch    Origin   Ref                                Active commit
Firefox  org.mozilla.firefox  121.0    stable  x86_64  flathub  org.mozilla.firefox/x86_64/stable  b4ed37eec155

Comment 14

[Thomas Brierley]

I've noticed the behaviour is a little slower when nothing else open in the browser.

Using the test case HTML I have to wait 20 to 30 seconds during which my available RAM hovers around 13-14 GiB, after that initial period available RAM suddenly plummets and the tab crashes just before hitting 0.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:lsalzman, could you have a look please?

For more information, please visit BugBot documentation.

Comment 16

[Andrew Osmond [:aosmond] (he/him)]

Sorry for the delay due to the holidays, could you provide an about:memory report? That might shed some light at least on where the allocations are. Thanks!

Comment 17

[Mayank Bansal]

(In reply to Andrew Osmond [:aosmond] (he/him) from comment #16)

Sorry for the delay due to the holidays, could you provide an about:memory report? That might shed some light at least on where the allocations are. Thanks!

If it helps, bug 1871208 has an about:memory report.

Comment 18

[Thomas Brierley]

Attachment: memory-report.json.gz

I captured this after the attached test case consumed around 12GB, with only a couple GB of RAM left (it crashes when out of RAM).

Comment 19

[Andrew Osmond [:aosmond] (he/him)]

Looks like it is mostly consumed by shmems in the content process:
12,288.19 MB ── shmem-allocated
12,288.19 MB ── shmem-mapped

Comment 20

[Thomas Brierley]

I've narrowed the difference in behaviour between defaults and my config to layers.acceleration.disabled.

When setting this to true, the test case leaks while open in foreground as per my initial description.

When setting this to false (the default) , no leak occurs while in foreground, however the tab eventually crashes on it's own, unless I close the tab prematurely after ~10 seconds then all RAM is consumed and the whole browser crashes.

I've also played with the interval by using a setInterval instead of a RAF loop and it seems to be very sensitive to the timing, if I make the interval greater than or less than the refresh rate interval (~16ms for my machine) then it doesn't occur so easily. Except for some magic numbers like 4ms where it occurs as easily as 16ms.

Comment 21

[Mayank Bansal]

Profile with all threads, IPC and allocation: https://share.firefox.dev/3HhWXhW

Comment 22

[Andrew Osmond [:aosmond] (he/him)]

I can reproduce this with SW-WR.

Comment 23

[Andrew Osmond [:aosmond] (he/him)]

If we use the buffer provider, the problem goes away. I can land that patch to fix this.

Comment 24

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1870488 - Part 3. Make OffscreenCanvas use PersistentBufferProvider on the display pipeline.

Comment 25

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1870488 - Part 1. Implement ImageBridgeChild::GetTileLockAllocator.

This patch adds support for allocationg shmem sections for
ImageBridgeChild. The recording infrastructure depends on it.

Comment 26

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1870488 - Part 2. Ensure TextureClient's mReadLock is only created on the IPDL actor thread.

This allows us to create a TextureClient on a different thread than the
actor without special effort on the part of the allocator. Similarly, it
also allows us to destroy a TextureClient on a different thread if it
has a readlock bound to it.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a8db51f72457
Part 1. Implement ImageBridgeChild::GetTileLockAllocator. r=gfx-reviewers,lsalzman
https://hg.mozilla.org/integration/autoland/rev/a3e17caaa159
Part 2. Ensure TextureClient's mReadLock is only created on the IPDL actor thread. r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/cf3ce7d3c82d
Part 3. Make OffscreenCanvas use PersistentBufferProvider on the display pipeline. r=lsalzman

Comment 28

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/a8db51f72457
https://hg.mozilla.org/mozilla-central/rev/a3e17caaa159
https://hg.mozilla.org/mozilla-central/rev/cf3ce7d3c82d

Comment 31

[Andrej (:aglavic)]

(In reply to Pulsebot from comment #27)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a8db51f72457
Part 1. Implement ImageBridgeChild::GetTileLockAllocator.
r=gfx-reviewers,lsalzman
https://hg.mozilla.org/integration/autoland/rev/a3e17caaa159
Part 2. Ensure TextureClient's mReadLock is only created on the IPDL actor
thread. r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/cf3ce7d3c82d
Part 3. Make OffscreenCanvas use PersistentBufferProvider on the display
pipeline. r=lsalzman

We're recorded a large improvement in CI from your patches!

== Change summary for alert #41064 (as of Thu, 18 Jan 2024 23:29:28 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
19%	offscreencanvas_webcodecs_worker_2d_vp9 offscreencanvas_webcodecs_worker_2d_vp9 Mean time across 100 frames:	windows10-64-ref-hw-2017-qr	e10s fission stylo webgl-ipc webrender	25.73 -> 20.81
19%	offscreencanvas_webcodecs_worker_2d_av1 offscreencanvas_webcodecs_worker_2d_av1 Mean time across 100 frames:	windows10-64-ref-hw-2017-qr	e10s fission stylo webgl-ipc webrender	26.53 -> 21.53
19%	offscreencanvas_webcodecs_main_2d_vp9 offscreencanvas_webcodecs_main_2d_vp9 Mean time across 100 frames:	windows10-64-ref-hw-2017-qr	e10s fission stylo webgl-ipc webrender	25.81 -> 20.98
18%	offscreencanvas_webcodecs_main_2d_av1 offscreencanvas_webcodecs_main_2d_av1 Mean time across 100 frames:	windows10-64-ref-hw-2017-qr	e10s fission stylo webgl-ipc webrender	26.50 -> 21.64
18%	offscreencanvas_webcodecs_main_2d_vp9 offscreencanvas_webcodecs_main_2d_vp9 Mean time across 100 frames:	windows10-64-shippable-qr	e10s fission stylo webgl-ipc webrender	13.65 -> 11.16
...	...	...	...	...
2%	offscreencanvas_webcodecs_main_2d_av1 offscreencanvas_webcodecs_main_2d_av1 Mean time across 100 frames:	windows10-64-shippable-qr	e10s fission stylo webrender-sw	13.76 -> 13.48

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=41064