1831548 - async RemoteTexture + accelerated Canvas on Linux: Tab crash (without crash report) occurs after a while on HTML5 Fish Bowl test page

Status: VERIFIED FIXED

(Core :: Graphics, defect)

Description

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Attachment: tab crash fishbowl.webm

Found in

• Firefox 114.0a1

Affected versions

• Firefox 114.0a1

Tested platforms

• Affected platforms: Ubuntu 22.04
• Unaffected platforms: macOS 13, Windows 11

Steps to reproduce

1. Launch Firefox
2. Access this test page
3. Select auto / 2000 fish and wait a bit

Expected result

• The tab crash doesn't occur

Actual result

• A tab crash occurs after a while

Regression range

• Will see for a regression, 113 doesn't seem to be affected

Additional notes

• The issue can be seen in the attachment
• Firefox 113 and 114 has different behaviors when selection auto. In 114 it will load for example ~300 fish and when the fps drops the number of fish stays still, while 114 will report 60fps constantly until it will load ~2000 or past then the tab will crash (seems that the fps suffered a bit when I was recording the tab crash)

Comment 1

[Darkspirit]

Reproduced on KDE Wayland, Debian Testing.
My whole desktop slowed down (mouse pointer was frozen sometimes) and then I got a tab crash without crash report.

mozregression --good 2023-01-05 --bad 2023-05-05 -a https://testdrive-archive.azurewebsites.net/Performance/FishBowl/

28:53.52 INFO: Last good revision: 54334826f02ea9aea488b9b35e44409d9851e414
28:53.52 INFO: First bad revision: 2b90b458178fa4de234b11771cb670c65c0cea03
28:53.52 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=54334826f02ea9aea488b9b35e44409d9851e414&tochange=2b90b458178fa4de234b11771cb670c65c0cea03

2b90b458178fa4de234b11771cb670c65c0cea03 sotaro — Bug 1829052 - Enable async RemoteTexture on nightly except Android r=gfx-reviewers,lsalzman

Edit: Also reproducible Asan Nightly, but it doesn't create a crash report.

$ firefox-asan/firefox -P fishbowlasan
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
Unsupported modifier, resource creation failed.
Unsupported modifier, resource creation failed.
Unsupported modifier, resource creation failed.
Unsupported modifier, resource creation failed.
Unsupported modifier, resource creation failed.
Unsupported modifier, resource creation failed.
[Parent 24751, IPC I/O Parent] WARNING: Message needs unreceived descriptors channel:61200005df40 message-type:11599877 header()->num_handles:1 num_fds:0 fds_i:0: file /builds/worker/checkouts/gecko/ipc/chromium/src/chrome/common/ipc_channel_posix.cc:489
Exiting due to channel error.

Tab crash does not occur with async RemoteTexture disabled:
mozregression --launch 2023-05-05 --pref webgl.out-of-process.async-present.force-sync:true -a https://testdrive-archive.azurewebsites.net/Performance/FishBowl/

Comment 2

[Darkspirit]

Tab crash also occurs with dmabuf webgl disabled:
mozregression --launch 2023-05-05 --pref widget.dmabuf-webgl.enabled:false -a https://testdrive-archive.azurewebsites.net/Performance/FishBowl/
MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-05-05 --pref widget.dmabuf-webgl.enabled:false -a https://testdrive-archive.azurewebsites.net/Performance/FishBowl/

Tab crash does not seem to occur if accelerated canvas is disabled:
mozregression --launch 2023-05-05 --pref gfx.canvas.accelerated:false -a https://testdrive-archive.azurewebsites.net/Performance/FishBowl/
MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-05-05 --pref gfx.canvas.accelerated:false -a https://testdrive-archive.azurewebsites.net/Performance/FishBowl/

Is this an fd exhaustion?

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

:sotaro, since you are the author of the regressor, bug 1829052, could you take a look?

For more information, please visit BugBot documentation.

Comment 4

[Sotaro Ikeda [:sotaro]]

:csasca, can you attach about:support and crash report to this bug? I could not reproduce the problem on Ubuntu 22.04.

Comment 5

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Attachment: about:support 114

Yes Sotaro, here's the about:support info. Unfortunately crash reports aren't generated for this particular tab crash (as mentioned by Darkspirit in Comment 1). If there is any other way to capture a tab crash error please let me know. Thanks!

Comment 6

[Darkspirit]

(In reply to Sotaro Ikeda [:sotaro] from comment #4)

:csasca, can you attach about:support and crash report to this bug? I could not reproduce the problem on Ubuntu 22.04.

You need Hardware WebRender, Dmabuf (on proprietary Nvidia try widget.dmabuf.force-enabled=true), accelerated Canvas (try gfx.canvas.accelerated.force-enabled=true), async RemoteTexture.

Comment 7

[Darkspirit]

Correction: You need Hardware WebRender, accelerated Canvas, async RemoteTexture.

Comment 8

[Sotaro Ikeda [:sotaro]]

From the following, the problem might happen by out of file descriptor.

• https://searchfox.org/mozilla-central/rev/ee6ab6eb2d222e0004604de62206baa67cac33af/ipc/chromium/src/chrome/common/ipc_channel_posix.cc#488

[Parent 24751, IPC I/O Parent] WARNING: Message needs unreceived descriptors channel:61200005df40 message-type:11599877 header()->num_handles:1 num_fds:0 fds_i:0: file /builds/worker/checkouts/gecko/ipc/chromium/src/chrome/common/ipc_channel_posix.cc:489
Exiting due to channel error.

Comment 9

[Gian-Carlo Pascutto [:gcp]]

I'm going to yank this to S2 because this is a crash that does not generate a crash report. if it's happening in the field we're blind to it.

Comment 11

[Martin Stránský [:stransky] (ni? me)]

Can you run with MOZ_LOG="Dmabuf:5" to see how dmabuf is utilized and if we fail to create/release one?

Comment 12

[Sotaro Ikeda [:sotaro]]

Attachment: patch - Add log

Comment 13

[Sotaro Ikeda [:sotaro]]

When fish count was 10, log out of RecvDispatchCommands() was like the following.

WebGLParent::RecvDispatchCommands()_E shmemBytes 6816
RemoteTextureMap::PushTexture() aTextureId 980 aOwnerId 1 **************************************
WebGLParent::RecvDispatchCommands()_X

When fish count was 2000, log out of RecvDispatchCommands() was like the following. command had a lot of DrawArraysInstanced calls.

WebGLParent::RecvDispatchCommands()_E shmemBytes 100000
WebGLParent::RecvDispatchCommands()_X
WebGLParent::RecvDispatchCommands()_E shmemBytes 100000
WebGLParent::RecvDispatchCommands()_X
WebGLParent::RecvDispatchCommands()_E shmemBytes 99984
WebGLParent::RecvDispatchCommands()_X
WebGLParent::RecvDispatchCommands()_E shmemBytes 99968
WebGLParent::RecvDispatchCommands()_X
WebGLParent::RecvDispatchCommands()_E shmemBytes 100000
WebGLParent::RecvDispatchCommands()_X
WebGLParent::RecvDispatchCommands()_E shmemBytes 79984
RemoteTextureMap::PushTexture() aTextureId 1202 aOwnerId 1 **************************************
WebGLParent::RecvDispatchCommands()_X

Comment 14

[Sotaro Ikeda [:sotaro]]

Attachment: patch - Add log

Added file descriptor limit and current file descriptor count.

Comment 15

[Sotaro Ikeda [:sotaro]]

(In reply to Catalin Sasca, Desktop QA [:csasca] from comment #0)

Created attachment 9331806 [details]
tab crash fishbowl.webm

Found in

• Firefox 114.0a1

Affected versions

• Firefox 114.0a1

Tested platforms

• Affected platforms: Ubuntu 22.04

Hmm, I could not reproduce the problem on my Ubuntu 22.04 PC :(

Comment 16

[Sotaro Ikeda [:sotaro]]

Hi Catalin, can you get a logout to console with the build of Attachment 9338214 [details] [diff] with the STR?

https://treeherder.mozilla.org/jobs?repo=try&revision=d665feb159b1a1bd813f56a53f8f357953c30183

Comment 17

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Attachment: tab crash gfx

Sure thing, here's the log (bottom of the text where the tab crash happened) with the provided build.
One other thing I saw is that by selecting the dedicated Nvidia gtx 960M gpu from the Nvidia X server, I wasn't able to reproduce the tab crash either and the performance was very much stable even above 2000 fish. As soon as I selected the integrated Intel gpu, the performance went down and the tab crash is present. Maybe that's why you couldn't reproduce the issue either as you sure have a dedicated gpu in your PC (or much powerful integrated than my laptop's one).
Please let me know if I can help with anything else.

Comment 18

[Sotaro Ikeda [:sotaro]]

I could reproduce the problem on Ubuntu 20.04 with VMWare with Attachment 9338214 [details] [diff]. When the crash happened, fd count was increased from 133 to 4096.

Comment 19

[Sotaro Ikeda [:sotaro]]

It seemed that pending WebGL IPC messages seemed to increase fd count.

Comment 20

[Sotaro Ikeda [:sotaro]]

Attachment: Bug 1831548 - Force sync IPC if there are many flushed cmds between GetFrontBuffer()

Comment 21

[Sotaro Ikeda [:sotaro]]

D181033 addressed the problem for me.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/698d86db485b
Force sync IPC if there are many flushed cmds between GetFrontBuffer() r=gfx-reviewers,lsalzman

Comment 23

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/698d86db485b

Comment 24

[Sotaro Ikeda [:sotaro]]

:csasca, can you check if the problem is addressed with latest nightly?

Comment 25

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Sure thing. Checked on Firefox 116.0a1 (2023-06-15) on Ubuntu 22.04 and the issue is no longer reproducibe, and now the fps needle work as expected as well and reports the correct fps.

Comment 26

[Sotaro Ikeda [:sotaro]]

Great! Thank you.

Comment 27

[Sotaro Ikeda [:sotaro]]

I could still reproduce the crash with multiple windows. Then created Bug 1839314.

Comment 28

[Sotaro Ikeda [:sotaro]]

Attachment: WIP: Bug 1831548 - Force sync IPC if there are many flushed cmds without calling ClientWebGLContext::GetFrontBuffer()

Comment 29

[Ina Popescu, Desktop QA]

I've reproduced this issue using Nightly 116.0a1(2023-06-15) following the STR from Comment 0 on Ubuntu 22.04.
Verified as fixed on the latest Nightly 117.0a1 and Firefox 116.0 versions under same configuration where the issue no longer persists.