2046954 - Vulkan HW video decoding fails in the RDD process unless MOZ_DISABLE_RDD_SANDBOX=1 is set

Status: RESOLVED FIXED

(Core :: Security: Process Sandboxing, defect)

Description

[Tymur Boiko]

Steps to reproduce:

1. Enable Vulkan hw video decode (media.hardware-video-decoding-vulkan.enabled = true).
2. Run Firefox without MOZ_DISABLE_RDD_SANDBOX.
   Open a video site (e.g. YouTube).
3. Repeat with MOZ_DISABLE_RDD_SANDBOX=1.

Actual results:

With the RDD sandbox enabled, Vulkan hw video decode fails or is unreliable (fallbacks to CPU). With MOZ_DISABLE_RDD_SANDBOX=1, the same build works.

Expected results:

Vulkan hw video decode should work with the RDD sandbox enabled.

Comment 1

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Tymur Boiko]

Attachment: Bug 2046954 - Sandbox - Fix RDD Vulkan sandbox policy for display server access. r=jld,stransky

Enable brokered socket/connect syscalls in the RDD sandbox and add narrow
MAY_CONNECT rules for Wayland and X11 display sockets so EGL can connect
for EGL_MESA_image_dma_buf_export during Vulkan video decode.

On XWayland, both the Wayland socket and the X11 socket (/tmp/.X11-unix/X*)
are now allowed since EGL may connect via either path.

Comment 3

[Tymur Boiko]

Tested on X11 — works. Wayland needs double-checking before I can confirm that path. Will follow up once tested.

Comment 4

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 5

[Tymur Boiko]

Martin, can you please check if this patch works on your setup? (I assume it's on Wayland, right?)
Updated the patch

Comment 6

[Martin Stránský [:stransky] (ni? me)]

Tested on AMD RX 6600 and mesa-26.1.3 / Fedora 44 and vulkan decode works for me with and without the patch (just media.hardware-video-decoding-vulkan.enabled = true). I can test also on Intel if needed.

Comment 7

[Martin Stránský [:stransky] (ni? me)]

Also yes, testing on Wayland.

Comment 8

[Tymur Boiko]

Just a reminder, please approve as we would like to release this in build 153 (by July 21st)

Comment 9

[Martin Stránský [:stransky] (ni? me)]

(In reply to Tymur Boiko from comment #8)

Just a reminder, please approve as we would like to release this in build 153 (by July 21st)

May be worth to ping people at Mozilla chat directly - (for instance gfx channel is at https://chat.mozilla.org/#/room/#gfx:mozilla.org).
I'll try to do today (there's a time zone shift for me).

Comment 10

[Tymur Boiko]

Thank you. Also, rebased to the latest main

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/34ba72e8f8df
https://hg.mozilla.org/integration/autoland/rev/2bd4d1634b6c
Sandbox - Fix RDD Vulkan sandbox policy for display server access. r=jld,sandbox-reviewers

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/77daa967f50c
https://hg.mozilla.org/integration/autoland/rev/a168b0e43d98
Revert "Bug 2046954 - Sandbox - Fix RDD Vulkan sandbox policy for display server access. r=jld,sandbox-reviewers" for causing failures at browser_sandbox_test.js.

Comment 13

[Atila Butkovits]

Backed out for causing failures at browser_sandbox_test.js.

Backout link: https://hg.mozilla.org/integration/autoland/rev/a168b0e43d98

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=2bd4d1634b6c006144d400db75688ee09a1374e8&selectedTaskRun=WW2PKcbvSDiQdbJfFEU1Ug.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=579899698&repo=autoland&task=WW2PKcbvSDiQdbJfFEU1Ug.0&lineNumber=32974

Comment 14

[Tymur Boiko]

Reproduced locally and fixed at my side. Updated the patch

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/5ec2f029d26a
https://hg.mozilla.org/integration/autoland/rev/468d16e93a0c
Sandbox - Fix RDD Vulkan sandbox policy for display server access. r=jld,sandbox-reviewers

Comment 16

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/468d16e93a0c

Comment 17

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:tboiko, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox154 to wontfix.

For more information, please visit BugBot documentation.

Comment 18

[Tymur Boiko]

Hi Martin,
Could you please help me with the above status? (I can be wrong)

Comment 19

[Martin Stránský [:stransky] (ni? me)]

Sure, we should keep it as is (wontfix for 154.0) as it becomes release next week.