1732580 - [snap] WebGL broken with EGL (snap default on Wayland), but fine with GLX.

Status: RESOLVED FIXED

(Core :: Security: Process Sandboxing, defect, P2)

Description

[Darkspirit]

Attachment: aboutsupport.txt

Gnome Wayland, Debian Testing, Intel

(Darkspirit from bug 1726510 comment 36)

$ sudo snap remove firefox
$ sudo snap install firefox
$ snap run firefox

WebGL and WebRender work according to about:support, but https://webglsamples.org/aquarium/aquarium.html does not work.

It does not appear your computer supports WebGL.
Click here for more information.
Status: WebGL creation failed: * tryNativeGL (FEATURE_FAILURE_NO_DISPLAY) * Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)

No word on about:support about this FEATURE_FAILURE_NO_DISPLAY thing.

$ sudo cat /var/log/syslog | grep denied

Sep 26 23:36:58 darkspirit-laptop kernel: [41105.417113] audit: type=1400 audit(1632692218.808:14756): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/sys/dev/i915/perf_stream_paranoid" pid=49327 comm="firefox" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Sep 26 23:36:59 darkspirit-laptop kernel: [41105.850276] audit: type=1400 audit(1632692219.240:14757): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/sys/dev/i915/perf_stream_paranoid" pid=49258 comm="Renderer" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

---

Manually installed firefox - without dangerous, but with acking - works the same as official firefox:
$ sudo snap remove firefox
$ sudo snap ack firefox_595.assert
$ sudo snap install firefox_595.snap
$ snap run firefox

https://webglsamples.org/aquarium/aquarium.html

It does not appear your computer supports WebGL.
Click here for more information.
Status: WebGL creation failed: * tryNativeGL (FEATURE_FAILURE_NO_DISPLAY) * Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)

/var/log/syslog:
Same two errors as above and this one is new:

Sep 26 23:21:31 darkspirit-laptop kernel: [40177.949121] audit: type=1400 audit(1632691291.299:14602): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/46373/environ" pid=46215 comm=427265616B70616420536572766572 requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

Comment 1

[Darkspirit]

$ snap info firefox

channels:
latest/stable: 92.0-3 2021-09-09 (595) 159MB -
latest/candidate: 92.0.1-1 2021-09-23 (625) 159MB -
latest/beta: 93.0b9-1 2021-09-24 (628) 155MB -
latest/edge: ↑
esr/stable: 78.14.0esr-1 2021-09-07 (591) 148MB -
esr/candidate: 91.1.0esr-1 2021-09-10 (603) 158MB -
esr/beta: ↑
esr/edge: ↑

$ sudo snap remove firefox --purge; sudo snap install firefox --channel=esr/stable; snap run firefox https://webglsamples.org/aquarium/aquarium.html

firefox (esr/stable) 78.14.0esr-1 from Mozilla✓ installed

works
Edit: because it uses the official Mozilla config (GLX/Xwayland)

$ sudo snap remove firefox; sudo snap install firefox --channel=esr/candidate; snap run firefox https://webglsamples.org/aquarium/aquarium.html

firefox (esr/candidate) 91.1.0esr-1 from Mozilla✓ installed

broken

$ sudo snap remove firefox; sudo snap install firefox --channel=latest/candidate; snap run firefox https://webglsamples.org/aquarium/aquarium.html

firefox (candidate) 92.0.1-1 from Mozilla✓ installed

broken

$ sudo snap remove firefox; sudo snap install firefox --channel=latest/beta; snap run firefox https://webglsamples.org/aquarium/aquarium.html

firefox (beta) 93.0b9-1 from Mozilla✓ installed

broken

Comment 2

[Darkspirit]

93.0b9: Problem does not occur with GLX/Xwayland. Problem also occurs with EGL/Xwayland.
$ DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html

Comment 3

[Darkspirit]

Tested:

• GLX is used on Gnome X11/Nvidia: fine
• EGL is used on Wayland because MOZ_ENABLE_WAYLAND is used (either Snap sets it as default or xwayland is not available)
• As of 94, EGL will also be used on X11 (bug 1695933).

This bug is also reproducible with Firefox Snap 92 when setting gfx.x11-egl.force-enabled to true on Gnome X11/Nvidia, but I don't see any DENIED in syslog.

Comment 4

[Robert Mader [:rmader]]

Hm, this reminds me of bug 1700601 and bug 1696691.

For the record: I'm going to land bug 1732002 to give us some more time to figure out stuff like this. Nvidia does not yet default to Wayland sessions on Ubuntu so setups with default configuration remain unaffected.

Comment 5

[Darkspirit]

Attachment: about:support from Firefox snap 92, Gnome X11, Intel

Also reproducible with Firefox Snap 92 and gfx.x11-egl.force-enabled=true on Gnome X11/Intel. No crash report.
Edit: Attachment is XFCE X11, but Gnome X11 looks the same.

glxtest works without problem, WebGL info on about:support is correct. It's just that WebGL doesn't work on websites.

Comment 6

[Robert Mader [:rmader]]

Ah I see. Sounds like it needs to get fixed by whoever maintains the Snap sandbox - likely little we can do in FF.

Comment 7

[Olivier Tilloy]

Pasting a comment of mine from bug 1726510:

« I instrumented the corresponding code and rebuilt the snap: in GetAndInitDisplay() (gfx/gl/GLLibraryEGL.cpp), egl.fGetDisplay() (eglGetDisplay()) always returns null. This requires further investigation. »

Comment 8

[Olivier Tilloy]

The apparmor denials on "/proc/sys/dev/i915/perf_stream_paranoid" mentioned in the description are red herrings: I can see them too, but if I edit the snap's generated apparmor profile (/var/lib/snapd/apparmor/profiles/snap.firefox.firefox) to allow read access to that file and reload it, the denials go away but the problem with WebGL persists.

Comment 9

[Olivier Tilloy]

I instrumented further the code, and here are my observations so far:

When launching firefox, the very first call to GLLibraryEGL::CreateDisplay() returns a valid EGL display. This is being called from RenderThread::CreateGLContextEGL(), from the app's main process.

Subsequent calls to GLLibraryEGL::CreateDisplay() follow a different code path (from GLContextProviderEGL::CreateHeadless(), from child (content) processes), and those invariably return null, which results in WebGL not working.

Comment 10

[Olivier Tilloy]

Running the app with MOZ_DISABLE_CONTENT_SANDBOX=1 "fixes" WebGL, so there is something in the content sandbox that prevents access to an EGL display.

Comment 11

[Darkspirit]

Disabling widget.dmabuf-webgl.enabled doesn't make a difference.

Comment 12

[Darkspirit]

Summary:
STR on Debian Testing:
$ sudo apt install snapd
$ sudo systemctl start snapd

$ sudo snap install firefox

When using a Wayland session:

• EGL/Wayland backend:
  broken: $ snap run firefox https://webglsamples.org/aquarium/aquarium.html
  works: $ MOZ_DISABLE_CONTENT_SANDBOX=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html
• EGL/Xwayland:
  broken: $ DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html
  works: $ MOZ_DISABLE_CONTENT_SANDBOX=1 DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html
• fine with glx/xwayland: $ DISABLE_WAYLAND=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html

When using an X11 session:

• EGL/X11:
  broken: $ MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html
  works: $ MOZ_DISABLE_CONTENT_SANDBOX=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html
• fine with GLX/X11: $ snap run firefox https://webglsamples.org/aquarium/aquarium.html

Comment 13

[Olivier Tilloy]

Also, reducing security.sandbox.content.level to 2 (from its default value of 4) in about:config makes the problem go away.

Comment 14

[Olivier Tilloy]

And setting security.sandbox.content.read_path_whitelist to $SNAP/ (e.g. /snap/firefox/595/, the trailing slash is important), also makes the problem go away, confirming that the content process sandboxing is what prevents the WebGL code from accessing the EGL library.

Comment 15

[Olivier Tilloy]

I'm tempted to patching SandboxBrokerPolicyFactory::InitContentPolicy() to call policy->AddPath(rdonly, "$SNAP/") when firefox is running as a snap. This will obviously require a thorough security review.

Comment 16

[Gian-Carlo Pascutto [:gcp]]

I would expect https://searchfox.org/mozilla-central/source/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#499 to pick up libraries shipped alongside Firefox. I guess what's happening here is that the library is shipped in the snap (but not in the default system), not next to the binary, and then "something" is done to make the dynamic linker pick it up?

Our sandbox knows about LD_LIBRARY_PATH and such https://searchfox.org/mozilla-central/rev/c3d7964c593e0bedabea2fea0b35ba243cf9e696/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#258 but I guess this is using something different?

In general readonly permission to trusted system library dirs should not be a security concern.

Comment 17

[Olivier Tilloy]

Indeed, the library is shipped by the snap. To be exact, it is shipped by the platform snap that the firefox snap uses as its base (gnome-3-38-2004), and the snap sees it at $SNAP/gnome-platform/usr/lib/x86_64-linux-gnu/libEGL.so. The snap's launcher modifies LD_LIBRARY_PATH accordingly. This is the value for a webcontent (child) process (where x21 is the snap's revision, because I manually installed an instrumented build):

LD_LIBRARY_PATH=/snap/firefox/x21/usr/lib/firefox:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/firefox/x21/usr/lib:/snap/firefox/x21/usr/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/usr/lib:/snap/firefox/x21/gnome-platform/lib:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/dri:/var/lib/snapd/lib/gl:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/libunity:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/pulseaudio

The path in question is there, so it's not immediately clear to me why it's not being added to the policy's list of readonly paths. Maybe the call to realpath(…) doesn't work well with the snap's confinement?

Comment 18

[Olivier Tilloy]

I was able to narrow down the one missing path (required to make WebGL work) to $SNAP/gnome-platform/usr/share/glvnd/egl_vendor.d/.

That path contains one single file (50_mesa.json):

{
    "file_format_version" : "1.0.0",
    "ICD" : {
        "library_path" : "libEGL_mesa.so.0"
    }
}

Comment 19

[Olivier Tilloy]

So the problem is not with LD_LIBRARY_PATH or with how the sandbox parses its value to add readonly permissions, it is with a configuration file that's located in a place that the policy doesn't allow read access to by default, and which contains information for EGL to locate the right implementation to load.

In that light, and considering that we might see other similar bugs (caused by configuration files for certain libs not being readable) in the future, my suggestion in comment 15 seems to be a valid and future-proof solution.

Comment 20

[Olivier Tilloy]

Attachment: Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f52ebe8f3b52
Allow read access to files under $SNAP/ in the webcontent sandbox. r=gcp

Comment 22

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/f52ebe8f3b52

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:olivier, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 24

[Olivier Tilloy]

Yes, this definitely needs to be uplifted to beta.

Comment 25

[Olivier Tilloy]

Comment on attachment 9244377 [details]
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Beta/Release Uplift Approval Request

• User impact if declined: WebGL doesn't work for users of the firefox snap in Wayland sessions.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Not risky because only the snap package is affected (the sandbox allows one additional folder for readonly access if the $SNAP environment variable is set).
• String changes made/needed:

Comment 26

[Olivier Tilloy]

This patch also appears to fix another problem that happens only in Wayland sessions, and was reported in Ubuntu (against the upcoming 21.10 release which ships the Firefox snap by default): https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1946599.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9244377 [details]
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Approved for 94.0b5.

Comment 28

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/111441e21e18

Comment 29

[Ryan VanderMeulen [:RyanVM]]

Does this need an ESR91 approval request also?

Comment 30

[Olivier Tilloy]

Yes indeed, this is clearly a candidate for an ESR91 uplift. Thanks for the suggestion Ryan.

Comment 31

[Olivier Tilloy]

Comment on attachment 9244377 [details]
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Snap packaging has changed significantly between ESR78 and ESR91. In ESR78 WebGL works in Wayland sessions, whereas it doesn't in ESR91, making it a very visible functional regression.
• User impact if declined: WebGL doesn't work for users of the firefox snap in Wayland sessions.
• Fix Landed on Version: 94.0b5
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Not risky because only the snap package is affected (the sandbox allows one additional folder for readonly access if the $SNAP environment variable is set).
• String or UUID changes made by this patch:

Comment 32

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9244377 [details]
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Approved for 91.3esr.

Comment 33

[Ryan VanderMeulen [:RyanVM]]

Looks like this depends on bug 1718084, though?

Comment 34

[Olivier Tilloy]

Yes, I hadn't realized this, but it depends on bug 1718084 indeed.

Comment 35

[Olivier Tilloy]

If pulling in the patches from bug 1718084 is deemed too risky for ESR91, I can rework this patch to not depend on them.
That would imply no unit tests, but on the upside the patch would be trivial.

Comment 36

[Ryan VanderMeulen [:RyanVM]]

I think we should take it unless there's a strong reason I'm missing. It's been baking for a couple months now, we're early in the ESR91 lifespan, and I won't be surprised if we eventually need to backport other patches that depend on it.

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr91/rev/13c638762db7

Comment 38

[Darkspirit]

(In reply to Darkspirit from comment #1)

$ sudo snap remove firefox; sudo snap install firefox --channel=latest/beta; snap run firefox https://webglsamples.org/aquarium/aquarium.html

firefox (beta) 93.0b9-1 from Mozilla✓ installed

broken

Debian Testing, Gnome Wayland, Intel

sudo snap remove firefox; sudo snap install firefox --channel=latest/beta

EGL/Wayland works:
snap run firefox https://webglsamples.org/aquarium/aquarium.html

EGL/Xwayland works:
DISABLE_WAYLAND=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html

Comment 41

[:gerard-majax]

FTR, the AddFutureDir we had to land on that patch was finally removed in bug 1750539.