1770905 - Crash when capturing recording on a page using Widevine

Status: RESOLVED FIXED

(Core :: Security: Process Sandboxing, defect, P2)

Description

[a_github]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0

Steps to reproduce:

Go to https://www.canalplus.com/live/?channel=450
Open dev tools
Start recording performance
Capture recording

Actual results:

Browser crash

In the console:

Sandbox: seccomp sandbox violation: pid 91287, tid 91341, syscall 89, args 140180885873351 140180769983168 4095 4294967295 140180770337952 2147483649.
Sandbox: attempt to open unexpected file /proc/91287/maps
Sandbox: attempt to open unexpected file 
Sandbox: attempt to open unexpected file linux-vdso.so.1
Sandbox: attempt to open unexpected file /usr/lib/firefox/libmozsandbox.so
Sandbox: attempt to open unexpected file /usr/lib/firefox/libxul.so
Sandbox: attempt to open unexpected file /usr/lib/libstdc++.so.6
Sandbox: attempt to open unexpected file /usr/lib/libm.so.6
Sandbox: attempt to open unexpected file /usr/lib/libgcc_s.so.1
Sandbox: attempt to open unexpected file /usr/lib/libc.so.6
Sandbox: attempt to open unexpected file /lib64/ld-linux-x86-64.so.2
Sandbox: attempt to open unexpected file /usr/lib/libnspr4.so
Sandbox: attempt to open unexpected file /usr/lib/firefox/liblgpllibs.so
Sandbox: attempt to open unexpected file /usr/lib/firefox/libmozsqlite3.so
Sandbox: attempt to open unexpected file /usr/lib/firefox/libmozgtk.so
Sandbox: attempt to open unexpected file /usr/lib/firefox/libmozwayland.so
Sandbox: attempt to open unexpected file /usr/lib/libasound.so.2
Sandbox: attempt to open unexpected file /usr/lib/libX11.so.6
Sandbox: attempt to open unexpected file /usr/lib/libXcomposite.so.1
Sandbox: attempt to open unexpected file /usr/lib/libXdamage.so.1
Sandbox: attempt to open unexpected file /usr/lib/libXext.so.6
Sandbox: attempt to open unexpected file /usr/lib/libXfixes.so.3
Sandbox: attempt to open unexpected file /usr/lib/libXrandr.so.2
Sandbox: attempt to open unexpected file /usr/lib/libplc4.so
Sandbox: attempt to open unexpected file /usr/lib/libssl3.so
Sandbox: attempt to open unexpected file /usr/lib/libsmime3.so
Sandbox: attempt to open unexpected file /usr/lib/libnss3.so
Sandbox: attempt to open unexpected file /usr/lib/libnssutil3.so
Sandbox: attempt to open unexpected file /usr/lib/libfreetype.so.6
Sandbox: attempt to open unexpected file /usr/lib/libfontconfig.so.1
Sandbox: attempt to open unexpected file /usr/lib/libgtk-3.so.0
Sandbox: attempt to open unexpected file /usr/lib/libgdk-3.so.0
Sandbox: attempt to open unexpected file /usr/lib/libpango-1.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libatk-1.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libcairo-gobject.so.2
Sandbox: attempt to open unexpected file /usr/lib/libcairo.so.2
Sandbox: attempt to open unexpected file /usr/lib/libgdk_pixbuf-2.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libgio-2.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libgobject-2.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libglib-2.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libdbus-glib-1.so.2
Sandbox: attempt to open unexpected file /usr/lib/libdbus-1.so.3
Sandbox: attempt to open unexpected file /usr/lib/libxcb-shm.so.0
Sandbox: attempt to open unexpected file /usr/lib/libX11-xcb.so.1
Sandbox: attempt to open unexpected file /usr/lib/libxcb.so.1
Sandbox: attempt to open unexpected file /usr/lib/libdl.so.2
Sandbox: attempt to open unexpected file /usr/lib/libpthread.so.0
Sandbox: attempt to open unexpected file /usr/lib/libXrender.so.1
Sandbox: attempt to open unexpected file /usr/lib/libplds4.so
Sandbox: attempt to open unexpected file /usr/lib/libz.so.1
Sandbox: attempt to open unexpected file /usr/lib/libbz2.so.1.0
Sandbox: attempt to open unexpected file /usr/lib/libpng16.so.16
Sandbox: attempt to open unexpected file /usr/lib/libharfbuzz.so.0
Sandbox: attempt to open unexpected file /usr/lib/libbrotlidec.so.1
Sandbox: attempt to open unexpected file /usr/lib/libexpat.so.1
Sandbox: attempt to open unexpected file /usr/lib/libgmodule-2.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libpangocairo-1.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libpangoft2-1.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libfribidi.so.0
Sandbox: attempt to open unexpected file /usr/lib/libepoxy.so.0
Sandbox: attempt to open unexpected file /usr/lib/libXi.so.6
Sandbox: attempt to open unexpected file /usr/lib/libatk-bridge-2.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libcloudproviders.so.0
Sandbox: attempt to open unexpected file /usr/lib/libtracker-sparql-3.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libxkbcommon.so.0
Sandbox: attempt to open unexpected file /usr/lib/libwayland-client.so.0
Sandbox: attempt to open unexpected file /usr/lib/libwayland-cursor.so.0
Sandbox: attempt to open unexpected file /usr/lib/libwayland-egl.so.1
Sandbox: attempt to open unexpected file /usr/lib/libXcursor.so.1
Sandbox: attempt to open unexpected file /usr/lib/libXinerama.so.1
Sandbox: attempt to open unexpected file /usr/lib/libthai.so.0
Sandbox: attempt to open unexpected file /usr/lib/libxcb-render.so.0
Sandbox: attempt to open unexpected file /usr/lib/libpixman-1.so.0
Sandbox: attempt to open unexpected file /usr/lib/libjpeg.so.8
Sandbox: attempt to open unexpected file /usr/lib/libtiff.so.5
Sandbox: attempt to open unexpected file /usr/lib/libmount.so.1
Sandbox: attempt to open unexpected file /usr/lib/libffi.so.8
Sandbox: attempt to open unexpected file /usr/lib/libpcre.so.1
Sandbox: attempt to open unexpected file /usr/lib/libsystemd.so.0
Sandbox: attempt to open unexpected file /usr/lib/libXau.so.6
Sandbox: attempt to open unexpected file /usr/lib/libXdmcp.so.6
Sandbox: attempt to open unexpected file /usr/lib/libgraphite2.so.3
Sandbox: attempt to open unexpected file /usr/lib/libbrotlicommon.so.1
Sandbox: attempt to open unexpected file /usr/lib/libatspi.so.0
Sandbox: attempt to open unexpected file /usr/lib/libstemmer.so.0
Sandbox: attempt to open unexpected file /usr/lib/libicuuc.so.71
Sandbox: attempt to open unexpected file /usr/lib/libicui18n.so.71
Sandbox: attempt to open unexpected file /usr/lib/libsqlite3.so.0
Sandbox: attempt to open unexpected file /usr/lib/libjson-glib-1.0.so.0
Sandbox: attempt to open unexpected file /usr/lib/libxml2.so.2
Sandbox: attempt to open unexpected file /usr/lib/libdatrie.so.1
Sandbox: attempt to open unexpected file /usr/lib/libzstd.so.1
Sandbox: attempt to open unexpected file /usr/lib/liblzma.so.5
Sandbox: attempt to open unexpected file /usr/lib/libblkid.so.1
Sandbox: attempt to open unexpected file /usr/lib/libcap.so.2
Sandbox: attempt to open unexpected file /usr/lib/libgcrypt.so.20
Sandbox: attempt to open unexpected file /usr/lib/liblz4.so.1
Sandbox: attempt to open unexpected file /usr/lib/libicudata.so.71
Sandbox: attempt to open unexpected file /usr/lib/libgpg-error.so.0
Sandbox: attempt to open unexpected file /usr/lib/librt.so.1
Sandbox: unexpected multiple open of file /home/user/.mozilla/firefox/9a6htdw7.default-release/gmp-widevinecdm/4.10.2449.0/libwidevinecdm.so
Sandbox: seccomp sandbox violation: pid 91287, tid 91291, syscall 77, args 24 163840 0 140180769996800 140180770001272 0.
[GMP 91287, ProfilerChild] WARNING: ftruncate failed to set shm size: Function not implemented: file /build/firefox/src/firefox-100.0.2/ipc/chromium/src/base/shared_memory_posix.cc:395
ExceptionHandler::GenerateDump cloned child 91348
ExceptionHandler::SendContinueSignalToChild sent continue signal to child
ExceptionHandler::WaitForContinueSignal waiting for continue signal...
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.

Expected results:

Show profiler results.

Comment 1

[a_github]

I reproduce also on https://integration.widevine.com/player

Comment 2

[a_github]

Issue is also present in Firefox Nightly, 102.0a1 (2022-05-24)

Comment 3

[Virgil Sangerean]

Reproduced issue on Ubuntu 16 & Ubuntu 20.
Not reproducible on Windows 10, Windows 11 and Mac 12 ARM.

Comment 4

[Alastor Wu [:alwu]]

Does this issue only happen while recording? or it could also happen alone? Would you mind to paste your about:support?
Thanks!

Comment 5

[Alastor Wu [:alwu]]

Bob, would you have any idea about how to diagnose this kind of sandbox error?
Thanks!

Comment 6

[a_github]

Here are the about:support contents: https://pastebin.com/58LaVSQB

I can only reproduce the issue while recording, then capturing performance in dev tools. No issues detected for simple Widevine-protected playback.

Comment 7

[Bob Owen (:bobowen)]

For Linux x86_64 looks like syscall 89 is __NR_readlink, but jld is probably best placed to help.

Comment 8

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

There are some conflicts between the profiler and the GeckoMediaPlugin process sandbox on Linux; I don't think anything relevant about the sandbox or the profiler's behavior has changed recently, so it's possible that this has been broken since bug 1744522 landed.

Specifically, there are two problems:

1. The profiler tries to create shared memory; we're not expecting that, so some of the system calls used (at least ftruncate) isn't allowed. This could be fixed, especially if we can support profiling GMP processes only on kernels ≥ 3.17 (when memfd_create was added). This is also why the browser is crashing: the child process fails to create the Shmem but the media code doesn't check for failure, so it sends an invalid Shmem to the parent, so we try to create a string from {nullptr, 0xffffffffffffffff}, which causes an assertion failure. That should also be fixed independently of the issues with sandboxing.

2. The string of complaints about opening files: it's trying to read its own list of memory mappings (/proc/self/maps), then presumably falls back on calling dl_iterate_phdr (to read the dynamic linker state in the process), and tries to open each of those files. I think it should be able to get unwinding info without looking at the filesystem — it's officially for exception handling, so it's part of the binary that's mapped into memory. Trying to read symbols from the binaries won't work; that would have to be done in the parent process, as a change to the profiler.

And I'm not sure what's happening with that readlink yet.

Maybe the right thing to do here is to turn off profiling for sandboxed GMP processes on Linux for now — I have a patch to do this — and then figure out how it can be re-enabled.

Comment 9

[Florian Quèze [:florian]]

Is the behavior related to trying to access the shared libraries related to LUL initialization? If so, we could keep profiling enabled and disable only nativestackwalking.

Comment 10

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Florian Quèze [:florian] from comment #9)

Is the behavior related to trying to access the shared libraries related to LUL initialization? If so, we could keep profiling enabled and disable only nativestackwalking.

Yes (mostly), and that's a good point — we can still record the pseudostack/markers.

I think native stack walking could work without needing to open any files — .eh_frame* and the section with the build ID are both in a part of the file that's mapped into memory, and there are phdrs that point to both of them, so we should have all the info we need without touching the filesystem. And I forgot this while writing comment #8, but symbolication is done in the frontend, even for developer builds where it reads from the local filesystem, so that's not directly affected.

Currently we seem to be trying to get the full path from /proc/self/maps, match that up with the dl_iterate_phdr info, and then re-open the file to read from it; in this case that's blocked by sandboxing, so the unwinder does something anyway (frame pointers?), and then the results aren't symbolicated, which I think is because it doesn't have the build ID. But in theory that's fixable, and it also just means that there are some unhelpful stack frames.

Also, the readlink is of /proc/self/exe, to fill in the executable's path (the dl_iterate_phdr entry for that has an empty name); that can be simulated (save the value ahead of time, trap the syscall, check if the path is /proc/self/exe and copy it out, else EINVAL), but since native stack walking is currently broken I can just make it quietly fail with EINVAL (the error used for not-a-symlink) as a minimum viable patch.

So, I think the thing to do here in the short term is:

1. Make shm allocation work in the non-/dev/shm case, which should cover every distro from the past ~7 years (if you aren't unmounting /proc for privacy hardening, but in that case you probably don't want to use DRM)
2. readlink → EINVAL

And that should give us a state where nothing will crash, old distros (e.g., RHEL ≤7) won't profile GMP processes, and more recent ones will get unusable native stacks for GMP but otherwise work.

Comment 11

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: Bug 1770905 - Add a validity check to the profiler's use of IPC Shmem.

If the child process failed to allocate Shmem (and this isn't a debug
build, where that would cause an assertion failure) when sending
profiling data back to the parent process, it will send an invalid Shmem
and the parent process should not try to use it.

Currently it does do that, and tries to construct an nsACString with
length (size_t)-1, which fails a release assertion in the XPCOM string
library and crashes the browser. With this patch, we'll simply fail to
profile the affected process.

Comment 12

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: Bug 1770905 - Quietly reject `readlink` in the Linux GeckoMediaPlugin sandbox.

The profiler may try to readlink /proc/self/exe to determine the
executable name; currently, its attempt to get information about loaded
objects is broken for other reasons, so this isn't helpful. Thus, this
patch has it fail with EINVAL (meaning "not a symbolic link) instead of
being treated as unexpected.

(In the future, if we need to, we could simulate that syscall by
recording the target of /proc/self/exe before sandboxing, and
recognizing that specific case in a trap function.)

Comment 13

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: Bug 1770905 - Allow GeckoMediaPlugin processes on Linux to be profiled if memfd_create is available.

There are two parts to this patch; both affect only Linux:

1. The GMP sandbox policy is adjusted to allow certain syscalls used in
   shared memory creation (ftruncate and fallocate). However, the file
   broker is not used; the process still has no access to files in /dev/shm.

2. The profiler is not initialized for GMP processes unless memfd_create
   is available (so the process can create shared memory to send
   profiling data back, without filesystem access), or the GMP sandbox
   is disabled (either at runtime or build time).

As of this patch, profiling GMP processes on Linux should succeed on
distros with kernel >=3.17 (Oct. 2014), but native stack frames won't
have symbols (and may be incorrectly unwound, not that it matters much
without symbols); see the bug for more info. Pseudo-stack frames and
markers should work, however.

Comment 14

[Donal Meehan [:dmeehan]]

:jld do you plan on landing this before next week's 103 soft freeze?

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e58d3a122ddd
Add a validity check to the profiler's use of IPC Shmem. r=gerald
https://hg.mozilla.org/integration/autoland/rev/4280a7d0ee17
Quietly reject `readlink` in the Linux GeckoMediaPlugin sandbox. r=gcp
https://hg.mozilla.org/integration/autoland/rev/9a4be9c8c0c6
Allow GeckoMediaPlugin processes on Linux to be profiled if memfd_create is available. r=gcp,mstange,media-playback-reviewers,padenot

Comment 16

[bszekely]

https://hg.mozilla.org/mozilla-central/rev/e58d3a122ddd
https://hg.mozilla.org/mozilla-central/rev/4280a7d0ee17
https://hg.mozilla.org/mozilla-central/rev/9a4be9c8c0c6