Closed Bug 1438215 Opened 7 years ago Closed 2 years ago

Sandbox breaks ATI fglrx driver

Categories

(Core :: Security: Process Sandboxing, enhancement, P3)

Product:
Core
Component:
Security: Process Sandboxing
Platform:
x86_64
Linux
Type:
enhancement

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: gcp, Assigned: gcp)

References

(Blocks 1 open bug)

Details

We had some concerns whether part of the SysV IPC restrictions in the sandbox would break AMD/ATI's old proprietary Linux drivers, so I had a check. Currently, level 1 already breaks WebGL: Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/fglrx/dri/fglrx_dri.so Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/Downloads/firefox/libatiuki.so.1 Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786 Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0 Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786 Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0 Sandbox: SandboxBroker: denied op=unlink rflags=0 perms=3 path=/dev/ati/card0 for pid=4786 Sandbox: Failed errno -13 op 9 flags 00 path /dev/ati/card0 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card1 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card2 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card3 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card4 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card5 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card6 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card7 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card8 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card9 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card10 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card11 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card12 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card13 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card14 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card15 Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atioes.conf Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/egl/atiapfxxES.blb Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786 Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0 Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/dri/card0 for pid=4786 Sandbox: Failed errno -13 op 0 flags 02 path /dev/dri/card0 Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786 Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
For the record this is a completely un-updated Ubuntu 14.04 install. (I tried an updated one before and fglrx conflicts with the Mesa updates)
Whitelisting the paths leads to a process crash: https://crash-stats.mozilla.com/report/index/ec329c5d-293f-4bca-89fb-8df600180214 [ 2979.640196] <3>[fglrx:__create_mapping] *ERROR* Could not find the group private data [ 2979.640201] <3>[fglrx:__mc_heap_map_virtual_space] *ERROR* Failed to map the virtual space [ 2979.640203] <3>[fglrx:mc_heap_map_virtual_space] *ERROR* Can not get virtual address [ 2979.640205] <3>[fglrx:MCIL_GetVirtualAddressInDescriptor] *ERROR* Can not get the virtual address [ 2979.710642] Web Content[5111]: segfault at f83 ip 00007f154df9c349 sp 00007ffcdd5e3ed0 error 4 in fglrx_dri.so[7f154c9d0000+250a000]
Program received signal SIGSYS, Bad system call. 0x00007f83ab3efbfd in open64 () at ../sysdeps/unix/syscall-template.S:81 81 ../sysdeps/unix/syscall-template.S: No such file or directory. (gdb) Continuing. Program received signal SIGSYS, Bad system call. 0x00007f83aa4c8b87 in readlink () at ../sysdeps/unix/syscall-template.S:81 81 ../sysdeps/unix/syscall-template.S: No such file or directory. (SIGSEGV happens) #0 0x00007f33adf0c349 in ?? () from /usr/lib/dri/fglrx_dri.so #1 0x00007f33adf3e518 in ?? () from /usr/lib/dri/fglrx_dri.so #2 0x00007f33adf3e6e7 in ?? () from /usr/lib/dri/fglrx_dri.so #3 0x00007f33acc6dae0 in ?? () from /usr/lib/dri/fglrx_dri.so #4 0x00007f33adb35955 in ?? () from /usr/lib/dri/fglrx_dri.so #5 0x00007f33adb35986 in ?? () from /usr/lib/dri/fglrx_dri.so #6 0x00007f33adb45a10 in ?? () from /usr/lib/dri/fglrx_dri.so #7 0x00007f33adb46006 in ?? () from /usr/lib/dri/fglrx_dri.so #8 0x00007f33aea4063a in ?? () from /usr/lib/dri/fglrx_dri.so #9 0x00007f33aea3aa37 in ?? () from /usr/lib/dri/fglrx_dri.so #10 0x00007f33b2249edc in ?? () from /usr/lib/fglrx/libGL.so.1 #11 0x00007f33b22408ca in glXQueryVersion () from /usr/lib/fglrx/libGL.so.1 #12 0x00007f33ca3b43c2 in mozilla::gl::GLXLibrary::EnsureInitialized() () from /home/morbo/Downloads/firefox/libxul.so #13 0x00007f33ca3b521e in mozilla::gl::CreateOffscreenPixmapContext(mozilla::gl::CreateContextFlags, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gl::S
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/Downloads/firefox/libatiadlxx.so Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap Sandbox: Failed errno -2 op 0 flags 00 path /etc/ati/atiapfuser.blb Sandbox: rewriting /proc/self/exe -> /proc/4962/exe Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe Sandbox: rewriting /proc/self/exe -> /proc/4962/exe Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf Sandbox: rewriting /proc/self/exe -> /proc/4962/exe Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe Sandbox: Sandbox: rewriting /proc/self/smaps -> /proc/5074/smaps rewriting /proc/self/smaps -> /proc/4715/smaps Sandbox: rewriting /proc/self/statm -> /proc/4715/statm Sandbox: rewriting /proc/self/statm -> /proc/4715/statm Sandbox: rewriting /proc/self/smaps -> /proc/4715/smaps ExceptionHandler::GenerateDump cloned child 5134
I wonder if it's trying to call something like socket() or fork(), where we quietly return an error, and then doesn't handle the error properly and runs into a null pointer. This could be tested by changing them to use a trap function that logs the syscall info and returns the error… or changing them to InvalidSyscall() and setting MOZ_SANDBOX_CRASH_ON_ERROR=0, although that will change the errno to ENOSYS.
A quick check with mozregression shows it was originally broken by handling more filesystem related calls in the file broker: https://hg.mozilla.org/mozilla-central/rev/a79ec9afac7b https://hg.mozilla.org/mozilla-central/rev/c838d2546cad (So at least this version of the driver hasn't worked for >1 year) But it's a bit strange the verbose log shows no -13 errors. We don't appear to be rejecting anything, such as that last readlink call.
Actually the first commit there enabled the policy, so the breakage might be due to lack of access to /dev/ati. I'll have to put toolking on the machine so I can debug on it.
Assignee: nobody → gpascutto
LIBGL_DEBUG=verbose shows verbose output from the driver, quite useful. Whitelisting /dev/ati/, /proc/ati/ and /dev/dri/ proceeds a bit further: libGL: AtiGetClientDriverName: 15.20.3 fglrx (screen 0) libGL: OpenDriver: trying /usr/lib/fglrx/dri/fglrx_dri.so Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/fglrx/dri/fglrx_dri.so libGL error: OpenDriver: failed to open /usr/lib/fglrx/dri/fglrx_dri.so, error[/usr/lib/fglrx/dri/fglrx_dri.so: cannot open shared object file: No such file or directory] libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so libGL error: OpenDriver: failed to open /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so, error[/usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so: cannot open shared object file: No such file or directory] libGL: OpenDriver: trying /usr/lib/dri/fglrx_dri.so Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/libatiuki.so.1 Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap ukiDynamicMajor: found major device number 250 ukiDynamicMajor: found major device number 250 ukiDynamicMajor: found major device number 250 ukiOpenDevice: node name is /dev/ati/card0 ukiOpenDevice: open result is 15, (OK) ukiGetBusid returned 'PCI:0:1:0' ukiOpenDevice: node name is /dev/ati/card1 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card1 ukiOpenDevice: UKI_ERR_NOT_ROOT ukiOpenDevice: node name is /dev/ati/card2 Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card15 ukiOpenDevice: UKI_ERR_NOT_ROOT ukiDynamicMajor: found major device number 250 ukiOpenByBusid: Searching for BusID PCI:0:1:0 ukiOpenDevice: node name is /dev/ati/card0 ukiOpenDevice: open result is 15, (OK) ukiOpenByBusid: ukiOpenMinor returns 15 ukiOpenByBusid: ukiGetBusid reports PCI:0:1:0 Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atioes.conf Sandbox: SandboxBroker: denied op=open rflags=0 perms=0 path=/proc/5919/cmdline for pid=5919 Sandbox: Failed errno -13 op 0 flags 00 path /proc/5919/cmdline Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/egl/atiapfxxES.blb Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf Sandbox: rewriting /proc/self/exe -> /proc/5919/exe Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf Sandbox: rewriting /proc/self/exe -> /proc/5919/exe Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe Sandbox: rewriting /proc/self/exe -> /proc/5919/exe Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe ukiDynamicMajor: found major device number 250 ukiDynamicMajor: found major device number 250 ukiOpenByBusid: Searching for BusID PCI:0:1:0 ukiOpenDevice: node name is /dev/ati/card0 ukiOpenDevice: open result is 28, (OK) ukiOpenByBusid: ukiOpenMinor returns 28 ukiOpenByBusid: ukiGetBusid reports PCI:0:1:0 Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf Sandbox: rewriting /proc/self/exe -> /proc/5919/exe Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/libatiadlxx.so Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap Sandbox: Failed errno -2 op 0 flags 00 path /etc/ati/atiapfuser.blb Sandbox: rewriting /proc/self/exe -> /proc/5919/exe Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe Sandbox: rewriting /proc/self/exe -> /proc/5919/exe Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf Sandbox: rewriting /proc/self/exe -> /proc/5919/exe Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe Program /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox (pid = 5919) received signal 11. I vaguely remember issues with this driver trying to figure out what program it's running in by inspecting the command line. Not sure that triggers the crash, though.
Better stack: #01: UnixExceptionHandler (/home/morbo/hg/firefox/js/src/ds/MemoryProtectionExceptionHandler.cpp:272) #02: WasmFaultHandler (/home/morbo/hg/firefox/js/src/wasm/WasmSignalHandlers.cpp:1475) #03: __restore_rt (sigaction.c:?) #04: ??? (/usr/lib/dri/fglrx_dri.so) #05: ??? (/usr/lib/dri/fglrx_dri.so) #06: ??? (/usr/lib/dri/fglrx_dri.so) #07: ??? (/usr/lib/dri/fglrx_dri.so) #08: ??? (/usr/lib/dri/fglrx_dri.so) #09: ??? (/usr/lib/dri/fglrx_dri.so) #10: ??? (/usr/lib/dri/fglrx_dri.so) #11: ??? (/usr/lib/dri/fglrx_dri.so) #12: is64bitelf (/usr/lib/dri/fglrx_dri.so) #13: driCreateAssociatedContextAttribsAMD (/usr/lib/dri/fglrx_dri.so) #14: glXGetCurrentContext (/usr/lib/fglrx/libGL.so.1) #15: glXQueryVersion (/usr/lib/fglrx/libGL.so.1) #16: mozilla::gl::GLXLibrary::fQueryVersion(_XDisplay*, int*, int*) const (/home/morbo/hg/firefox/gfx/gl/GLXLibrary.h:119) Despite the hint it's looking at the binary, it crashes with level=1 only, which does suggest it's a syscall thing.
Priority: -- → P1
Lowering the sandbox to level 1 and disabling brokering makes this work. That seems to indicate it is, after all, a broker thing. Interestingly, disabling brokering and putting the level to 4 makes Firefox itself crash.
I should've realized sooner that the latter is because level 4 enables chroot() and of course nothing then works without brokering.
At this point I'm pretty sure the problem in the ATI driver is due to brokering, period. It seems to dislike the fd passing. That's also why logging didn't show anything interesting. We're not blocking the file it wants. IIRC Jed mentioned B2G also had problems with same-process checks in drivers.
Not sure what to do about this. Telemetry shows about 0.3% are on ATI drivers. The most popular driver is exactly the one I'm testing with: ATI Technologies Inc. 4.5.13399 Compatibility Profile Context 15.201.1151 https://sql.telemetry.mozilla.org/queries/51800/source These users will have been without WebGL for a few releases now. Unless this magically works in some Ubuntu update. Which I guess I can test now.
(In reply to Gian-Carlo Pascutto [:gcp] from comment #13) > IIRC Jed mentioned B2G also had problems with same-process checks in drivers. For reference, this is bug 930258 comment #9. That pid check was happening in the kernel, and those pids are from the kernel's data structures (see [1] and other uses of task_tgid_nr in that file), so nothing we can do to the getpid() syscall could affect it. If fglrx is similar, we don't have a lot of options. [1] https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/drivers/gpu/msm/kgsl.c#L481
This was the B2G problem with another vendors' GPU drivers: https://bugzilla.mozilla.org/show_bug.cgi?id=930258#c9 https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/drivers/gpu/msm/kgsl.c#L481 It seems ATI might have been doing a similar check. About the only way out here would be to disable the sandbox entirely (little sense in doing seccomp filtering but allowing open()) if we detect an affected ATI config. It's not clear what those are. Keeping the code as is will break (or crash) the tabs of affected users if they encounter a site that uses WebGL, and can still be worked around by disabling the sandbox manually. We'll keep an eye out for reports from the field in Bugzilla that boil down to this one, but given the Telemetry, this will be WONTFIX for now.
See Also: → 1434711
(In reply to Gian-Carlo Pascutto [:gcp] from comment #16) > This was the B2G problem with another vendors' GPU drivers: > https://bugzilla.mozilla.org/show_bug.cgi?id=930258#c9 > https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/ > drivers/gpu/msm/kgsl.c#L481 > > It seems ATI might have been doing a similar check. > > About the only way out here would be to disable the sandbox entirely (little > sense in doing seccomp filtering but allowing open()) if we detect an > affected ATI config. It's not clear what those are. Keeping the code as is > will break (or crash) the tabs of affected users if they encounter a site > that uses WebGL, and can still be worked around by disabling the sandbox > manually. > > We'll keep an eye out for reports from the field in Bugzilla that boil down > to this one, but given the Telemetry, this will be WONTFIX for now. These exact issues showed up as bug 1434711.
Flags: needinfo?(gpascutto)
I'm not clear on how bug 1434711 relates to this one. I thought what we'd observed so far is that fglrx would just return errors, not crash, with the sandbox policy as it stands — the crash mentioned in comment #2 required local patches to allow access to /dev/ati. Also, bug 1434711 shows that the amdgpu drivers have the same ID strings as fglrx, which suggests that the code is related and that fglrx's intolerance of fd-passing could also be present in amdgpu. Which would be a problem, because amdgpu is a supported driver, but it's also open-source and might accept patches and/or bug reports.
I'll prefer to take AMDGPU-PRO things in a new bug, even if ends up being the same underlying (rebranded) code. If only because we're more motivated to support a current driver than a deprecated one.
Flags: needinfo?(gpascutto)
Works for me!
Moving to p3 because no activity for at least 24 weeks.
Priority: P1 → P3
Severity: normal → S3

See comment 16.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.