Open Bug 1438215 Opened 2 years ago Updated 7 days ago

Sandbox breaks ATI fglrx driver

Categories

(Core :: Security: Process Sandboxing, enhancement, P3)

x86_64
Linux
enhancement

Tracking

()

People

(Reporter: gcp, Assigned: gcp)

References

Details

We had some concerns whether part of the SysV IPC restrictions in the sandbox would break AMD/ATI's old proprietary Linux drivers, so I had a check.

Currently, level 1 already breaks WebGL:

Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/fglrx/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/Downloads/firefox/libatiuki.so.1
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
Sandbox: SandboxBroker: denied op=unlink rflags=0 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 9 flags 00 path /dev/ati/card0
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card1
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card2
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card3
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card4
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card5
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card6
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card7
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card8
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card9
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card10
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card11
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card12
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card13
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card14
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card15
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atioes.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/egl/atiapfxxES.blb
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/dri/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/dri/card0
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
For the record this is a completely un-updated Ubuntu 14.04 install. (I tried an updated one before and fglrx conflicts with the Mesa updates)
Whitelisting the paths leads to a process crash:

https://crash-stats.mozilla.com/report/index/ec329c5d-293f-4bca-89fb-8df600180214

[ 2979.640196] <3>[fglrx:__create_mapping] *ERROR* Could not find the group private data
[ 2979.640201] <3>[fglrx:__mc_heap_map_virtual_space] *ERROR* Failed to map the virtual space
[ 2979.640203] <3>[fglrx:mc_heap_map_virtual_space] *ERROR* Can not get virtual address
[ 2979.640205] <3>[fglrx:MCIL_GetVirtualAddressInDescriptor] *ERROR* Can not get the virtual address
[ 2979.710642] Web Content[5111]: segfault at f83 ip 00007f154df9c349 sp 00007ffcdd5e3ed0 error 4 in fglrx_dri.so[7f154c9d0000+250a000]
Program received signal SIGSYS, Bad system call.
0x00007f83ab3efbfd in open64 () at ../sysdeps/unix/syscall-template.S:81
81      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb)
Continuing.

Program received signal SIGSYS, Bad system call.
0x00007f83aa4c8b87 in readlink () at ../sysdeps/unix/syscall-template.S:81
81      ../sysdeps/unix/syscall-template.S: No such file or directory.


(SIGSEGV happens)

#0  0x00007f33adf0c349 in ?? () from /usr/lib/dri/fglrx_dri.so
#1  0x00007f33adf3e518 in ?? () from /usr/lib/dri/fglrx_dri.so
#2  0x00007f33adf3e6e7 in ?? () from /usr/lib/dri/fglrx_dri.so
#3  0x00007f33acc6dae0 in ?? () from /usr/lib/dri/fglrx_dri.so
#4  0x00007f33adb35955 in ?? () from /usr/lib/dri/fglrx_dri.so
#5  0x00007f33adb35986 in ?? () from /usr/lib/dri/fglrx_dri.so
#6  0x00007f33adb45a10 in ?? () from /usr/lib/dri/fglrx_dri.so
#7  0x00007f33adb46006 in ?? () from /usr/lib/dri/fglrx_dri.so
#8  0x00007f33aea4063a in ?? () from /usr/lib/dri/fglrx_dri.so
#9  0x00007f33aea3aa37 in ?? () from /usr/lib/dri/fglrx_dri.so
#10 0x00007f33b2249edc in ?? () from /usr/lib/fglrx/libGL.so.1
#11 0x00007f33b22408ca in glXQueryVersion () from /usr/lib/fglrx/libGL.so.1
#12 0x00007f33ca3b43c2 in mozilla::gl::GLXLibrary::EnsureInitialized() () from /home/morbo/Downloads/firefox/libxul.so
#13 0x00007f33ca3b521e in mozilla::gl::CreateOffscreenPixmapContext(mozilla::gl::CreateContextFlags, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gl::S
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/Downloads/firefox/libatiadlxx.so
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
Sandbox: Failed errno -2 op 0 flags 00 path /etc/ati/atiapfuser.blb
Sandbox: rewriting /proc/self/exe -> /proc/4962/exe
Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe
Sandbox: rewriting /proc/self/exe -> /proc/4962/exe
Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: rewriting /proc/self/exe -> /proc/4962/exe
Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe
Sandbox: Sandbox: rewriting /proc/self/smaps -> /proc/5074/smaps
rewriting /proc/self/smaps -> /proc/4715/smaps
Sandbox: rewriting /proc/self/statm -> /proc/4715/statm
Sandbox: rewriting /proc/self/statm -> /proc/4715/statm
Sandbox: rewriting /proc/self/smaps -> /proc/4715/smaps
ExceptionHandler::GenerateDump cloned child 5134
I wonder if it's trying to call something like socket() or fork(), where we quietly return an error, and then doesn't handle the error properly and runs into a null pointer.  This could be tested by changing them to use a trap function that logs the syscall info and returns the error… or changing them to InvalidSyscall() and setting MOZ_SANDBOX_CRASH_ON_ERROR=0, although that will change the errno to ENOSYS.
A quick check with mozregression shows it was originally broken by handling more filesystem related calls in the file broker:
https://hg.mozilla.org/mozilla-central/rev/a79ec9afac7b
https://hg.mozilla.org/mozilla-central/rev/c838d2546cad

(So at least this version of the driver hasn't worked for >1 year)

But it's a bit strange the verbose log shows no -13 errors. We don't appear to be rejecting anything, such as that last readlink call.
Actually the first commit there enabled the policy, so the breakage might be due to lack of access to /dev/ati.

I'll have to put toolking on the machine so I can debug on it.
Assignee: nobody → gpascutto
Duplicate of this bug: 1438909
LIBGL_DEBUG=verbose shows verbose output from the driver, quite useful.

Whitelisting /dev/ati/, /proc/ati/ and /dev/dri/ proceeds a bit further:


libGL: AtiGetClientDriverName: 15.20.3 fglrx (screen 0)
libGL: OpenDriver: trying /usr/lib/fglrx/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/fglrx/dri/fglrx_dri.so
libGL error: OpenDriver: failed to open /usr/lib/fglrx/dri/fglrx_dri.so, error[/usr/lib/fglrx/dri/fglrx_dri.so: cannot open shared object file: No such file or directory]
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so
libGL error: OpenDriver: failed to open /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so, error[/usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so: cannot open shared object file: No such file or directory]
libGL: OpenDriver: trying /usr/lib/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/libatiuki.so.1
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
ukiDynamicMajor: found major device number 250
ukiDynamicMajor: found major device number 250
ukiDynamicMajor: found major device number 250
ukiOpenDevice: node name is /dev/ati/card0
ukiOpenDevice: open result is 15, (OK)
ukiGetBusid returned 'PCI:0:1:0'
ukiOpenDevice: node name is /dev/ati/card1
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card1
ukiOpenDevice: UKI_ERR_NOT_ROOT
ukiOpenDevice: node name is /dev/ati/card2
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card15
ukiOpenDevice: UKI_ERR_NOT_ROOT
ukiDynamicMajor: found major device number 250
ukiOpenByBusid: Searching for BusID PCI:0:1:0
ukiOpenDevice: node name is /dev/ati/card0
ukiOpenDevice: open result is 15, (OK)
ukiOpenByBusid: ukiOpenMinor returns 15
ukiOpenByBusid: ukiGetBusid reports PCI:0:1:0
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atioes.conf
Sandbox: SandboxBroker: denied op=open rflags=0 perms=0 path=/proc/5919/cmdline for pid=5919
Sandbox: Failed errno -13 op 0 flags 00 path /proc/5919/cmdline
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/egl/atiapfxxES.blb
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
ukiDynamicMajor: found major device number 250
ukiDynamicMajor: found major device number 250
ukiOpenByBusid: Searching for BusID PCI:0:1:0
ukiOpenDevice: node name is /dev/ati/card0
ukiOpenDevice: open result is 28, (OK)
ukiOpenByBusid: ukiOpenMinor returns 28
ukiOpenByBusid: ukiGetBusid reports PCI:0:1:0
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/libatiadlxx.so
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
Sandbox: Failed errno -2 op 0 flags 00 path /etc/ati/atiapfuser.blb
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe

Program /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox (pid = 5919) received signal 11.

I vaguely remember issues with this driver trying to figure out what program it's running in by inspecting the command line. Not sure that triggers the crash, though.
Better stack:
#01: UnixExceptionHandler (/home/morbo/hg/firefox/js/src/ds/MemoryProtectionExceptionHandler.cpp:272)
#02: WasmFaultHandler (/home/morbo/hg/firefox/js/src/wasm/WasmSignalHandlers.cpp:1475)
#03: __restore_rt (sigaction.c:?)
#04: ??? (/usr/lib/dri/fglrx_dri.so)
#05: ??? (/usr/lib/dri/fglrx_dri.so)
#06: ??? (/usr/lib/dri/fglrx_dri.so)
#07: ??? (/usr/lib/dri/fglrx_dri.so)
#08: ??? (/usr/lib/dri/fglrx_dri.so)
#09: ??? (/usr/lib/dri/fglrx_dri.so)
#10: ??? (/usr/lib/dri/fglrx_dri.so)
#11: ??? (/usr/lib/dri/fglrx_dri.so)
#12: is64bitelf (/usr/lib/dri/fglrx_dri.so)
#13: driCreateAssociatedContextAttribsAMD (/usr/lib/dri/fglrx_dri.so)
#14: glXGetCurrentContext (/usr/lib/fglrx/libGL.so.1)
#15: glXQueryVersion (/usr/lib/fglrx/libGL.so.1)
#16: mozilla::gl::GLXLibrary::fQueryVersion(_XDisplay*, int*, int*) const (/home/morbo/hg/firefox/gfx/gl/GLXLibrary.h:119)

Despite the hint it's looking at the binary, it crashes with level=1 only, which does suggest it's a syscall thing.
Priority: -- → P1
Lowering the sandbox to level 1 and disabling brokering makes this work. That seems to indicate it is, after all, a broker thing.

Interestingly, disabling brokering and putting the level to 4 makes Firefox itself crash.
I should've realized sooner that the latter is because level 4 enables chroot() and of course nothing then works without brokering.
At this point I'm pretty sure the problem in the ATI driver is due to brokering, period. It seems to dislike the fd passing. That's also why logging didn't show anything interesting. We're not blocking the file it wants.

IIRC Jed mentioned B2G also had problems with same-process checks in drivers.
Not sure what to do about this. 

Telemetry shows about 0.3% are on ATI drivers. The most popular driver is exactly the one I'm testing with:
ATI Technologies Inc.	4.5.13399 Compatibility Profile Context 15.201.1151

https://sql.telemetry.mozilla.org/queries/51800/source

These users will have been without WebGL for a few releases now. Unless this magically works in some Ubuntu update. Which I guess I can test now.
(In reply to Gian-Carlo Pascutto [:gcp] from comment #13)
> IIRC Jed mentioned B2G also had problems with same-process checks in drivers.

For reference, this is bug 930258 comment #9.  That pid check was happening in the kernel, and those pids are from the kernel's data structures (see [1] and other uses of task_tgid_nr in that file), so nothing we can do to the getpid() syscall could affect it.  If fglrx is similar, we don't have a lot of options.

[1] https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/drivers/gpu/msm/kgsl.c#L481
This was the B2G problem with another vendors' GPU drivers:
https://bugzilla.mozilla.org/show_bug.cgi?id=930258#c9
https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/drivers/gpu/msm/kgsl.c#L481

It seems ATI might have been doing a similar check.

About the only way out here would be to disable the sandbox entirely (little sense in doing seccomp filtering but allowing open()) if we detect an affected ATI config. It's not clear what those are. Keeping the code as is will break (or crash) the tabs of affected users if they encounter a site that uses WebGL, and can still be worked around by disabling the sandbox manually.

We'll keep an eye out for reports from the field in Bugzilla that boil down to this one, but given the Telemetry, this will be WONTFIX for now.
See Also: → 1434711
Duplicate of this bug: 1434711
(In reply to Gian-Carlo Pascutto [:gcp] from comment #16)
> This was the B2G problem with another vendors' GPU drivers:
> https://bugzilla.mozilla.org/show_bug.cgi?id=930258#c9
> https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/
> drivers/gpu/msm/kgsl.c#L481
> 
> It seems ATI might have been doing a similar check.
> 
> About the only way out here would be to disable the sandbox entirely (little
> sense in doing seccomp filtering but allowing open()) if we detect an
> affected ATI config. It's not clear what those are. Keeping the code as is
> will break (or crash) the tabs of affected users if they encounter a site
> that uses WebGL, and can still be worked around by disabling the sandbox
> manually.
> 
> We'll keep an eye out for reports from the field in Bugzilla that boil down
> to this one, but given the Telemetry, this will be WONTFIX for now.

These exact issues showed up as bug 1434711.
Flags: needinfo?(gpascutto)
I'm not clear on how bug 1434711 relates to this one.  I thought what we'd observed so far is that fglrx would just return errors, not crash, with the sandbox policy as it stands — the crash mentioned in comment #2 required local patches to allow access to /dev/ati.

Also, bug 1434711 shows that the amdgpu drivers have the same ID strings as fglrx, which suggests that the code is related and that fglrx's intolerance of fd-passing could also be present in amdgpu.  Which would be a problem, because amdgpu is a supported driver, but it's also open-source and might accept patches and/or bug reports.
I'll prefer to take AMDGPU-PRO things in a new bug, even if ends up being the same underlying (rebranded) code. If only because we're more motivated to support a current driver than a deprecated one.
Flags: needinfo?(gpascutto)
Works for me!
Moving to p3 because no activity for at least 24 weeks.
Priority: P1 → P3
You need to log in before you can comment on or make changes to this bug.