1438215 - Sandbox breaks ATI fglrx driver

Status: NEW

(Core :: Security: Process Sandboxing, enhancement, P3)

Description

[Gian-Carlo Pascutto [:gcp]]

We had some concerns whether part of the SysV IPC restrictions in the sandbox would break AMD/ATI's old proprietary Linux drivers, so I had a check.

Currently, level 1 already breaks WebGL:

Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/fglrx/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/Downloads/firefox/libatiuki.so.1
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
Sandbox: SandboxBroker: denied op=unlink rflags=0 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 9 flags 00 path /dev/ati/card0
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card1
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card2
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card3
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card4
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card5
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card6
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card7
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card8
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card9
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card10
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card11
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card12
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card13
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card14
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card15
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atioes.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/egl/atiapfxxES.blb
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/dri/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/dri/card0
Sandbox: SandboxBroker: denied op=open rflags=2 perms=3 path=/dev/ati/card0 for pid=4786
Sandbox: Failed errno -13 op 0 flags 02 path /dev/ati/card0

Comment 1

[Gian-Carlo Pascutto [:gcp]]

For the record this is a completely un-updated Ubuntu 14.04 install. (I tried an updated one before and fglrx conflicts with the Mesa updates)

Comment 2

[Gian-Carlo Pascutto [:gcp]]

Whitelisting the paths leads to a process crash:

https://crash-stats.mozilla.com/report/index/ec329c5d-293f-4bca-89fb-8df600180214

[ 2979.640196] <3>[fglrx:__create_mapping] *ERROR* Could not find the group private data
[ 2979.640201] <3>[fglrx:__mc_heap_map_virtual_space] *ERROR* Failed to map the virtual space
[ 2979.640203] <3>[fglrx:mc_heap_map_virtual_space] *ERROR* Can not get virtual address
[ 2979.640205] <3>[fglrx:MCIL_GetVirtualAddressInDescriptor] *ERROR* Can not get the virtual address
[ 2979.710642] Web Content[5111]: segfault at f83 ip 00007f154df9c349 sp 00007ffcdd5e3ed0 error 4 in fglrx_dri.so[7f154c9d0000+250a000]

Comment 3

[Gian-Carlo Pascutto [:gcp]]

Program received signal SIGSYS, Bad system call.
0x00007f83ab3efbfd in open64 () at ../sysdeps/unix/syscall-template.S:81
81      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb)
Continuing.

Program received signal SIGSYS, Bad system call.
0x00007f83aa4c8b87 in readlink () at ../sysdeps/unix/syscall-template.S:81
81      ../sysdeps/unix/syscall-template.S: No such file or directory.


(SIGSEGV happens)

#0  0x00007f33adf0c349 in ?? () from /usr/lib/dri/fglrx_dri.so
#1  0x00007f33adf3e518 in ?? () from /usr/lib/dri/fglrx_dri.so
#2  0x00007f33adf3e6e7 in ?? () from /usr/lib/dri/fglrx_dri.so
#3  0x00007f33acc6dae0 in ?? () from /usr/lib/dri/fglrx_dri.so
#4  0x00007f33adb35955 in ?? () from /usr/lib/dri/fglrx_dri.so
#5  0x00007f33adb35986 in ?? () from /usr/lib/dri/fglrx_dri.so
#6  0x00007f33adb45a10 in ?? () from /usr/lib/dri/fglrx_dri.so
#7  0x00007f33adb46006 in ?? () from /usr/lib/dri/fglrx_dri.so
#8  0x00007f33aea4063a in ?? () from /usr/lib/dri/fglrx_dri.so
#9  0x00007f33aea3aa37 in ?? () from /usr/lib/dri/fglrx_dri.so
#10 0x00007f33b2249edc in ?? () from /usr/lib/fglrx/libGL.so.1
#11 0x00007f33b22408ca in glXQueryVersion () from /usr/lib/fglrx/libGL.so.1
#12 0x00007f33ca3b43c2 in mozilla::gl::GLXLibrary::EnsureInitialized() () from /home/morbo/Downloads/firefox/libxul.so
#13 0x00007f33ca3b521e in mozilla::gl::CreateOffscreenPixmapContext(mozilla::gl::CreateContextFlags, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gl::S

Comment 4

[Gian-Carlo Pascutto [:gcp]]

Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/Downloads/firefox/libatiadlxx.so
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
Sandbox: Failed errno -2 op 0 flags 00 path /etc/ati/atiapfuser.blb
Sandbox: rewriting /proc/self/exe -> /proc/4962/exe
Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe
Sandbox: rewriting /proc/self/exe -> /proc/4962/exe
Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: rewriting /proc/self/exe -> /proc/4962/exe
Sandbox: Recording mapping /home/morbo/Downloads/firefox/firefox -> /proc/4962/exe
Sandbox: Sandbox: rewriting /proc/self/smaps -> /proc/5074/smaps
rewriting /proc/self/smaps -> /proc/4715/smaps
Sandbox: rewriting /proc/self/statm -> /proc/4715/statm
Sandbox: rewriting /proc/self/statm -> /proc/4715/statm
Sandbox: rewriting /proc/self/smaps -> /proc/4715/smaps
ExceptionHandler::GenerateDump cloned child 5134

Comment 5

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I wonder if it's trying to call something like socket() or fork(), where we quietly return an error, and then doesn't handle the error properly and runs into a null pointer.  This could be tested by changing them to use a trap function that logs the syscall info and returns the error… or changing them to InvalidSyscall() and setting MOZ_SANDBOX_CRASH_ON_ERROR=0, although that will change the errno to ENOSYS.

Comment 6

[Gian-Carlo Pascutto [:gcp]]

A quick check with mozregression shows it was originally broken by handling more filesystem related calls in the file broker:
https://hg.mozilla.org/mozilla-central/rev/a79ec9afac7b
https://hg.mozilla.org/mozilla-central/rev/c838d2546cad

(So at least this version of the driver hasn't worked for >1 year)

But it's a bit strange the verbose log shows no -13 errors. We don't appear to be rejecting anything, such as that last readlink call.

Comment 7

[Gian-Carlo Pascutto [:gcp]]

Actually the first commit there enabled the policy, so the breakage might be due to lack of access to /dev/ati.

I'll have to put toolking on the machine so I can debug on it.

Comment 9

[Gian-Carlo Pascutto [:gcp]]

LIBGL_DEBUG=verbose shows verbose output from the driver, quite useful.

Whitelisting /dev/ati/, /proc/ati/ and /dev/dri/ proceeds a bit further:


libGL: AtiGetClientDriverName: 15.20.3 fglrx (screen 0)
libGL: OpenDriver: trying /usr/lib/fglrx/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/fglrx/dri/fglrx_dri.so
libGL error: OpenDriver: failed to open /usr/lib/fglrx/dri/fglrx_dri.so, error[/usr/lib/fglrx/dri/fglrx_dri.so: cannot open shared object file: No such file or directory]
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so
libGL error: OpenDriver: failed to open /usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so, error[/usr/lib/x86_64-linux-gnu/dri/fglrx_dri.so: cannot open shared object file: No such file or directory]
libGL: OpenDriver: trying /usr/lib/dri/fglrx_dri.so
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/libatiuki.so.1
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
ukiDynamicMajor: found major device number 250
ukiDynamicMajor: found major device number 250
ukiDynamicMajor: found major device number 250
ukiOpenDevice: node name is /dev/ati/card0
ukiOpenDevice: open result is 15, (OK)
ukiGetBusid returned 'PCI:0:1:0'
ukiOpenDevice: node name is /dev/ati/card1
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card1
ukiOpenDevice: UKI_ERR_NOT_ROOT
ukiOpenDevice: node name is /dev/ati/card2
Sandbox: Failed errno -2 op 2 flags 00 path /dev/ati/card15
ukiOpenDevice: UKI_ERR_NOT_ROOT
ukiDynamicMajor: found major device number 250
ukiOpenByBusid: Searching for BusID PCI:0:1:0
ukiOpenDevice: node name is /dev/ati/card0
ukiOpenDevice: open result is 15, (OK)
ukiOpenByBusid: ukiOpenMinor returns 15
ukiOpenByBusid: ukiGetBusid reports PCI:0:1:0
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atioes.conf
Sandbox: SandboxBroker: denied op=open rflags=0 perms=0 path=/proc/5919/cmdline for pid=5919
Sandbox: Failed errno -13 op 0 flags 00 path /proc/5919/cmdline
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/egl/atiapfxxES.blb
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
ukiDynamicMajor: found major device number 250
ukiDynamicMajor: found major device number 250
ukiOpenByBusid: Searching for BusID PCI:0:1:0
ukiOpenDevice: node name is /dev/ati/card0
ukiOpenDevice: open result is 28, (OK)
ukiOpenByBusid: ukiOpenMinor returns 28
ukiOpenByBusid: ukiGetBusid reports PCI:0:1:0
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl_perfcount.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: Failed errno -2 op 0 flags 02000000 path /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/libatiadlxx.so
Sandbox: Failed errno -2 op 1 flags 00 path /etc/ld.so.nohwcap
Sandbox: Failed errno -2 op 0 flags 00 path /etc/ati/atiapfuser.blb
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe
Sandbox: Failed errno -2 op 0 flags 00 path /etc/X11/atiogl.conf
Sandbox: rewriting /proc/self/exe -> /proc/5919/exe
Sandbox: Recording mapping /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox -> /proc/5919/exe

Program /home/morbo/hg/firefox/obj-x86_64-pc-linux-gnu/dist/bin/firefox (pid = 5919) received signal 11.

I vaguely remember issues with this driver trying to figure out what program it's running in by inspecting the command line. Not sure that triggers the crash, though.

Comment 10

[Gian-Carlo Pascutto [:gcp]]

Better stack:
#01: UnixExceptionHandler (/home/morbo/hg/firefox/js/src/ds/MemoryProtectionExceptionHandler.cpp:272)
#02: WasmFaultHandler (/home/morbo/hg/firefox/js/src/wasm/WasmSignalHandlers.cpp:1475)
#03: __restore_rt (sigaction.c:?)
#04: ??? (/usr/lib/dri/fglrx_dri.so)
#05: ??? (/usr/lib/dri/fglrx_dri.so)
#06: ??? (/usr/lib/dri/fglrx_dri.so)
#07: ??? (/usr/lib/dri/fglrx_dri.so)
#08: ??? (/usr/lib/dri/fglrx_dri.so)
#09: ??? (/usr/lib/dri/fglrx_dri.so)
#10: ??? (/usr/lib/dri/fglrx_dri.so)
#11: ??? (/usr/lib/dri/fglrx_dri.so)
#12: is64bitelf (/usr/lib/dri/fglrx_dri.so)
#13: driCreateAssociatedContextAttribsAMD (/usr/lib/dri/fglrx_dri.so)
#14: glXGetCurrentContext (/usr/lib/fglrx/libGL.so.1)
#15: glXQueryVersion (/usr/lib/fglrx/libGL.so.1)
#16: mozilla::gl::GLXLibrary::fQueryVersion(_XDisplay*, int*, int*) const (/home/morbo/hg/firefox/gfx/gl/GLXLibrary.h:119)

Despite the hint it's looking at the binary, it crashes with level=1 only, which does suggest it's a syscall thing.

Comment 11

[Gian-Carlo Pascutto [:gcp]]

Lowering the sandbox to level 1 and disabling brokering makes this work. That seems to indicate it is, after all, a broker thing.

Interestingly, disabling brokering and putting the level to 4 makes Firefox itself crash.

Comment 12

[Gian-Carlo Pascutto [:gcp]]

I should've realized sooner that the latter is because level 4 enables chroot() and of course nothing then works without brokering.

Comment 13

[Gian-Carlo Pascutto [:gcp]]

At this point I'm pretty sure the problem in the ATI driver is due to brokering, period. It seems to dislike the fd passing. That's also why logging didn't show anything interesting. We're not blocking the file it wants.

IIRC Jed mentioned B2G also had problems with same-process checks in drivers.

Comment 14

[Gian-Carlo Pascutto [:gcp]]

Not sure what to do about this. 

Telemetry shows about 0.3% are on ATI drivers. The most popular driver is exactly the one I'm testing with:
ATI Technologies Inc.	4.5.13399 Compatibility Profile Context 15.201.1151

https://sql.telemetry.mozilla.org/queries/51800/source

These users will have been without WebGL for a few releases now. Unless this magically works in some Ubuntu update. Which I guess I can test now.

Comment 15

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #13)
> IIRC Jed mentioned B2G also had problems with same-process checks in drivers.

For reference, this is bug 930258 comment #9.  That pid check was happening in the kernel, and those pids are from the kernel's data structures (see [1] and other uses of task_tgid_nr in that file), so nothing we can do to the getpid() syscall could affect it.  If fglrx is similar, we don't have a lot of options.

[1] https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/drivers/gpu/msm/kgsl.c#L481

Comment 16

[Gian-Carlo Pascutto [:gcp]]

This was the B2G problem with another vendors' GPU drivers:
https://bugzilla.mozilla.org/show_bug.cgi?id=930258#c9
https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/drivers/gpu/msm/kgsl.c#L481

It seems ATI might have been doing a similar check.

About the only way out here would be to disable the sandbox entirely (little sense in doing seccomp filtering but allowing open()) if we detect an affected ATI config. It's not clear what those are. Keeping the code as is will break (or crash) the tabs of affected users if they encounter a site that uses WebGL, and can still be worked around by disabling the sandbox manually.

We'll keep an eye out for reports from the field in Bugzilla that boil down to this one, but given the Telemetry, this will be WONTFIX for now.

Comment 18

[Jeff Gilbert [:jgilbert]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #16)
> This was the B2G problem with another vendors' GPU drivers:
> https://bugzilla.mozilla.org/show_bug.cgi?id=930258#c9
> https://github.com/mozilla-b2g/codeaurora_kernel_msm/blob/7731d63c809d/
> drivers/gpu/msm/kgsl.c#L481
> 
> It seems ATI might have been doing a similar check.
> 
> About the only way out here would be to disable the sandbox entirely (little
> sense in doing seccomp filtering but allowing open()) if we detect an
> affected ATI config. It's not clear what those are. Keeping the code as is
> will break (or crash) the tabs of affected users if they encounter a site
> that uses WebGL, and can still be worked around by disabling the sandbox
> manually.
> 
> We'll keep an eye out for reports from the field in Bugzilla that boil down
> to this one, but given the Telemetry, this will be WONTFIX for now.

These exact issues showed up as bug 1434711.

Comment 19

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I'm not clear on how bug 1434711 relates to this one.  I thought what we'd observed so far is that fglrx would just return errors, not crash, with the sandbox policy as it stands — the crash mentioned in comment #2 required local patches to allow access to /dev/ati.

Also, bug 1434711 shows that the amdgpu drivers have the same ID strings as fglrx, which suggests that the code is related and that fglrx's intolerance of fd-passing could also be present in amdgpu.  Which would be a problem, because amdgpu is a supported driver, but it's also open-source and might accept patches and/or bug reports.

Comment 20

[Gian-Carlo Pascutto [:gcp]]

I'll prefer to take AMDGPU-PRO things in a new bug, even if ends up being the same underlying (rebranded) code. If only because we're more motivated to support a current driver than a deprecated one.

Comment 21

[Jeff Gilbert [:jgilbert]]

Works for me!

Comment 22

[Sylvestre Ledru [:Sylvestre]]

Moving to p3 because no activity for at least 24 weeks.