Sandboxed processes crash due to glibc 2.31 dlopen() blocking SIGSYS while opening files
Categories
(Core :: Security: Process Sandboxing, defect, P1)
Tracking
()
People
(Reporter: emilio, Assigned: jld)
References
Details
Attachments
(2 files)
Happens also on release... MOZ_DISABLE_CONTENT_SANDBOX=1 fixes it.
I thought they changed some syscall or something, but this patch doesn't fix it so it's something else:
diff --git a/security/sandbox/linux/SandboxFilter.cpp b/security/sandbox/linux/SandboxFilter.cpp
index 4afae8c1ef71..ff38be5fe991 100644
--- a/security/sandbox/linux/SandboxFilter.cpp
+++ b/security/sandbox/linux/SandboxFilter.cpp
@@ -405,6 +405,7 @@ class SandboxPolicyCommon : public SandboxPolicyBase {
}
ResultExpr EvaluateSyscall(int sysno) const override {
+ return Allow();
// If a file broker client was provided, route syscalls to it;
// otherwise, fall through to the main policy, which will deny
// them.
I dont' get any sane debugging info on the teminal, and rr is not working whatsoever. But with that patch I can get a bit further and get:
Crash Annotation GraphicsCriticalError: |[C0][GFX1]: no fonts - init: 1 fonts: 74 loader: 0 (t=0.294001) [GFX1]: no fonts - init: 1 fonts: 74 loader: 0
Assertion failure: [GFX1]: no fonts - init: 1 fonts: 74 loader: 0, at /home/emilio/src/moz/gecko/gfx/2d/Logging.h:740
#01: ???[/home/emilio/src/moz/gecko/obj-debug-static-analysis/dist/bin/libxul.so +0x649505a]
|
Reporter | |
Comment 1•4 years ago
|
||
Also:
Assertion failure: ((bool)(__builtin_expect(!!(!NS_FAILED_impl(rv)), 1))) (Deserializing security info should not fail), at /home/emilio/src/moz/gecko/netwerk/protocol/http/HttpChannelChild.cpp
|
|
Reporter | |
Comment 2•4 years ago
|
||
Ok, with the unconditional Allow() filter rr works, I'll poke a bit at what's going on.
|
|
Reporter | |
Comment 3•4 years ago
|
||
There's also a bunch of "Chrome file doesn't exist:" lines in the content process, for files that do exist...
[pid 38318] access("/home/emilio/src/moz/gecko/obj-debug-static-analysis/dist/bin/chrome/toolkit/content/global/browser-content.js", F_OK <unfinished ...>
[pid 38323] epoll_wait(4, <unfinished ...>
[pid 38261] futex(0x7f1333549230, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 38259] poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=84, events=POLLIN}], 3, -1 <unfinished ...>
[pid 38318] <... access resumed>) = -1 ENOENT (No such file or directory)
[pid 38261] <... futex resumed>) = 0
[pid 38259] <... poll resumed>) = 1 ([{fd=84, revents=POLLIN}])
[pid 38318] write(1, "Chrome file doesn't exist: /home"..., 138 <unfinished ...>
Chrome file doesn't exist: /home/emilio/src/moz/gecko/obj-debug-static-analysis/dist/bin/chrome/toolkit/content/global/browser-content.js
|
|
Reporter | |
Comment 4•4 years ago
|
||
So I'm starting to think this was a kernel update and not glibc update what caused this... Current version for posterity is 5.4.0-2.fc32.x86_64 #1 SMP Mon Nov 25 22:45:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux... Will try to use an older kernel and see if it works.
|
|
Reporter | |
Comment 5•4 years ago
|
||
Nope, 5.4.0-0.rc8.git0.2.fc32.x86_64 #1 SMP Wed Nov 20 18:14:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux doesn't work, and is what I was using yesterday :)
|
|
Reporter | |
Comment 6•4 years ago
|
||
Last update seems to have been:
Install libtracker-control-2.3.1-2.fc32.x86_64 @rawhide
Install libtracker-miner-2.3.1-2.fc32.x86_64 @rawhide
Upgrade neovim-0.4.3.0.git.13109.c62abb284-1.fc32.x86_64 @copr:copr.fedorainfracloud.org:agriffis:neovim-nightly
Upgraded neovim-0.4.3.0.git.13081.5f4c5aa56-1.fc32.x86_64 @@System
Upgrade ModemManager-1.10.8-1.fc32.x86_64 @rawhide
Upgraded ModemManager-1.10.6-2.fc32.x86_64 @@System
Upgrade ModemManager-glib-1.10.8-1.fc32.x86_64 @rawhide
Upgraded ModemManager-glib-1.10.6-2.fc32.x86_64 @@System
Upgrade NetworkManager-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade NetworkManager-adsl-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-adsl-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade NetworkManager-bluetooth-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-bluetooth-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade NetworkManager-config-connectivity-fedora-1:1.22.0-0.2.fc32.noarch @rawhide
Upgraded NetworkManager-config-connectivity-fedora-1:1.22.0-0.1.fc32.noarch @@System
Upgrade NetworkManager-libnm-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-libnm-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade NetworkManager-ppp-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-ppp-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade NetworkManager-team-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-team-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade NetworkManager-wifi-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-wifi-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade NetworkManager-wwan-1:1.22.0-0.2.fc32.x86_64 @rawhide
Upgraded NetworkManager-wwan-1:1.22.0-0.1.fc32.x86_64 @@System
Upgrade alsa-lib-1.2.1.2-1.fc32.x86_64 @rawhide
Upgraded alsa-lib-1.2.1.1-1.fc32.x86_64 @@System
Upgrade alsa-lib-devel-1.2.1.2-1.fc32.x86_64 @rawhide
Upgraded alsa-lib-devel-1.2.1.1-1.fc32.x86_64 @@System
Upgrade alsa-ucm-1.2.1.2-1.fc32.noarch @rawhide
Upgraded alsa-ucm-1.2.1.1-1.fc32.noarch @@System
Upgrade conmon-2:2.0.4-0.2.dev.git42bce45.fc32.x86_64 @rawhide
Upgraded conmon-2:2.0.4-0.1.dev.gitf6d23b5.fc32.x86_64 @@System
Upgrade container-selinux-2:2.123.0-0.3.dev.git0b25a4a.fc32.noarch @rawhide
Upgraded container-selinux-2:2.123.0-0.1.dev.git661a904.fc32.noarch @@System
Upgrade cups-1:2.3.0-2.fc32.x86_64 @rawhide
Upgraded cups-1:2.3.0-1.fc32.x86_64 @@System
Upgrade cups-client-1:2.3.0-2.fc32.x86_64 @rawhide
Upgraded cups-client-1:2.3.0-1.fc32.x86_64 @@System
Upgrade cups-filesystem-1:2.3.0-2.fc32.noarch @rawhide
Upgraded cups-filesystem-1:2.3.0-1.fc32.noarch @@System
Upgrade cups-ipptool-1:2.3.0-2.fc32.x86_64 @rawhide
Upgraded cups-ipptool-1:2.3.0-1.fc32.x86_64 @@System
Upgrade cups-libs-1:2.3.0-2.fc32.x86_64 @rawhide
Upgraded cups-libs-1:2.3.0-1.fc32.x86_64 @@System
Upgrade fuse-overlayfs-0.7.2-2.0.dev.git8c59873.fc32.x86_64 @rawhide
Upgraded fuse-overlayfs-0.7.1-2.0.dev.gite0d2ffa.fc32.x86_64 @@System
Upgrade fwupd-1.3.5-1.fc32.x86_64 @rawhide
Upgraded fwupd-1.3.4-1.fc32.x86_64 @@System
Upgrade glibc-2.30.9000-21.fc32.x86_64 @rawhide
Upgraded glibc-2.30.9000-20.fc32.x86_64 @@System
Upgrade glibc-all-langpacks-2.30.9000-21.fc32.x86_64 @rawhide
Upgraded glibc-all-langpacks-2.30.9000-20.fc32.x86_64 @@System
Upgrade glibc-common-2.30.9000-21.fc32.x86_64 @rawhide
Upgraded glibc-common-2.30.9000-20.fc32.x86_64 @@System
Upgrade glibc-devel-2.30.9000-21.fc32.x86_64 @rawhide
Upgraded glibc-devel-2.30.9000-20.fc32.x86_64 @@System
Upgrade glibc-headers-2.30.9000-21.fc32.x86_64 @rawhide
Upgraded glibc-headers-2.30.9000-20.fc32.x86_64 @@System
Upgrade glibc-langpack-en-2.30.9000-21.fc32.x86_64 @rawhide
Upgraded glibc-langpack-en-2.30.9000-20.fc32.x86_64 @@System
Upgrade glibc-static-2.30.9000-21.fc32.x86_64 @rawhide
Upgraded glibc-static-2.30.9000-20.fc32.x86_64 @@System
Upgrade gnome-boxes-3.35.2-1.fc32.x86_64 @rawhide
Upgraded gnome-boxes-3.34.1-1.fc32.x86_64 @@System
Upgrade grubby-8.40-38.fc32.x86_64 @rawhide
Upgraded grubby-8.40-37.fc32.x86_64 @@System
Upgrade json-c-0.13.1-8.fc32.x86_64 @rawhide
Upgraded json-c-0.13.1-7.fc32.x86_64 @@System
Upgrade libmbim-1.20.2-1.fc32.x86_64 @rawhide
Upgraded libmbim-1.20.0-1.fc32.x86_64 @@System
Upgrade libmbim-utils-1.20.2-1.fc32.x86_64 @rawhide
Upgraded libmbim-utils-1.20.0-1.fc32.x86_64 @@System
Upgrade libosinfo-1.7.0-1.fc32.x86_64 @rawhide
Upgraded libosinfo-1.6.0-2.fc32.x86_64 @@System
Upgrade libtracker-sparql-2.3.1-2.fc32.x86_64 @rawhide
Upgraded libtracker-sparql-2.3.0-2.fc32.x86_64 @@System
Upgrade lua-json-1.3.2-13.fc32.noarch @rawhide
Upgraded lua-json-1.3.2-12.fc31.noarch @@System
Upgrade lua-socket-3.0-0.21.rc1.fc32.x86_64 @rawhide
Upgraded lua-socket-3.0-0.20.rc1.fc31.x86_64 @@System
Upgrade osinfo-db-tools-1.7.0-1.fc32.x86_64 @rawhide
Upgraded osinfo-db-tools-1.6.0-1.fc31.x86_64 @@System
Upgrade perl-Errno-1.30-449.fc32.x86_64 @rawhide
Upgraded perl-Errno-1.30-448.fc32.x86_64 @@System
Upgrade perl-IO-1.40-449.fc32.x86_64 @rawhide
Upgraded perl-IO-1.40-448.fc32.x86_64 @@System
Upgrade perl-interpreter-4:5.30.1-449.fc32.x86_64 @rawhide
Upgraded perl-interpreter-4:5.30.1-448.fc32.x86_64 @@System
Upgrade perl-libs-4:5.30.1-449.fc32.x86_64 @rawhide
Upgraded perl-libs-4:5.30.1-448.fc32.x86_64 @@System
Upgrade perl-macros-4:5.30.1-449.fc32.x86_64 @rawhide
Upgraded perl-macros-4:5.30.1-448.fc32.x86_64 @@System
Upgrade systemd-244-1.fc32.x86_64 @rawhide
Upgraded systemd-244~rc1-1.fc32.x86_64 @@System
Upgrade systemd-container-244-1.fc32.x86_64 @rawhide
Upgraded systemd-container-244~rc1-1.fc32.x86_64 @@System
Upgrade systemd-devel-244-1.fc32.x86_64 @rawhide
Upgraded systemd-devel-244~rc1-1.fc32.x86_64 @@System
Upgrade systemd-libs-244-1.fc32.x86_64 @rawhide
Upgraded systemd-libs-244~rc1-1.fc32.x86_64 @@System
Upgrade systemd-pam-244-1.fc32.x86_64 @rawhide
Upgraded systemd-pam-244~rc1-1.fc32.x86_64 @@System
Upgrade systemd-rpm-macros-244-1.fc32.noarch @rawhide
Upgraded systemd-rpm-macros-244~rc1-1.fc32.noarch @@System
Upgrade systemd-udev-244-1.fc32.x86_64 @rawhide
Upgraded systemd-udev-244~rc1-1.fc32.x86_64 @@System
Upgrade tracker-2.3.1-2.fc32.x86_64 @rawhide
Upgraded tracker-2.3.0-2.fc32.x86_64 @@System
Upgrade tracker-miners-2.3.1-2.fc32.x86_64 @rawhide
Upgraded tracker-miners-2.3.0-2.fc32.x86_64 @@System
Upgrade usb_modeswitch-data-20191128-1.fc32.noarch @rawhide
Upgraded usb_modeswitch-data-20170806-5.fc31.noarch @@System
Upgrade webkit2gtk3-2.27.3-1.fc32.x86_64 @rawhide
Upgraded webkit2gtk3-2.27.2-2.fc32.x86_64 @@System
Upgrade webkit2gtk3-devel-2.27.3-1.fc32.x86_64 @rawhide
Upgraded webkit2gtk3-devel-2.27.2-2.fc32.x86_64 @@System
Upgrade webkit2gtk3-jsc-2.27.3-1.fc32.x86_64 @rawhide
Upgraded webkit2gtk3-jsc-2.27.2-2.fc32.x86_64 @@System
Upgrade webkit2gtk3-jsc-devel-2.27.3-1.fc32.x86_64 @rawhide
Upgraded webkit2gtk3-jsc-devel-2.27.2-2.fc32.x86_64 @@System
|
||
Comment 7•4 years ago
|
||
Run with MOZ_SANDBOX_LOGGING=1 and see if anything interesting turns up?
|
|
Reporter | |
Comment 8•4 years ago
|
||
|
|
Reporter | |
Comment 9•4 years ago
|
||
|
Assignee | |
Comment 10•4 years ago
|
||
For reference, the process is chrooted to a deleted directory (if the distro allows unprivileged user namespaces), so short-circuiting the file broker hooks with Allow() means you get the real syscalls for filesystem access, which fail because there's no filesystem. That can be turned off along with the rest of the namespace usage by setting MOZ_ASSUME_USER_NS=0, but there's no setting to control it independently short of editing the source.
Also, from discussion in chat, it looks like content sandbox level 1 is broken again (probably by me when rearranging things for the RDD sandbox.) Probably we should either test it in CI somehow or get rid of it.
|
|
Reporter | |
Comment 11•4 years ago
|
||
So I verified that the last syscall that we try to do is the /proc/sys/crypto/fips_enabled syscall that gets blocked.
I tried unblocking it and it didn't help:
diff --git a/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp b/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp
--- a/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp
+++ b/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp
@@ -272,16 +272,17 @@ SandboxBrokerPolicyFactory::SandboxBroke
// Bug 1575985: WASM library sandbox needs RW access to /dev/null
policy->AddPath(rdwr, "/dev/null");
// Read permissions
policy->AddPath(rdonly, "/dev/urandom");
policy->AddPath(rdonly, "/proc/cpuinfo");
policy->AddPath(rdonly, "/proc/meminfo");
+ policy->AddPath(rdonly, "/proc/sys/crypto/fips_enabled");
policy->AddDir(rdonly, "/sys/devices/cpu");
policy->AddDir(rdonly, "/sys/devices/system/cpu");
policy->AddDir(rdonly, "/lib");
policy->AddDir(rdonly, "/lib64");
policy->AddDir(rdonly, "/usr/lib");
policy->AddDir(rdonly, "/usr/lib32");
policy->AddDir(rdonly, "/usr/lib64");
policy->AddDir(rdonly, "/etc");
The process is killed by a SIGSYS one, and from talking with :jld, it seems like something is blocking the signal handler and resetting it to the default handler.
It seems we dlopen nss or something, and this recent glibc commit resets the signal handler from the sandbox. Ugh
|
|
Reporter | |
Comment 12•4 years ago
|
||
Just to confirm, what happens here is that we try to load a DLL after the sandbox is initialized, and that triggers the sigprogmask, then the process dies. From strace:
[pid 77711] rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
[pid 77711] openat(AT_FDCWD, "/home/emilio/src/moz/mozilla-central/obj-debug/dist/bin/libsoftokn3.so", O_RDONLY|O_CLOEXEC) = 257
[pid 77711] --- SIGSYS {si_signo=SIGSYS, si_code=SYS_SECCOMP, si_errno=EXDEV, si_call_addr=0x7f70b6950988, si_syscall=__NR_openat, si_arch=AUDIT_ARCH_X86_64} ---
[pid 77711] +++ killed by SIGSYS (core dumped) +++
|
|
||
Comment 13•4 years ago
|
||
As per request of the glibc patch author, I sent an email to libc-alpha detailing the fallout from this: https://sourceware.org/ml/libc-alpha/2019-12/msg00079.html
|
|
Assignee | |
Comment 14•4 years ago
|
||
I reproduced the rr failure and filed a bug: https://github.com/mozilla/rr/issues/2413
|
||
Comment 15•4 years ago
|
||
Gian-Carlo, a quick note since I'm not subscribed to the libc-alpha list - one of the RHBZ dupes does claim that this also affects Chrome/Chromium: "This is also seen with chrome. using --disable-seccomp-filter-sandbox fixes this issue on google chrome." https://bugzilla.redhat.com/show_bug.cgi?id=1778559#c0
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Comment 16•4 years ago
|
||
The Chrome failure mentioned on the RedHat bug appears to be Chromium bug 1025739 which is their version of our bug 1597792; I commented about it over there as well.
I also searched the Chromium bug tracker for sandboxing bugs newer than the glibc patch and I don't see anything related, although it may have been masked by would-be users running into the nanosleep bug first.
|
|
Assignee | |
Comment 17•4 years ago
|
||
We were considering having the seccomp filter program intercept sigprocmask unless it comes from a specific instance of the syscall instruction, by inspecting the instruction pointer value from the trap. The Chromium code we're using has support for this, but it's tied to a flag that makes the entire policy non-enforcing (to allow everything but log would-be failures), and it's blocked from being used in non-debug builds for safety.
It turns out to be relatively simple to extend bpf_dsl with the ability to test the instruction pointer, and I've done this locally, but while experimenting with it I noticed something important: returning from the SIGSYS handler restores the signal context. Which means that calling the real sigprocmask in a trap function gets reverted, but also that we should be able to just block sigprocmask completely and emulate it by changing the mask passed to sigreturn.
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Comment 18•4 years ago
|
||
glibc has landed a workaround that will make Firefox stop crashing (see also comments on the RedHat bug confirming it works), but rough consensus on the mailing list is that what we're doing with SECCOMP_RET_TRAP isn't sustainable in the longer term, and there's a new feature (SECCOMP_RET_USER_NOTIF in kernel 5.0) that we can and should migrate to (with fallback to using SECCOMP_RET_TRAP for older kernels). I think I'll file a separate bug for that work and then close this one, because this particular failure is effectively fixed and the information in the comments has served its purpose.
|
||
Comment 19•4 years ago
|
||
Just to be clear here, the signal blocking in dlopen has not been removed from the glibc master branch yet.
|
|
Assignee | |
Comment 20•4 years ago
|
||
I'll leave this bug open until the patches I linked in comment #18 are merged into glibc (vs. just applied locally in Fedora).
|
||
Comment 21•4 years ago
|
||
These landed in glibc on December 13 (a day after the last comment) as:
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=f8ed116aa574435c6e28260f21963233682d3b57
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=f7649d5780aa4682393b9daedd653e4d9c12784c
And were part of the glibc 2.31 stable tag.
|
|
Assignee | |
Updated•4 years ago
|
Description
•