1797438 - Firefox hang with seccomp enabled on OpenShift

Status: UNCONFIRMED

(Core :: Security: Process Sandboxing, defect, P3)

Description

[lijingmu]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0

Steps to reproduce:

Recently when I run firefox firefox-102.3.0esr on OpenShift 4.11, it hangs because seccomp: runtime/default is enabled by default.
When I enabled sandbox logging I saw Sandbox: EOF from pid

export MOZ_SANDBOX_LOGGING=1
xvfb-run firefox www.bing.com
510) RET 0x50019  // errno = 25
511) RET 0x30004  // Trap #4
512) RET 0x30003  // Trap #3
513) RET 0x30002  // Trap #2
514) RET 0x7fff0000  // Allowed
515) RET 0x30001  // Trap #1
Sandbox: using seccomp tsync
Sandbox: EOF from pid 162

When I set MOZ_DISABLE_CONTENT_SANDBOX=1, I can see more sandbox logging. e.g. Sandbox: SandboxBroker: denied op=readlink

export MOZ_DISABLE_CONTENT_SANDBOX=1
433) RET 0x50019  // errno = 25
434) RET 0x30004  // Trap #4
435) RET 0x30003  // Trap #3
436) RET 0x30002  // Trap #2
437) RET 0x7fff0000  // Allowed
438) RET 0x30001  // Trap #1
Sandbox: using seccomp tsync
Sandbox: rewriting /proc/self/exe -> /proc/567/exe
Sandbox: SandboxBroker: denied op=readlink rflags=0 perms=1 path=/proc/567/exe for pid=567
Sandbox: Failed errno -13 op readlink flags 00 path /proc/567/exe
Sandbox: Failed errno -2 op open flags 02000000 path /opt/firefox-102.3.0esr/libavcodec.so.59
Sandbox: Failed errno -2 op open flags 02000000 path /lib64/glibc-hwcaps/x86-64-v3/libavcodec.so.59

Actual results:

It seems only works when I disabled seccomp: runtime/default or I use export MOZ_DISABLE_CONTENT_SANDBOX=1 for firefox on OpenShift 4.11.
Anyway we can debug or any configurations required for secure computing mode?

Expected results:

works on OpenShift 4.11 with default secure computing mode

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Security: Process Sandboxing' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[lijingmu]

I found FF will not hang if I set sandbox level to 3 by options.setPreference('security.sandbox.content.level', 3); on Linux x86_64.
I am wondering how will seccomp: runtime/default affect FF's sandboxing. Without this setting FF will hang without any logs on OCP 4.11.
Is there any best practice about how to set this sandboxing level with default seccomp or how to debug/log it?

Comment 3

[Gian-Carlo Pascutto [:gcp]]

Firefox has its own sandbox which requires an extensive and carefully tuned set of seccomp rules. If OpenShift tries to apply a generic seccomp ruleset, it will most likely cause breakage, as you observed. Weakening the Firefox sandbox by changing the security.sandbox.content.level would lower the browsers' security, so I'd recommend against it - it makes no sense to break the carefully tuned Firefox policy to install a weaker and/or less compatible generic one.

The proper fix would be to disable/remove OpenShift's seccomp policy and let Firefox install the proper one. This seems to be how OpenShift works by default anwyay, from their docs, so I'm not sure how to interpret runtime/default is enabled by default...works on OpenShift 4.11 with default secure computing mode. Is "default secure computing mode" a setting that is enabled in OpenShift, or do they ship Firefox broken by default?

I'm tempted to close this bug as INVALID, as what you are seeing is fully expected.

Comment 4

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I don't think it is expected for this to break in general — seccomp-bpf has well-defined rules for combining policies — but it is true that external seccomp-bpf policies can cause problems if they block syscalls we're trying to use, and especially if they do so using SECCOMP_RET_KILL* instead of SECCOMP_RET_ERRNO.

What I can see from the logs is that the content process is immediately killed (or exits) after starting the sandbox, with no other logging; the EOF from pid %d is from the parent process when it closes the socketpair used for the file broker. But also, the second log shows another process type (probably RDD given that it's using the file broker and looking for libavcodec) which does work. And it's reported that turning off the content sandbox or lowering the level (to 3, which means not blocking access to sockets/networking) avoids the bug.

This doesn't make a lot of sense; if the RDD process were also broken then that would suggest that it's something about using namespaces and/or chrooting the process, because that's the big difference between content sandbox levels 3 and 4, but we also do that for RDD and it seems to work.

Is there anything in the system log (dmesg) when this happens? From the kernel source (the seccomp_log function) it looks like kill_thread and kill_process are logged by default, and that should include the syscall number and the filter return value.

It would also help if I knew where the source code for OpenShift's seccomp policy is, because I could just look through it for the “interesting” syscalls I know we use, but in any case I suspect this is going to end up as something that would have to be changed in the policy and not in Firefox.

Comment 5

[lijingmu]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #3)

Firefox has its own sandbox which requires an extensive and carefully tuned set of seccomp rules. If OpenShift tries to apply a generic seccomp ruleset, it will most likely cause breakage, as you observed. Weakening the Firefox sandbox by changing the security.sandbox.content.level would lower the browsers' security, so I'd recommend against it - it makes no sense to break the carefully tuned Firefox policy to install a weaker and/or less compatible generic one.

The proper fix would be to disable/remove OpenShift's seccomp policy and let Firefox install the proper one. This seems to be how OpenShift works by default anwyay, from their docs, so I'm not sure how to interpret runtime/default is enabled by default...works on OpenShift 4.11 with default secure computing mode. Is "default secure computing mode" a setting that is enabled in OpenShift, or do they ship Firefox broken by default?

I'm tempted to close this bug as INVALID, as what you are seeing is fully expected.

Thanks for your clarification. OpenShift Container Platform ships with a default seccomp profile that is referenced as runtime/default since version 4.11. I can see Seccomp: 2 with grep Seccomp /proc/1/status in the container. I found FF will hang when running in container on OpenShift 4.11 until I downgrade security.sandbox.content.level. OCP also allows users to custom seccomp profile by providing their own json profile, but as you said FF requires an extensive and carefully tuned set of seccomp rules. I do not know what should be in the profile.
https://access.redhat.com/documentation/en-us/openshift_container_platform/4.11/html/security_and_compliance/seccomp-profiles

Comment 6

[lijingmu]

(In reply to Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧ from comment #4)

I don't think it is expected for this to break in general — seccomp-bpf has well-defined rules for combining policies — but it is true that external seccomp-bpf policies can cause problems if they block syscalls we're trying to use, and especially if they do so using SECCOMP_RET_KILL* instead of SECCOMP_RET_ERRNO.

What I can see from the logs is that the content process is immediately killed (or exits) after starting the sandbox, with no other logging; the EOF from pid %d is from the parent process when it closes the socketpair used for the file broker. But also, the second log shows another process type (probably RDD given that it's using the file broker and looking for libavcodec) which does work. And it's reported that turning off the content sandbox or lowering the level (to 3, which means not blocking access to sockets/networking) avoids the bug.

This doesn't make a lot of sense; if the RDD process were also broken then that would suggest that it's something about using namespaces and/or chrooting the process, because that's the big difference between content sandbox levels 3 and 4, but we also do that for RDD and it seems to work.

Is there anything in the system log (dmesg) when this happens? From the kernel source (the seccomp_log function) it looks like kill_thread and kill_process are logged by default, and that should include the syscall number and the filter return value. Now I can only see I can see Seccomp: 2 with grep Seccomp /proc/1/status in the container.

It would also help if I knew where the source code for OpenShift's seccomp policy is, because I could just look through it for the “interesting” syscalls I know we use, but in any case I suspect this is going to end up as something that would have to be changed in the policy and not in Firefox.

Thanks for your analysis and clues, I think we are close to the truth. I did not find the logs you mentioned. I am going to give your analysis to OCP and ask about whether they have extra seccomp-bpf policies or blocked syscalls.