1756857 - Make the sandbox not crash inside of libsanitizer's crash handler on UBSan builds

Status: NEW

(Core :: Security: Process Sandboxing, defect, P3)

Description

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

UBSan seems to mostly coexist with sandboxing, but one problem has been found (reported by karlt on Matrix): the sanitizer crash handler, used if the program crashes for normal reasons and probably also for UBSan faults, tries to fork to run a symbolizer for the crash stack. Normally on Nightly/m-c builds we crash on unexpected syscalls in order to get a report with the call stack, but in this case we expect it and we should return failure. (In general, we don't want a SIGSEGV handler to end up in the SIGSYS handler, because if the latter crashes then we have a problem.)

One option is to return an error for fork on UBSan builds and otherwise maintain the current behavior, but we could also turn off the crash-on-error behavior (controllable via the variable gSandboxCrashOnError and the env var `MOZ_SANDBOX_CRASH_ON_ERROR) entirely on those builds. If we do the former, we'd want to have some kind of test coverage to make sure that there aren't regressions and to try to ensure that there isn't some other exceptional case we're not handling correctly, and I don't know if we do.