1531030 - IOInterposer detours racing with launcher process background thread

Status: RESOLVED FIXED

(Core :: XPCOM, defect, P1)

Description

[Calixte Denizet (:calixte)]

This bug is for crash report bp-26d1c33c-a516-476c-b759-47c240190227.

Top 10 frames of crashing thread:

0  @0x246 
1 kernelbase.dll GetFileAttributesExW 
2 wintrust.dll SortedCatalogOpen 
3 wintrust.dll CatalogLoadSortedFileData 
4 wintrust.dll alloca_probe 
5 wintrust.dll CatUtil_NoContentMsgDecode 
6 wintrust.dll I_CertDiagControl 
7 wintrust.dll alloca_probe 
8 wintrust.dll WinVerifyTrust 
9 mozglue.dll static bool `anonymous namespace'::SignedBinary::VerifySignatureInternal mozglue/build/Authenticode.cpp:184

There is 1 crash in nightly 67 with buildid 20190226215106. In analyzing the backtrace, the regression may have been introduced by patch [1] to fix bug 1460433.

[1] https://hg.mozilla.org/mozilla-central/rev?node=7f034553129d

Comment 2

[(No longer employed by Mozilla) Aaron Klotz]

The fundamental problem here is that, when the launcher process fails, it spawns a background thread to log the failure to telemetry while allowing the main thread to continue to start as the browser process.

The IOInterposer attempts to hook NT file I/O system call stubs. Notice that all crash reports are only on x86-64. This is because we use an atomic NOP-space patch on x86, and we do not yet support these hooks on ARM64.

The launcher's background thread is attempting to issue I/O system calls concurrently with IOInterposer attempting to write detours to those functions. The launcher thread is then being sent off into the weeds.

I am clearing the regression keyword for a couple of reasons:

• Yes, these crashes appeared after landing bug 1460433, but this is because of a bad interaction between two components; I technically did not regress an existing component;
• IOInterposer is Nightly only.

Comment 3

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1531030: Use MOZ_DISABLE_POISON_IO_INTERPOSER to disable PoisonIOInterposer when it is unsafe to initialize; r?erahm

If we are running a background thread in the launcher process to log failures,
then allowing the main thread to proceed with monkeypatching system calls is a
Bad Idea. This patch gives us an environment variable that, when set, indicates
that it is unsafe for PoisonIOInterposer to run.

This scenario is an uncommon one, but one that we must account for nonetheless.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7e2fced298cb
Use MOZ_DISABLE_POISON_IO_INTERPOSER to disable PoisonIOInterposer when it is unsafe to initialize; r=erahm

Comment 5

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/7e2fced298cb

Comment 7

[Julien Cristau [:jcristau]]

Nightly only per comment 2.