ThreadSanitizer: data race [@ SetBit<2>] vs. [@ GetBit<2>] through [@ mozilla::xpcom::CreateInstanceImpl]
Categories
(Core :: XPCOM, defect)
Tracking
()
Tracking | Status | |
---|---|---|
firefox73 | --- | affected |
People
(Reporter: decoder, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(2 files, 2 obsolete files)
The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision d1ac49b9eb3e.
Quick Analysis: It looks like there is a race updating the bits in xpcom::gInitCalled
vs. reading them. As gInitCalled
is not atomic, this could cause the respective one-time initializer function in CallInitFunc
to run twice. Depending on which of the functions is run twice, this could cause problems or be benign.
In this particular case, the function that could be called twice is nsNetStartup()
, which calls nsStandardURL::InitGlobalObjects()
, which is also not race-free according to
Normally I would recommend making gInitCalled
atomic, but this looks like a hot code path and we should probably figure out first if this would cause a performance regression (that said, Atomic<bool, Relaxed>
has nearly no performance overhead in reality).
In a separate Necko bug, we should probably also make nsStandardURL::gInitialized
atomic.
General information about TSan reports
Why fix races?
Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.
Rating
If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.
False Positives / Benign Races
Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].
[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf
Suppressing unfixable races
If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp
. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.
Reporter | ||
Comment 1•4 years ago
|
||
Comment 2•4 years ago
|
||
Triage: still an issue
Updated•3 years ago
|
Comment 3•3 years ago
|
||
This resolves the races on those emulated bitfields.
To be honest this is probably overkill because there's currently only 7 components,
so all of this is just saving 56 statically allocated bytes, but hey it preserves
the design.
This creates a minor usability regression: MOZ_ATOMIC_BITFIELDS requires you
to specify the backing storage type's width (8, 16, 32, or 64), so if we ever
need 65+ components it will fall over.
Comment 4•3 years ago
|
||
Summary of discussion with some xpcom folks:
A) Yes the fact that these bitfields are racey is a problem, and these fields should be made atomic, but we'd rather just use Atomic bools for simplicitly (memory savings isn't significant since there's very few components).
B) More fundamentally, kmag believes components aren't actually supposed to be initialized off-main-thread. Currently, this is only a problem with one component (nsNetStartup?), as all others are always used only on the main thread.
We would like the netwerk team to look into finding a way to ensure that their component is always initialized on the main thread before it's accessed off-main-thread.
C) When (B) is sorted out, we should make it a hard runtime error to access the component off-main-thread when gInitCalled = false
.
Valentin, would you be able to get someone looking into (B)?
Updated•3 years ago
|
Comment 5•3 years ago
|
||
But also make it illegal to do the initialization off-main-thread.
Updated•3 years ago
|
Comment 6•3 years ago
|
||
Above is the implementation of (A) and (C).
(In reply to Alexis Beingessner [:Gankra] from comment #4)
B) More fundamentally, kmag believes components aren't actually supposed to be initialized off-main-thread. Currently, this is only a problem with one component (nsNetStartup?), as all others are always used only on the main thread.
That component really should not be initialized on the main thread. In fact, we assert the opposite
We would like the netwerk team to look into finding a way to ensure that their component is always initialized on the main thread before it's accessed off-main-thread.
C) When (B) is sorted out, we should make it a hard runtime error to access the component off-main-thread when
gInitCalled = false
.Valentin, would you be able to get someone looking into (B)?
I think moving the initialization earlier in startup is the right call here. I'll file a bug for that.
Comment 8•3 years ago
|
||
There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:Gankra, could you have a look please?
For more information, please visit auto_nag documentation.
Comment 9•3 years ago
|
||
(In reply to Valentin Gosu [:valentin] (he/him) from comment #7)
(In reply to Alexis Beingessner [:Gankra] from comment #4)
B) More fundamentally, kmag believes components aren't actually supposed to be initialized off-main-thread. Currently, this is only a problem with one component (nsNetStartup?), as all others are always used only on the main thread.
That component really should not be initialized on the main thread. In fact, we assert the opposite
We definitely assert that it's initialized on the main thread. Even if we didn't assert we were on the main thread there, the IDN service constructor asserts that it is only called on the main thread, which it definitely needs to be, because it uses preferences code which is not threadsafe.
Comment 10•3 years ago
|
||
Unassigning myself as it seems the remaining work is kmag and valentin needing to come to an agreement on what should happen.
Updated•2 years ago
|
Comment 11•2 years ago
|
||
In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.
Comment 12•11 months ago
|
||
Did bug 1697164 solve this issue already, by chance?
Comment 13•11 months ago
|
||
(In reply to Jens Stutte [:jstutte] from comment #12)
Did bug 1697164 solve this issue already, by chance?
I don't know. It may have, but the fact that there was ever a data race when the only place that should access the value off thread is an assertion that it has already been set to true on the main thread makes me anxious. That means that the assertion code is probably racing with the initialization code, which isn't good. And from my memory, that could mean we wind up running the initialization code off main thread, which would be bad, since it isn't threadsafe.
Updated•11 months ago
|
Description
•