Open Bug 1606800 Opened 4 years ago Updated 11 months ago

ThreadSanitizer: data race [@ SetBit<2>] vs. [@ GetBit<2>] through [@ mozilla::xpcom::CreateInstanceImpl]

Categories

(Core :: XPCOM, defect)

x86_64
Linux
defect

Tracking

()

Tracking Status
firefox73 --- affected

People

(Reporter: decoder, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(2 files, 2 obsolete files)

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision d1ac49b9eb3e.

Quick Analysis: It looks like there is a race updating the bits in xpcom::gInitCalled vs. reading them. As gInitCalled is not atomic, this could cause the respective one-time initializer function in CallInitFunc to run twice. Depending on which of the functions is run twice, this could cause problems or be benign.

In this particular case, the function that could be called twice is nsNetStartup(), which calls nsStandardURL::InitGlobalObjects(), which is also not race-free according to

https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/netwerk/base/nsStandardURL.cpp#58

Normally I would recommend making gInitCalled atomic, but this looks like a hot code path and we should probably figure out first if this would cause a performance regression (that said, Atomic<bool, Relaxed> has nearly no performance overhead in reality).

In a separate Necko bug, we should probably also make nsStandardURL::gInitialized atomic.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Triage: still an issue

Assignee: nobody → a.beingessner

This resolves the races on those emulated bitfields.

To be honest this is probably overkill because there's currently only 7 components,
so all of this is just saving 56 statically allocated bytes, but hey it preserves
the design.

This creates a minor usability regression: MOZ_ATOMIC_BITFIELDS requires you
to specify the backing storage type's width (8, 16, 32, or 64), so if we ever
need 65+ components it will fall over.

Summary of discussion with some xpcom folks:

A) Yes the fact that these bitfields are racey is a problem, and these fields should be made atomic, but we'd rather just use Atomic bools for simplicitly (memory savings isn't significant since there's very few components).

B) More fundamentally, kmag believes components aren't actually supposed to be initialized off-main-thread. Currently, this is only a problem with one component (nsNetStartup?), as all others are always used only on the main thread.

We would like the netwerk team to look into finding a way to ensure that their component is always initialized on the main thread before it's accessed off-main-thread.

C) When (B) is sorted out, we should make it a hard runtime error to access the component off-main-thread when gInitCalled = false.

Valentin, would you be able to get someone looking into (B)?

Assignee: a.beingessner → nobody
Flags: needinfo?(valentin.gosu)
Attachment #9200966 - Attachment is obsolete: true

But also make it illegal to do the initialization off-main-thread.

Assignee: nobody → a.beingessner
Status: NEW → ASSIGNED

Above is the implementation of (A) and (C).

(In reply to Alexis Beingessner [:Gankra] from comment #4)

B) More fundamentally, kmag believes components aren't actually supposed to be initialized off-main-thread. Currently, this is only a problem with one component (nsNetStartup?), as all others are always used only on the main thread.

That component really should not be initialized on the main thread. In fact, we assert the opposite

We would like the netwerk team to look into finding a way to ensure that their component is always initialized on the main thread before it's accessed off-main-thread.

C) When (B) is sorted out, we should make it a hard runtime error to access the component off-main-thread when gInitCalled = false.

Valentin, would you be able to get someone looking into (B)?

I think moving the initialization earlier in startup is the right call here. I'll file a bug for that.

Flags: needinfo?(valentin.gosu)

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:Gankra, could you have a look please?
For more information, please visit auto_nag documentation.

Flags: needinfo?(kmaglione+bmo)
Flags: needinfo?(a.beingessner)

(In reply to Valentin Gosu [:valentin] (he/him) from comment #7)

(In reply to Alexis Beingessner [:Gankra] from comment #4)

B) More fundamentally, kmag believes components aren't actually supposed to be initialized off-main-thread. Currently, this is only a problem with one component (nsNetStartup?), as all others are always used only on the main thread.

That component really should not be initialized on the main thread. In fact, we assert the opposite

We definitely assert that it's initialized on the main thread. Even if we didn't assert we were on the main thread there, the IDN service constructor asserts that it is only called on the main thread, which it definitely needs to be, because it uses preferences code which is not threadsafe.

Flags: needinfo?(kmaglione+bmo)

Unassigning myself as it seems the remaining work is kmag and valentin needing to come to an agreement on what should happen.

Assignee: a.beingessner → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(a.beingessner) → needinfo?(valentin.gosu)
QA Whiteboard: qa-not-actionable
Attachment #9203567 - Attachment is obsolete: true

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Severity: critical → --

Did bug 1697164 solve this issue already, by chance?

Flags: needinfo?(valentin.gosu) → needinfo?(kmaglione+bmo)

(In reply to Jens Stutte [:jstutte] from comment #12)

Did bug 1697164 solve this issue already, by chance?

I don't know. It may have, but the fact that there was ever a data race when the only place that should access the value off thread is an assertion that it has already been set to true on the main thread makes me anxious. That means that the assertion code is probably racing with the initialization code, which isn't good. And from my memory, that could mean we wind up running the initialization code off main thread, which would be bad, since it isn't threadsafe.

Flags: needinfo?(kmaglione+bmo)
Severity: -- → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: