1607138 - ThreadSanitizer: data race [@ mozilla::ShutdownXPCOM] vs. [@ mozilla::ThreadEventTarget::Dispatch] on gXPCOMThreadsShutDown

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision c0a6eb95b65c.

This is a race on the non-atomic global gXPCOMThreadsShutDown. I don't know enough about our shutdown routines to tell how this should be fixed or if this is even expected. Depending on the code creating the timer in question here, making gXPCOMThreadsShutDown a (non-relaxed) atomic could solve the problem. But if the timer existed pre-shutdown and is not cancelled by the shutdown, then it could still fire randomly afterwards.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Aria Beingessner [:Gankra]]

Attachment: Bug 1607138 - Remove supression for seemingly fixed issue. r?decoder

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fb3f09394f1d
Remove supression for seemingly fixed issue. r=decoder

Comment 4

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/fb3f09394f1d

Comment 5

[Aria Beingessner [:Gankra]]

Attachment: Bug 1607138 - Backed out changeset fb3f09394f1d. r?decoder

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0508775e5be3
Backed out changeset fb3f09394f1d. r=decoder

Comment 7

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/0508775e5be3

Comment 8

[Aria Beingessner [:Gankra]]

(was supposed to be leave-open, sorry!)

Comment 9

[Kris Wright :KrisWright]

Attachment: Bug 1607138 - gXPCOMThreadsShutdown should be atomic

This commit makes gXPCOMThreadsShutdown atomic. I've deliberated on this one for a while because I was mostly interested in how timer threads may be trying to init during shutdown, but these aren't the only places where we are making accesses into gXPCOMThreadsShutdown so it should be made atomic regardless.

Comment 10

[Kris Wright :KrisWright]

Attachment: Bug 1607138 - Remove suppression for gXPCOMThreadsShutdown

This is no longer going to be detected by tsan. I figure with full shutdown enabled we should be able to see if there were other issues around this one.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4c798c8a8aaf
gXPCOMThreadsShutdown should be atomic r=xpcom-reviewers,mccr8
https://hg.mozilla.org/integration/autoland/rev/40e53357bfe1
Remove suppression for gXPCOMThreadsShutdown r=decoder

Comment 12

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/4c798c8a8aaf
https://hg.mozilla.org/mozilla-central/rev/40e53357bfe1

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.