1681950 - ThreadSanitizer: data race [@ __xstat64] vs. [@ __xstat64] via [@ mozilla::PreferencesWriter::Write]

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on autoland revision 18f7558b7bb0ee71dec915906de3d4efd8e11954.

For detailed crash information, see attachment.

Looks like we are trying to write preferences from two different threads in an unsafe way.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Andrew McCreight [:mccr8]]

Kris, could you please take a look? I'm not familiar with this code, but it looks like the nsIFile for preferences is being written to on multiple threads at once. Thanks.

Comment 3

[Andrew McCreight [:mccr8]]

Sounds like Gankra has already started looking at this.

Comment 4

[Aria Beingessner [:Gankra]]

Attachment: Bug 1681950 - properly serialize the PreferencesWriter's operations. r?kriswright

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/82ef6d7419cc
properly serialize the PreferencesWriter's operations. r=KrisWright

Comment 6

[Natalia Csoregi [:nataliaCs]]

Backed out for causing leaks and assertion failures

backout: https://hg.mozilla.org/integration/autoland/rev/020f33180480c9eab71b641ee1233c88d4fefe89

push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=82ef6d7419cc43ed92a7aa64f51956ee7127a145&group_state=expanded

failure log:

• Assertion failure: !mRawPtr, at /builds/worker/workspace/obj-build/dist/include/mozilla/StaticPtr.h:51
• TEST-UNEXPECTED-FAIL | leakcheck | default 44 bytes leaked (Mutex)

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6183c0cb65cc
properly serialize the PreferencesWriter's operations. r=KrisWright

Comment 8

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/6183c0cb65cc

Comment 9

[Julien Cristau [:jcristau]]

Is this something we'd want to uplift to esr78?

Comment 10

[Aria Beingessner [:Gankra]]

Comment on attachment 9192657 [details]
Bug 1681950 - properly serialize the PreferencesWriter's operations. r?kriswright

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: This reduces the rate of preference values getting corrupted, potentially resulting in crashes and other predicable behaviors.
• User impact if declined: Prefs will continue to get messed up.
• Fix Landed on Version: 85
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's a small local patch that landed in nightly 6 months ago with no issues.
• String or UUID changes made by this patch:

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9192657 [details]
Bug 1681950 - properly serialize the PreferencesWriter's operations. r?kriswright

Well-tested, high-value reliability fix. Approved for 78.12esr.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/9d6f6a2c2117