1652300 - ThreadSanitizer: data race [@ mozilla::net::CacheEntry::InvokeAvailableCallback] vs. [@ mozilla::net::CacheEntry::OnHandleClosed]

Status: RESOLVED FIXED

(Core :: Networking: Cache, defect, P2)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on autoland revision 6468f2d9b8aec7e0dc039da130e5501f3034bb3f.

For detailed crash information, see attachment.

This looks like a read/write race on mState and it popped up intermittently on M13 when I tried to remove the suppression for bug 1607212. When fixing this, we should try to remove the suppression for that bug as well.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Honza Bambas (:mayhemer)]

We need to add a lock around here:
https://hg.mozilla.org/mozilla-central/file/6468f2d9b8aec7e0dc039da130e5501f3034bb3f/netwerk/cache2/CacheEntry.cpp#l820

Comment 3

[Honza Bambas (:mayhemer)]

Attachment: Bug 1652300 - Keep lock around mState in `CacheEntry::InvokeAvailableCallback`, r=michal

Comment 4

[Honza Bambas (:mayhemer)]

https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=WrQPErH9QTKZ9J3jQ15cNw.5&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=c630c4853a8b8beef3a991be7d8cb76f80a4fbe6

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/407a4fc5a61d
Keep lock around mState in `CacheEntry::InvokeAvailableCallback`, r=michal,necko-reviewers

Comment 6

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/407a4fc5a61d

Comment 7

[Aria Beingessner [:Gankra]]

Attachment: Bug 1652300 - Remove supression for fixed issue. r?decoder

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d17e9080e130
Remove supression for fixed issue. r=decoder

Comment 9

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/d17e9080e130