1607212 - ThreadSanitizer: data race [@ mozilla::net::CacheEntry::InvokeCallback] vs. [@ mozilla::net::CacheEntry::Purge]

Status: RESOLVED DUPLICATE of bug 1615121

(Core :: Networking, defect, P2)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision c0a6eb95b65c.

This is very likely a benign race. Looking at the comments in the affected code, the race on mState should not be an issue because it transitions from READY to REVALIDATING and the code racing with this does not care about either of these states. I am concerned about suppressing this entire function though, because it means we don't find any other races related to the CacheEntry callbacks.

From a correctness standpoint, if mState is shared across threads, then it should be an atomic though, otherwise the code can be optimized to not see the updates performed to that variable in some cases. Depending on the order assumptions of this code, it might even have to be stricter than relaxed to avoid reordering, which could be the source of subtle bugs.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Andy Grover [:grover]]

Michal maybe this is easy to correct?

Comment 3

[Michal Novotny [:michal]]

Honza knows this part of the cache better

Comment 4

[Honza Bambas (:mayhemer)]

I think we should add a lock over CacheEntry::Purge (or parts of it). Making mState an atomic is a total overkill with a bad perf impact. I decided to use a lock instead of atomics, because the code is already complex enough and we need to move several members based on state transitions where other threads need to see a consistent state of all the affected members.

Comment 5

[Honza Bambas (:mayhemer)]

https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=G8zG6115RUmUNF2FtM4VvQ.1&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=b3a6bca7157b8b18bddcd3c19b37a424d439efcb

Comment 7

[Honza Bambas (:mayhemer)]

We should update https://searchfox.org/mozilla-central/rev/31d8600b73dc85b4cdbabf45ac3f1a9c11700d8e/mozglue/build/TsanOptions.cpp#181-182

Comment 8

[Christian Holler (:decoder)]

(In reply to Honza Bambas (:mayhemer) from comment #7)

We should update https://searchfox.org/mozilla-central/rev/31d8600b73dc85b4cdbabf45ac3f1a9c11700d8e/mozglue/build/TsanOptions.cpp#181-182

Making a try-run for this now and will push a patch to this bug. Leaving needinfo for that purpose.

Comment 9

[Christian Holler (:decoder)]

Attachment: Bug 1607212 - Update TSan suppressions for fixed Necko cache race. r?mayhemer

Comment 10

[Christian Holler (:decoder)]

Green on try :)

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e106c60b7be4
Update TSan suppressions for fixed Necko cache race. r=mayhemer

Comment 12

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a55b81e0bdc
Backed out changeset e106c60b7be4 for tsan fails on CacheEntry.cpp. CLOSED TREE

Comment 13

[Natalia Csoregi [:nataliaCs]]

Backed out for tsan fails on CacheEntry.cpp

backout: https://hg.mozilla.org/integration/autoland/rev/8a55b81e0bdc7fa79ef9ee96cdcfc3d1eac61566

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=6c8547a37a638627bfef90a36236edd74f5ffaf7&searchStr=linux%2C18.04%2Cx64%2Ctsan%2Copt%2Cmochitests%2Ctest-linux1804-64-tsan%2Fopt-mochitest-plain-e10s-13%2Cm%2813%29&group_state=expanded&selectedTaskRun=a7iva-8KQTWRqZMn12sRBA.0

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=309167292&repo=autoland&lineNumber=2492

Comment 14

[Honza Bambas (:mayhemer)]

Worth filing a new bug and fixing it separately.

Comment 15

[Christian Holler (:decoder)]

(In reply to Honza Bambas (:mayhemer) from comment #14)

Worth filing a new bug and fixing it separately.

I filed bug 1652300 for this and we can do the fix + suppression removal there instead.