1607142 - ThreadSanitizer: data race [@ MarkDirty] vs. [@ IsDirty]

Status: RESOLVED FIXED

(Core :: Networking: Cache, defect, P2)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision c0a6eb95b65c.

Simple race on mIsDirty in the cache. At first, this looks harmless because it will just cause the data to be purged or not purged this time from the cache. However, I noticed that the code calls CacheFile::IsWriteInProgress before trying to purge the cache. At the same time, the main thread is performing a write operation to the same cache, so I don't know if this code is even expected to race. If this race is expected, then making mIsDirty a (relaxed) atomic bool might solve the problem with near-zero performance overhead. The code explicitly says // Not using lock for performance reasons. so if this code tries to synchronize by other means to avoid concurring with a write operation, then that mechanism might be broken.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Andy Grover [:grover]]

Michal wdyt?

Comment 3

[Michal Novotny [:michal]]

I haven't checked all members but at least CacheFile::mDataIsDirty can also be set on the main thread, so the data race can happen on more members than just CacheFileMetadata::mIsDirty. I.e. the solution here is to use the lock.

Comment 4

[Michal Novotny [:michal]]

Attachment: Bug 1607142 - Hold lock in CacheFile::IsWriteInProgress to avoid data race

This method is called on cache I/O thread while the members are modified usually on the main thread which results in data race.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:grover, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/015bbebb10aa
Hold lock in CacheFile::IsWriteInProgress to avoid data race r=mayhemer

Comment 7

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/015bbebb10aa