1602313 - ThreadSanitizer: data race [@ mozilla::net::nsHttpChannel::OnCacheEntryCheck] vs. [@ ~AutoCacheWaitFlags]

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision 6989fcd6bab3.

I've investigated this report and this seems to be yet another instance of a race on adjacent bitfields. There are several adjacent members declared as bitfields in nsHttpChannel:

https://searchfox.org/mozilla-central/rev/d24696b5abaf9fb75f7985952eab50d5f4ed52ac/netwerk/protocol/http/nsHttpChannel.h#659-736

The attached read/write race is a race on separate fields (mCachedContentIsValid vs. mCacheEntriesToWaitFor) but it is not race-free to access these from different threads because the compiler can optimize reads and writes to these bitfields into larger chunks, touching more fields than just the updated one. In this particular case, the race might be benign because one thread is only reading, however, if any of these fields were to be written in a write/write race, then this can lead to wrong results in the fields.

@Necko Team, do you know if these fields can also be written by different threads at the same time? In any case, I suggest that we fix this somehow and make people aware about this property of bitfield members.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Dragana Damjanovic [:dragana]]

Honza, can you take a look?

Comment 3

[Honza Bambas (:mayhemer)]

(In reply to Dragana Damjanovic [:dragana] from comment #2)

Honza, can you take a look?

Yes. This needs to either be separated to a full bool member or synchronized (former preferred).

We write on a background (cache io) thread here:
https://hg.mozilla.org/mozilla-central/file/6989fcd6bab3/netwerk/protocol/http/nsHttpChannel.cpp#l4266

Comment 4

[Honza Bambas (:mayhemer)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9e7f4d0edcf8b9c64b6da1f18cab2b5d79cd0d8f

Comment 5

[Honza Bambas (:mayhemer)]

Attachment: Bug 1602313 - Make nsHttpChannel::mCachedContentIsValid an atomic bool, r=kershaw

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4837cc28da18
Make nsHttpChannel::mCachedContentIsValid an atomic bool, r=kershaw

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/4837cc28da18