1614637 - ThreadSanitizer: data race [@ assign_assuming_AddRef] vs. [@ operator bool] in CacheStorageService

Status: RESOLVED FIXED

(Core :: Networking: Cache, defect, P1)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision 11c9c5ce3955.

This is a race on mPurgeTimer being checked vs. it being set to nullptr. The comment at the read location seems to indicate that this might be expected, but it would still cause random behavior.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Honza Bambas (:mayhemer)]

Purpose of the if (mPurgeTimer) return; is only an optimization to not schedule the event below.

The most proper fix is to lock access over mPurgeTimer in OnMemoryConsumotionChange as well.

Comment 3

[Michal Novotny [:michal]]

Attachment: Bug 1614637 - ThreadSanitizer: data race [@ assign_assuming_AddRef] vs. [@ operator bool] in CacheStorageService

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4758bb49e763
ThreadSanitizer: data race [@ assign_assuming_AddRef] vs. [@ operator bool] in CacheStorageService r=mayhemer

Comment 5

[Alexandru Michis [:malexandru]]

Backed out changeset 4758bb49e763 for causing multiple failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/c078d08afa05139cdd28659604214a3bf79adde9

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&tochange=c078d08afa05139cdd28659604214a3bf79adde9&fromchange=437fbc225b27dd06d8b77875d16f4b680e2d8e04&selectedJob=288537786

Failure logs:

• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288535841&repo=autoland&lineNumber=21865
• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288535986&repo=autoland&lineNumber=25229
• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288535920&repo=autoland&lineNumber=7751
• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288535924&repo=autoland&lineNumber=2208
• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288537786&repo=autoland&lineNumber=1992

Comment 6

[Michal Novotny [:michal]]

The problem is that normal order of locking is CacheStorageService::mLock -> CacheEntry::mLock -> CacheFile::mLock. The patch introduces a different order and deadlock may happen:

CacheStorageService::OnMemoryConsumptionChange() << grabs CacheStorageService::mLock
CacheFileMetadata::EnsureBuffer()
CacheFileMetadata::SetElement()
CacheFile::SetElement() << holds CacheFile::mLock

Comment 7

[Honza Bambas (:mayhemer)]

Ah... yes... then we probably need a relaxed/bool atomic flag for this... (Honza not happy)

Christian, can this be easily excluded?

Comment 8

[Christian Holler (:decoder)]

Adding a runtime suppression is not recommend because we can only do that based on stacks (and that could cause other bugs to be hidden).

If you don't want to make this an atomic in general, you could make an #ifdef MOZ_TSAN change that only affects our TSan builds. In this case, I think that would be preferred over using a runtime suppression.

Comment 9

[Honza Bambas (:mayhemer)]

Thanks, makes perfect sense!

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aa36dde06adc
ThreadSanitizer: data race [@ assign_assuming_AddRef] vs. [@ operator bool] in CacheStorageService r=mayhemer

Comment 11

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/aa36dde06adc

Comment 12

[Christian Holler (:decoder)]

Can you also remove the related suppression found in mozglue/build/TsanOptions.cpp (you can search for the bug number)? You can flag me for review. Thanks!

Comment 13

[Michal Novotny [:michal]]

Attachment: Bug 1614637 - Remove suppression for data race in CacheStorageService::OnMemoryConsumptionChange()

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/013ae9b8ed68
Remove suppression for data race in CacheStorageService::OnMemoryConsumptionChange() r=decoder

Comment 15

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/013ae9b8ed68