1601632 - ThreadSanitizer: data race [@ mozilla::ScriptPreloader::MaybeFinishOffThreadDecode] vs. [@ mozilla::ScriptPreloader::OffThreadDecodeCallback] on mToken

Status: RESOLVED FIXED

(Core :: XPConnect, defect, P3)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision 6989fcd6bab3.

Filing this because it is amongst the frequent races in Mochitests. This race is apparently on updating the mToken pointer. Should that be an atomic to ensure that the update is properly visible?

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Jason Orendorff [:jorendorff]]

Hmm. Looks real; ScriptPreloader::WaitForCachedScript calls MaybeFinishOffThreadDecode() three times, sometimes with mMonitor held, sometimes not, which seems fishy.

The first line of MaybeFinishOffThreadDecode reads mToken which I think has to be done with the lock held, since it is set here with the mutex held.

Kris, can you take this?

Comment 3

[Intermittent Failures Robot]

1 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1601632&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 4

[Aria Beingessner [:Gankra]]

Attachment: Bug 1601632 - Remove supression for seemingly fixed issue. r?decoder

Depends on D95128

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f14d44bacf2
Remove supression for seemingly fixed issue. r=decoder

Comment 6

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 2 changesets (bug 1601632, bug 1608068) for ScriptPreloader related tsan failures.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=C6W8jfnMS0yxJEu78h7T1A.0&fromchange=0ec69f07cdc9abf59a99619ab789c113e5e93f35&searchStr=linux%2C18.04%2Cx64%2Ctsan%2Copt%2Cmochitests%2Ctest-linux1804-64-tsan%2Fopt-mochitest-media-e10s%2Cmda3&tochange=d01cd574728d7c07dbfab85f48987f2d0a4b1fd6

Backout link: https://hg.mozilla.org/integration/autoland/rev/d01cd574728d7c07dbfab85f48987f2d0a4b1fd6

Failure log: https://treeherder.mozilla.org/logviewer?job_id=320532020&repo=autoland&lineNumber=10577

Comment 7

[Aria Beingessner [:Gankra]]

Attachment: tsan-backtrace-nov-2020.txt

roundabout triage: still an issue

Comment 8

[Ted Campbell [:tcampbell]]

Attachment: Bug 1601632 - Make ScriptPreloader::mToken updates Atomic. r?kmag!,gankro!

The mToken is used across threads without full locking so it must be marked
as atomic. At the same time, we can use exchange operations to read or
write outside of the mMonitor lock.

Depends on D99402

Comment 9

[Ted Campbell [:tcampbell]]

Attachment: Bug 1601632 - Cleanup ScriptPreloader Runnable behaviour. r?kmag!,gankro!

Add mWaitingForDecode to indicate that main-thread is blocked on the
Monitor and should be woken. This avoids generating some of the unused
Runnables. Also ensure mFinishDecodeRunnablePending is only accessed while
holding the lock.

Depends on D99403

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2497e2dd2ae8
Make ScriptPreloader::mToken updates Atomic. r=kmag,Gankro,decoder
https://hg.mozilla.org/integration/autoland/rev/9e1d4db4e5d0
Cleanup ScriptPreloader Runnable behaviour. r=kmag,Gankro,decoder

Comment 11

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/2497e2dd2ae8
https://hg.mozilla.org/mozilla-central/rev/9e1d4db4e5d0