ThreadSanitizer: lock-order-inversion (potential deadlock) [@ mozilla::detail::MutexImpl::lock] through [@ mozilla::ScriptPreloader::OffThreadDecodeCallback]
Categories
(Core :: JavaScript Engine, defect, P3)
Tracking
()
People
(Reporter: decoder, Assigned: tcampbell)
References
(Blocks 1 open bug)
Details
Attachments
(3 files)
The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision 12fb5e522dd3.
This looks different to the profiler-related deadlock that I filed the other day, even though both of these are related to off-thread compilation.
General information about TSan reports
Why fix races?
Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.
Rating
If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.
False Positives / Benign Races
Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].
[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf
Suppressing unfixable races
If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.
|
Reporter | |
Comment 1•4 years ago
|
||
|
||
Comment 2•4 years ago
|
||
Based on the stack traces, my blind guess would be that this could cause content processes to dead-lock.
|
|
||
Comment 3•4 years ago
|
||
Neha, do you know who would be the best person (timewise) to fix issues with the ScriptPreloader (this issue and Bug 1615265)?
|
||
Comment 5•4 years ago
|
||
(In reply to Nicolas B. Pierron [:nbp] from comment #3)
Neha, do you know who would be the best person (timewise) to fix issues with the ScriptPreloader (this issue and Bug 1615265)?
I'll send this bug to Fission triage since three XPConnect peers will be there to discuss.
|
|
||
Comment 6•4 years ago
|
||
kmag talked with decoder about this bug. kmag doesn't think this lock inversion can cause a real deadlock, so we can suppress the warning.
P3
|
||
Comment 7•4 years ago
|
||
Triage: still an issue.
Is it possible that this is another instance of https://github.com/google/sanitizers/issues/488 ? We've run into this with monitors before -- see Bug 1651770 for an example.
|
||
Comment 8•4 years ago
|
||
Looks like kmag's response is in comment 6 so canceling the NI.
|
|
Reporter | |
Comment 9•4 years ago
|
||
(In reply to Alexis Beingessner [:Gankra] from comment #7)
Is it possible that this is another instance of https://github.com/google/sanitizers/issues/488 ? We've run into this with monitors before -- see Bug 1651770 for an example.
No, that bug requires both Mutexes to be taken on the same thread. Here, two different threads are involved, so it cannot be this particular problem.
|
|
Reporter | |
Comment 10•4 years ago
|
||
(In reply to Chris Peterson [:cpeterson] from comment #6)
kmag talked with decoder about this bug. kmag doesn't think this lock inversion can cause a real deadlock, so we can suppress the warning.
P3
I don't remember talking about this and it would be best to document this in the bug. Also, given what we have learned in the last months about these types of issues, we should not suppress but fix this, even if we believe that it currently doesn't cause problems. For two reasons: 1) We have seen in other bugs, that our analysis for benign issues can easily be wrong and 2) Any kind of refactoring might change the conditions here and turn it into a deadlock after all.
Kris, I know you are quite busy, but if you find a spare minute, can you explain here why this isn't an issue right now and what it would take to fix? Also CCing Andrew in case he wants to take a look.
|
Assignee | |
Comment 11•3 years ago
|
||
Be consistent about always calling MaybeFinishOffThreadDecode without
holding the lock to simplify code. This lets us remove a TSAN deadlock
exception.
|
||
Updated•3 years ago
|
|
||
Comment 12•3 years ago
|
||
Pushed by tcampbell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/99f2216bb7eb Simplify locking for ScriptPreloader::MaybeFinishOffThreadDecode. r=kmag,Gankro,decoder
|
||
Comment 13•3 years ago
|
||
| bugherder | ||
|
||
Updated•3 years ago
|
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Description
•