Closed Bug 1608462 Opened 1 year ago Closed 5 months ago

ThreadSanitizer: lock-order-inversion (potential deadlock) [@ mozilla::detail::MutexImpl::lock] through [@ mozilla::ScriptPreloader::OffThreadDecodeCallback]

Categories

(Core :: JavaScript Engine, defect, P3)

x86_64
Linux
defect

Tracking

()

RESOLVED FIXED
86 Branch
Tracking Status
firefox74 --- wontfix
firefox86 --- fixed

People

(Reporter: decoder, Assigned: tcampbell)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision 12fb5e522dd3.

This looks different to the profiler-related deadlock that I filed the other day, even though both of these are related to off-thread compilation.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Based on the stack traces, my blind guess would be that this could cause content processes to dead-lock.

Flags: needinfo?(kmaglione+bmo)
Priority: -- → P1
See Also: → 1615265

Neha, do you know who would be the best person (timewise) to fix issues with the ScriptPreloader (this issue and Bug 1615265)?

Flags: needinfo?(nkochar)

Maybe Chris can help.

Flags: needinfo?(nkochar) → needinfo?(cpeterson)

(In reply to Nicolas B. Pierron [:nbp] from comment #3)

Neha, do you know who would be the best person (timewise) to fix issues with the ScriptPreloader (this issue and Bug 1615265)?

I'll send this bug to Fission triage since three XPConnect peers will be there to discuss.

Fission Milestone: --- → ?
Flags: needinfo?(cpeterson)

kmag talked with decoder about this bug. kmag doesn't think this lock inversion can cause a real deadlock, so we can suppress the warning.

P3

Fission Milestone: ? → ---
Priority: P1 → P3

Triage: still an issue.

Is it possible that this is another instance of https://github.com/google/sanitizers/issues/488 ? We've run into this with monitors before -- see Bug 1651770 for an example.

Looks like kmag's response is in comment 6 so canceling the NI.

Flags: needinfo?(kmaglione+bmo)

(In reply to Alexis Beingessner [:Gankra] from comment #7)

Is it possible that this is another instance of https://github.com/google/sanitizers/issues/488 ? We've run into this with monitors before -- see Bug 1651770 for an example.

No, that bug requires both Mutexes to be taken on the same thread. Here, two different threads are involved, so it cannot be this particular problem.

(In reply to Chris Peterson [:cpeterson] from comment #6)

kmag talked with decoder about this bug. kmag doesn't think this lock inversion can cause a real deadlock, so we can suppress the warning.

P3

I don't remember talking about this and it would be best to document this in the bug. Also, given what we have learned in the last months about these types of issues, we should not suppress but fix this, even if we believe that it currently doesn't cause problems. For two reasons: 1) We have seen in other bugs, that our analysis for benign issues can easily be wrong and 2) Any kind of refactoring might change the conditions here and turn it into a deadlock after all.

Kris, I know you are quite busy, but if you find a spare minute, can you explain here why this isn't an issue right now and what it would take to fix? Also CCing Andrew in case he wants to take a look.

Flags: needinfo?(kmaglione+bmo)

Be consistent about always calling MaybeFinishOffThreadDecode without
holding the lock to simplify code. This lets us remove a TSAN deadlock
exception.

Assignee: nobody → tcampbell
Status: NEW → ASSIGNED
Pushed by tcampbell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/99f2216bb7eb
Simplify locking for ScriptPreloader::MaybeFinishOffThreadDecode. r=kmag,Gankro,decoder
Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch
Flags: needinfo?(kmaglione+bmo)
Duplicate of this bug: 1637878
You need to log in before you can comment on or make changes to this bug.