1608068 - ThreadSanitizer: data race [@ setUnchecked] vs. [@ slotSpan]

Status: RESOLVED FIXED

(Core :: JavaScript: GC, defect, P1)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision 12fb5e522dd3.

Looks like another GC race.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Jason Orendorff [:jorendorff]]

Paul, please take a look and set a priority for this bug. Thanks.

Comment 3

[Paul Bone [:pbone]]

It looks like the shape being manipulated is also being freed and the main thread races with background finalisation. I'm more concerned that we're freeing an object that is in use (It isn't rooted, barrered or traced properly?) and that this could lead to a security exploit.

Decoder can you move this to the appropriate security group? I cannot.

Comment 4

[Jon Coppeard (:jonco)]

This is a race between the JSObject finalizer accessing the base shape of a dictionary-mode object (to get the slot span) and the base shape being changed from unowned to owned in Shape::makeOwnBaseShape. The shape and base shape are not themselves being freed.

Looking at makeOwnBaseShape it seems like slotSpan_ won't be copied across to the new base shape. Jan, do you know if that's right?

The slot span is used for memory accounting so it's not the end of the world if it's wrong.

Comment 5

[Jan de Mooij [:jandem]]

(In reply to Jon Coppeard (:jonco) from comment #4)

Looking at makeOwnBaseShape it seems like slotSpan_ won't be copied across to the new base shape. Jan, do you know if that's right?

It's a bit obscure but I think that's because the BaseShape's slotSpan_ field is only used for BaseShapes for dictionary shapes. Dictionary objects have an owned BaseShape and makeOwnBaseShape starts with an unowned BaseShape so it's not a dictionary shape.

Comment 6

[Jon Coppeard (:jonco)]

(In reply to Jan de Mooij [:jandem] from comment #5)
The crash data suggests we're finalizing a dictionary mode object though so I don't understand what's going on here.

Comment 7

[Jon Coppeard (:jonco)]

Decoder, is there any way we can reproduce this? I can't currently see how this can happen.

Comment 8

[Christian Holler (:decoder)]

I tried to find steps to reproduce, but failed so far. All I can tell is that the race occurred in Mochitests, around toolkit/components/extensions/test/mochitest/test_ext_webnavigation_incognito.html. I ran Mochitests again with the suppression for this bug disabled and retriggered the chunk several times, no luck. Either the race is gone, it is suppressed by another (similar) rule or it is intermittent enough to not show up in 10 retriggers.

Comment 9

[Christian Holler (:decoder)]

Thinking about it, the chunking also heavily changed since I found that race and that very likely influences GC. Maybe it would make sense to combine gczeal with TSan on some Mochitests? We can also try fuzzing the JS shell with TSan soon, if this bug doesn't require some browser-only objects.

Comment 11

[Christian Holler (:decoder)]

I'm adding a suppression here now to avoid further intermittent failures. But we should still try to reproduce this and find a proper solution.

Comment 13

[Christian Holler (:decoder)]

Attachment: Bug 1608068 - Add temporary TSan suppression for GC issue. r?jonco

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/integration/autoland/rev/96faa463ec39072269280e1e09e04dff7abbd227
https://hg.mozilla.org/mozilla-central/rev/96faa463ec39

Comment 15

[Steven DeTar [:sdetar]]

decoder, are you still working on this bug?

Comment 16

[Steve Fink [:sfink] [:s:]]

Just wondering: is it possible to capture TSAN bugs in rr? I don't know if you have any automation around generating such traces for these bugs, but it could be very helpful here.

Comment 17

[Jon Coppeard (:jonco)]

(In reply to Jon Coppeard (:jonco) from comment #6)
My previous comments about dictionary mode objects are wrong. I thought this was accessing the base shape in NativeObject::slotSpan because it was taking the if branch for dictionary mode object. I just worked out that other branch accesses the base shape too, via Shape::slotSpan() / Shape::getObjectClass().

Comment 18

[Christian Holler (:decoder)]

(In reply to Steve Fink [:sfink] [:s:] from comment #16)

Just wondering: is it possible to capture TSAN bugs in rr? I don't know if you have any automation around generating such traces for these bugs, but it could be very helpful here.

To be honest: I don't know. I have never tried it, but rr limits all code execution to a single core, so it might at least need a specific scheduling to reproduce under rr at all. We could try for this bug, but the problem is that we never managed to reproduce this at all by running the tests ourselves.

I was hoping that JS shell fuzzing with TSan might find us a reproducible test for this, but I'm hitting issues with shell-only races that need fixing first before we can go down that way.

Comment 19

[Jon Coppeard (:jonco)]

Attachment: Bug 1608068 - Get the class pointer from the ObjectGroup in NativeObject::slotSpan r?jandem

This changes NativeObject::slotSpan() to get the class from the object group rather than getting it from the base shape to avoid this race. It's a bit annoying to have to do this (and not at all obvious). What do you think?

Comment 20

[Steve Fink [:sfink] [:s:]]

(In reply to Christian Holler (:decoder) from comment #18)

(In reply to Steve Fink [:sfink] [:s:] from comment #16)

Just wondering: is it possible to capture TSAN bugs in rr? I don't know if you have any automation around generating such traces for these bugs, but it could be very helpful here.

To be honest: I don't know. I have never tried it, but rr limits all code execution to a single core, so it might at least need a specific scheduling to reproduce under rr at all. We could try for this bug, but the problem is that we never managed to reproduce this at all by running the tests ourselves.

While this is true in practice (anything run under rr runs on a single core), --chaos (aka -h) will report multiple cores as being present and will schedule multiple threads time-sliced on that core. Well, --chaos also tries to make tricky scheduling decisions to flush out bugs; the minimal change you need is --num-cores=N. Mostly, this should be identical to running on multiple cores except it's slower. That's not entirely true, since rr assumes that the program is race-free to begin with, but I'm hoping that TSAN would handle that part.

I was hoping that JS shell fuzzing with TSan might find us a reproducible test for this, but I'm hitting issues with shell-only races that need fixing first before we can go down that way.

Yeah, that's still a problem. If this isn't easily reproducible under TSAN, it seems unlikely that it'll be easy to reproduce under TSAN under rr.

Comment 21

[Jon Coppeard (:jonco)]

This isn't a sec-high. The worst that can happen is that the malloc memory count gets out of sync.

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/integration/autoland/rev/8bce3fa59864c8ad3bcf61c2b83f8760baa49e54
https://hg.mozilla.org/mozilla-central/rev/8bce3fa59864

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:decoder, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 24

[Jon Coppeard (:jonco)]

The race happens when doing memory accounting. Absent any evidence that this is causing problems I'd say let this one ride the trains.

Comment 25

[Aria Beingessner [:Gankra]]

Attachment: Bug 1608068 - Remove supression for seemingly fixed issue. r?decoder

Comment 26

[Aria Beingessner [:Gankra]]

Re-pushing, as it was paired with the patch for https://bugzilla.mozilla.org/show_bug.cgi?id=1601632 which got backed out. This one might still be fine though.

Comment 27

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Remove supression for seemingly fixed issue. r=decoder
https://hg.mozilla.org/integration/autoland/rev/8e58925909cccd52a321809d5094d96e7979ed0e
https://hg.mozilla.org/mozilla-central/rev/8e58925909cc