1607588 - ThreadSanitizer: data race [@ nssSlot_GetToken] vs. [@ nssToken_Destroy] with potential use-after-free during shutdown

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P2)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision bc5880b621d5.

This is a race I saw during the security/manager/ssl/tests/unit/test_nss_shutdown.js test. It looks like a use-after-free on an nssToken. The main thread has ShutdownNSS() on the stack, so I assume this could be a shutdown problem, but it is unclear to me if this is test-only or if the same can happen in production. At most, this could cause intermittent test failures (crashes) of the particular test. And indeed we have bug 1563837 and bug 1546720 on file for intermittent crashes of this particular test.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[J.C. Jones [:jcj] (he/him)]

I spent some time this morning going through this code comparing to prior notes for NSS shutdown. It's good that we do have finally an exact case of what's happening during this race, but sadly we already knew the final culprits, and the ultimate cause being that NSS is synchronizing the wrong things during slot destruction... which is rough to fix.

I'm marking this P2 because it's worth a few hours searching before we decide P1 or P3.

Comment 3

[Aria Beingessner [:Gankra]]

Attachment: Bug 1607588 - Remove supression for seemingly fixed issue. r?decoder

Depends on D94852

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac0770d9bdfa
Remove supression for seemingly fixed issue. r=decoder

Comment 5

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/ac0770d9bdfa

Comment 6

[Aria Beingessner [:Gankra]]

Attachment: tsan-backtrace-dec-2020.txt

I hit this again while turning on firefox's full shutdown code for tsan.

nssToken_Destroy

vs

nssSlot_GetToken

Comment 7

[Aria Beingessner [:Gankra]]

Attachment: Bug 1607588 - add back supression for code that gets stressed more in a full shutdown. r?decoder

Depends on D100264

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e17a24cf19c8
add back supression for code that gets stressed more in a full shutdown. r=decoder

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/e17a24cf19c8

Comment 10

[Intermittent Failures Robot]

1 failures in 2956 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1607588&startday=2021-01-04&endday=2021-01-10&tree=all

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:beurdouche, maybe it's time to close this bug?

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:beurdouche, maybe it's time to close this bug?

Comment 13

[John Schanck [:jschanck]]

Attachment: Bug 1607588 - Remove suppression for NSS shutdown race. r?decoder

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/51dfe326a194
Remove suppression for NSS shutdown race. r=decoder

Comment 15

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/51dfe326a194