1606803 - ThreadSanitizer: data race [@ _pr_ipv6_is_present] vs. [@ _pr_init_ipv6] on ipv6_is_present

Status: RESOLVED DUPLICATE of bug 1686141

(Core :: Networking, defect, P3)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision d1ac49b9eb3e.

This is a race on the global ipv6_is_present and makes one thread potentially call pr_GetAddrInfoByNameFB here:

https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/nsprpub/pr/src/misc/prnetdb.c#2114

Not sure if this has any actual effect or is expected. I will blacklist this for now, let me know if this is expected or can/should be fixed.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Andy Grover [:grover]]

PRBool _pr_ipv6_is_present(void)
{
    if (PR_CallOnce(&_pr_init_ipv6_once, _pr_init_ipv6) != PR_SUCCESS) {
        return PR_FALSE;
    }
    return ipv6_is_present;
}

PR_CallOnce ensures ipv6_is_present global will be initialized, and _pr_init_ipv6 called only once.

We could maybe correct the race by wrapping ipv6_is_present in a PRRWLock. Kai wdyt?

Comment 3

[Nathan Froyd [:froydnj]]

If PR_CallOnce provides sufficient memory barriers/locking, it should be unnecessary to add further locking.

Comment 4

[Kai Engert (:KaiE:)]

Based on the API description of PR_CallOnce, there shouldn't be a race.

From https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR/Reference/PR_CallOnce :
"While the first thread executes this function, other threads attempting the same initialization will be blocked until it has been completed."

Based on the above, I'd expect the following to happen:

• first thread arrives at
  PR_CallOnce(&_pr_init_ipv6_once, _pr_init_ipv6)
• first thread proceeds to execute _pr_init_ipv6()
• second thread arrives at
  PR_CallOnce(&_pr_init_ipv6_once, _pr_init_ipv6)
• first thread is still inside _pr_init_ipv6(), the second thread should be blocked until the first thread is ready

If the above is correct, a second thread inside _pr_ipv6_is_present() shouldn't be able to access ipv6_is_present prior to the completion of _pr_init_ipv6().

Can we assume PR_CallOnce works as documented, or do we need to verify that?

Comment 5

[Nathan Froyd [:froydnj]]

PR_CallOnce works just fine.

Is part of the problem that the call of _pr_init_ipv6 is performed outside of a lock (inside PR_CallOnce), so TSan doesn't see the synchronization between threads?

Comment 6

[Kai Engert (:KaiE:)]

See also: bug 273649

Comment 7

[Kai Engert (:KaiE:)]

Adding a test for PR_CallOnce in bug 1609712. Seems to work as expected on Linux.

Comment 8

[Aria Beingessner [:Gankra]]

Attachment: Bug 1606803 - Remove supression for seemingly fixed issue. r?decoder

Depends on D94493

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d4953114117e
Remove supression for seemingly fixed issue. r=decoder

Comment 10

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/d4953114117e

Comment 11

[Aria Beingessner [:Gankra]]

Attachment: tsan-backtrace-nov-2020.txt

I think I just randomly hit this, might need to revert + reopen.

Comment 12

[Aria Beingessner [:Gankra]]

Attachment: Bug 1606803 - revert d4953114117e, still happens rarely. r?decoder

Specifically an instance was being blocked by our nsHostResolver suppression.

Depends on D93416

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b46517fe6e6b
revert d4953114117e, still happens rarely. r=decoder

Comment 14

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/b46517fe6e6b

Comment 15

[Aria Beingessner [:Gankra]]

:nika has pointed out that C++11 now guarantees proper synchronization of local static initializers, so we can potentially replace all uses of PR_CallOnce with

T GetMyCachedValue() {
  static T sValue = ([] {
     // Code to compute value
  })();
  return sValue;
}

And presumably the compiler will have a decent implementation that tsan is also happy with.

Comment 16

[Aria Beingessner [:Gankra]]

Scratch that, forgot that nsprpub is Actual C so this pattern isn't usable.

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:jstutte, maybe it's time to close this bug?

Comment 18

[Jens Stutte [:jstutte]]

Could it be that bug 1685552 solved this implicitely? At least the call to GetSubjectAltNames() is gone now.

Comment 19

[Jens Stutte [:jstutte]]

(btw: I think this bug has the wrong module assigned)

Comment 20

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Looking at the original report, bug 1686141 probably addressed this.