1606864 - ThreadSanitizer: data race [@ mozilla::net::nsSocketTransport::Close] vs. [@ mozilla::net::nsSocketTransport::OnSocketDetached]

Status: RESOLVED FIXED

(Core :: Networking, defect, P2)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision c0a6eb95b65c.

This looks like an interesting race. We have this code:

https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/netwerk/base/nsSocketTransport2.cpp#2636

racing vs.

https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/netwerk/base/nsSocketTransport2.cpp#2419

I see two things that can happen here:

1. If mFDFastOpenInProgress is true then this can lead to a null crash in either of the two functions (because the other sets it to nullptr).

2. The two threads can race on mDoNotRetryToConnect causing random reconnecting (or not) on the socket thread. I am seeing that as a separate race report actually.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Aria Beingessner [:Gankra]]

Attachment: tsan-backtrace-oct-2020.txt

Triage: still an issue

Comment 3

[Aria Beingessner [:Gankra]]

Attachment: tsan-backtrace-dec-2020.txt

New variant,

racing on mOutputClosed via [@nsSocketTransport::OnMsgInputClosed] vs [@nsSocketTransport::OpenOutputStream].

Although this doesn't match the existing signature, it's following the same theme of "event handlers don't seem to synchronize their state atomically". OnMsgInputClosed is "socket thread only" but OpenOutputStream doesn't have that indicator.

Source snapshots:

https://searchfox.org/mozilla-central/rev/6bb59b783b193f06d6744c5ccaac69a992e9ee7b/netwerk/base/nsSocketTransport2.cpp#1990

vs

https://searchfox.org/mozilla-central/rev/6bb59b783b193f06d6744c5ccaac69a992e9ee7b/netwerk/base/nsSocketTransport2.cpp#2666

Comment 4

[Aria Beingessner [:Gankra]]

Also while checking this, I noticed the "oct 2020" backtrace is yet another variant,

racing on mThroughCaptivePortal via [@nsHttpConnection()] vs [@nsHttpHandler::Observe]

(via [@nsSocketTransport::OnSocketDetached] vs [@CaptivePortalService::Complete])

Source snapshots:

https://searchfox.org/mozilla-central/rev/6bb59b783b193f06d6744c5ccaac69a992e9ee7b/netwerk/protocol/http/nsHttpConnection.cpp#119

vs

https://searchfox.org/mozilla-central/rev/6bb59b783b193f06d6744c5ccaac69a992e9ee7b/netwerk/protocol/http/nsHttpHandler.cpp#2505

Comment 5

[Aria Beingessner [:Gankra]]

Attachment: Bug 1606864 - Add more supressions for new variants. r?decoder

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c0b341fa6af6
Add more supressions for new variants. r=decoder

Comment 7

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/c0b341fa6af6

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:jstutte, maybe it's time to close this bug?

Comment 9

[Jens Stutte [:jstutte]]

No, we just added supressions.

Comment 10

[Valentin Gosu [:valentin] (he/him)]

Attachment: Bug 1606864 - Make mDoNotRetryToConnect atomic r=#necko

Fixes race between nsSocketTransport::Close and
nsSocketTransport::RecoverFromError called from OnSocketDetached.

Depends on D128183

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Make mDoNotRetryToConnect atomic r=necko-reviewers,decoder,dragana
https://hg.mozilla.org/integration/autoland/rev/9a3d7d1527ad169b3dfe00180579c206be8b3c70
https://hg.mozilla.org/mozilla-central/rev/9a3d7d1527ad

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta/ESR91 approval when you get a chance.

Comment 13

[Valentin Gosu [:valentin] (he/him)]

Comment on attachment 9245708 [details]
Bug 1606864 - Make mDoNotRetryToConnect atomic r=#necko

Beta/Release Uplift Approval Request

• User impact if declined: Data race
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce: Difficult to reproduce manually.
• List of other uplifts needed: Bug 1712671
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small change makes variables atomic.
• String changes made/needed:

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration:
• User impact if declined: Data race
• Fix Landed on Version: 95
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small change makes variables atomic.
• String or UUID changes made by this patch:

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9245708 [details]
Bug 1606864 - Make mDoNotRetryToConnect atomic r=#necko

Approved for 94.0b9 and 91.3esr.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/0609ab3e68b9

Comment 16

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-esr91/rev/4a64d8b1e257