1674835 - ThreadSanitizer: data race [@ nsHttpTransaction::ReadSegments] vs. [@ nsHttpChannel::OnStartRequest]

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P1)

Description

[Aria Beingessner [:Gankra]]

Attachment: tsan-backtrace-nov-2020.txt

I found this will testing my WIP branch to properly instrument the Rust stdlib (removing some large supressions that hid bugs like this).

mSecurityInfo (the actual pointer) is being assigned as it's also being dereferenced without proper synchronization.

As an additional note, nsISupports is not an appropriate type to use for multi-threaded situations, as not all nsISupports use atomic refcounting. Please use a more restrictive type.

Marking as security as this is racing on the value of a virtual pointer.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf
Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Daniel Veditz [:dveditz]]

Valentine: this looks like a trouble race -- any ideas?

Gankra: Do you know what test was running at the time this trace was captured? might help if the developers could catch it "live" so to speak.

Comment 2

[Aria Beingessner [:Gankra]]

Prerequisite: have the patches in Bug 1671691 and remove the "race:nsHttpTransaction::ReadSegments\n" suppression from TsanOptions.cpp

Then tsan Mochitest-8 block will occasionally run into it: https://treeherder.mozilla.org/jobs?repo=try&revision=1bed40011aa3e20a16092bc78f8a806fe5a51265&selectedTaskRun=DcZD69U-R46T2ddu5FoWIg.0

It looks like it might have happened during layout/generic/test/test_scroll_position_restore.html

Comment 3

[Aria Beingessner [:Gankra]]

Oh huh, as Kershaw has pointed out above, it might happen without the Rust patches? And I just got a 1/1000 hit? Huh!

Comment 4

[Valentin Gosu [:valentin] (he/him)]

Seems like a problem in any case.
The fix isn't that hard.

Comment 5

[Valentin Gosu [:valentin] (he/him)]

Attachment: Bug 1674835 - Hold mutex when accessing mSecurityInfo r=#necko

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Hold mutex when accessing mSecurityInfo r=necko-reviewers,kershaw
https://hg.mozilla.org/integration/autoland/rev/080712a4d749fa0407b63d8c355ee59c24939474

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/080712a4d749

Comment 8

[Valentin Gosu [:valentin] (he/him)]

Comment on attachment 9186677 [details]
Bug 1674835 - Hold mutex when accessing mSecurityInfo r=#necko

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: This patch fixes data race on securityInfo pointer.
• User impact if declined: Without it there's a potential for crashes whenever the race happens.
• Fix Landed on Version: 85
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We now hold a lock to make sure the pointer is updated atomically.
  The impact should be minimal - at worst it could change the timing of operations and lead to a bug, but that is unlikely.
• String or UUID changes made by this patch:

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9186677 [details]
Bug 1674835 - Hold mutex when accessing mSecurityInfo r=#necko

Approved for 78.7esr.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/ab059355908d

Comment 11

[Aria Beingessner [:Gankra]]

Attachment: Bug 1674835 - remove supression for fixed issue. r?decoder

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

remove supression for fixed issue. r=decoder
https://hg.mozilla.org/integration/autoland/rev/15ea118b8e5602a20e6b0193ec4535b809b009e0
https://hg.mozilla.org/mozilla-central/rev/15ea118b8e56