1602009 - ThreadSanitizer: data race [@ mozilla::net::TLSServerConnectionInfo::HandshakeCallback] vs. [@ mozilla::net::TLSServerConnectionInfo::SetSecurityObserver] on mSecurityObserver

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[Christian Holler (:decoder)]

The attached crash information was detected while running CI tests with ThreadSanitizer on mozilla-central revision 6989fcd6bab3.

I briefly looked into this and it looks like it is simply racing on assigning mSecurityObserver vs. checking it in TLSServerConnectionInfo::HandshakeCallback. The TLSServerConnectionInfo::SetSecurityObserver uses a mutex for setting the observer, but TLSServerConnectionInfo::HandshakeCallback is reading mSecurityObserver without it.

Is this intended behavior? I guess even with the mutex being used to check mSecurityObserver the order between these two is still not defined and it could lead to the observer not being notified for the handshake callback (unless there is some kind of invisible guarantee that synchronizes these).

I've only seen this race happening on XPCShell tests so far.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Christian Holler (:decoder)]

Attachment: Detailed Crash Information

Comment 2

[Kershaw Chang [:kershaw]]

Attachment: Bug 1602009 - Make sure mSecurityObserver can be only accessed within a lock

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac461b0e7204
Make sure mSecurityObserver can be only accessed within a lock r=valentin

Comment 4

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/ac461b0e7204

Comment 5

[Liz Henry (:lizzard) (relman/hg->git project)]

May need uplift (if we uplift the work in bug 1527242).

Comment 6

[Junior [inactive]]

Comment on attachment 9115440 [details]
Bug 1602009 - Make sure mSecurityObserver can be only accessed within a lock

Beta/Release Uplift Approval Request

• User impact if declined: high intermittent rate of treeherder
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: Bug 1527242, Bug 1602009
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): small change. Note that we should uplift this first and then uplift Bug 1527242
• String changes made/needed:

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9115440 [details]
Bug 1602009 - Make sure mSecurityObserver can be only accessed within a lock

Fix for test failures, OK for uplift to beta. This should not affect 72 since we've built and released the last 72 beta, but maybe it will help with 73 early betas.

Comment 8

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/releases/mozilla-beta/rev/4ca29ff7ef60

Comment 9

[Kershaw Chang [:kershaw]]

(In reply to Junior [:junior] from comment #6)

Comment on attachment 9115440 [details]
Bug 1602009 - Make sure mSecurityObserver can be only accessed within a lock

Beta/Release Uplift Approval Request

• User impact if declined: high intermittent rate of treeherder
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: Bug 1527242, Bug 1602009
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): small change. Note that we should uplift this first and then uplift Bug 1527242
• String changes made/needed:

Thanks for doing this for me.