1682938 - ThreadSanitizer: data race [@ ReverbInputBuffer::writeIndex] vs. [@ ReverbInputBuffer::write]

Status: RESOLVED FIXED

(Core :: Web Audio, defect)

Description

[Aria Beingessner [:Gankra]]

Attachment: tsan-backtrace-dec-2020.txt

Found while bringing up wpt under tsan. Tentatively marking as security in case this can cause buffer overruns.

It looks like m_writeIndex needs to be atomic. The documentation on writeIndex even says "Background threads can call this to check if there's anything to read". Not sure if a more robust synchronization of the state is also required.

This code is extremely old and pulled in from Chrome, so presumably they have fixed this issue upstream by now?

Permalinks to problematic lines:

ReverbInputBuffer::writeIndex

ReverbInputBuffer::write

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf
Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Comment 1

[Andrew McCreight [:mccr8]]

Yeah, looks like you are right. It appears to be atomic now in Chromium: https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/platform/audio/reverb_input_buffer.h

Comment 2

[Andrew McCreight [:mccr8]]

Here's the commit: https://chromium.googlesource.com/chromium/src/+/98eec82ac5d51f5c4923b858d4513d9d4db786b3%5E%21/third_party/blink/renderer/platform/audio/reverb_input_buffer.h

Comment 3

[Andrew McCreight [:mccr8]]

Oops, here's the actual commit that fixes the race: https://chromium.googlesource.com/chromium/src/+/e027b261dc44c339f395f2dba24c36923c405018

Comment 4

[Aria Beingessner [:Gankra]]

Attachment: Bug 1682938 - fixup ReverbInputBuffer's writeIndex. r?mccr8

This mirrors the change made in https://chromium.googlesource.com/chromium/src/+/e027b261dc44c339f395f2dba24c36923c405018%5E%21/

Resolving an issue that chromium fixed in 2017, but was only now noticed in the firefox copy due to tsan. See also:
https://bugs.chromium.org/p/chromium/issues/detail?id=732918

I'm not sure why they changed the implementation to no longer silently drop writes that are too long, so I'm making the assertion a release one to be safe.

Comment 5

[Andrew McCreight [:mccr8]]

Chrome also only noticed it due to TSan. They filed it as a security bug, but it doesn't appear to have been assigned a rating.

Comment 6

[Aria Beingessner [:Gankra]]

Comment on attachment 9193742 [details]
Bug 1682938 - fixup ReverbInputBuffer's writeIndex. r?mccr8

Security Approval Request

• How easily could an exploit be constructed based on the patch?: Unclear. I expect this would mostly only allow you to read uninitialized/old waveform values into the sound system, creating some garbled noises. Google didn't do an analysis of how exploitable this was.

A comment I added indicates concurrency needs to be minded, but honestly that's just clear from patch making things atomic. I can remove it if desired, though.

• Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: Yes
• Which older supported branches are affected by this flaw?: all branches
• If not all supported branches, which bug introduced the flaw?: None
• Do you have backports for the affected branches?: No
• If not, how different, hard to create, and risky will they be?: It should backport cleanly to anything from 2019. Any earlier and some minor changes will need to be made for reformatting. Otherwise this code has largely been untouched since it was added.
• How likely is this patch to cause regressions; how much testing does it need?: It changes the "out of bounds write" behaviour from "silently dropping the write" to a release assert (because the chrome devs also applied that change). I'm not confident I've tested that it's fine sufficiently.

Comment 7

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Comment on attachment 9193742 [details]
Bug 1682938 - fixup ReverbInputBuffer's writeIndex. r?mccr8

Approved to land and uplift

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/autoland/rev/798d7b63c7111c67291e49306e5eb07891136962

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/798d7b63c711

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

As part of a security bug pattern analysis, we are requesting your help with a high level analysis of this bug. It is our hope to develop static analysis (or potentially runtime/dynamic analysis) in the future to identify classes of bugs.

Please visit this google form to reply.

Comment 11

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/80664afb7099

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/90258b5cfdc9