1540883 - Crash in [@ CDeviceEnumerator::OnPropertyValueChanged]

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P3)

Description

[Gabriele Svelto [:gsvelto]]

This bug is for crash report bp-d15e9dc2-d955-469f-ac4b-c34e80190331.

Top 8 frames of crashing thread:

0 mmdevapi.dll CDeviceEnumerator::OnPropertyValueChanged 
1 mmdevapi.dll CLocalEndpointEnumerator::OnMediaNotification 
2 mmdevapi.dll CMediaNotifications::OnMediaNotificationWorkerHandler 
3 ntdll.dll TppSimplepExecuteCallback 
4 ntdll.dll TppWorkerThread 
5 kernel32.dll BaseThreadInitThunk 
6 mozglue.dll static void patched_BaseThreadInitThunk mozglue/build/WindowsDllBlocklist.cpp:712
7 ntdll.dll RtlUserThreadStart 

Relatively low-volume crash, but many comments mention this happening when (un)plugging headphones or when using Bluetooth headsets so the relatively low rate might be because it's a corner-case that can be hit only when doing that. The code on the stack seems to be outside of our codebase though so I'm unsure if it's a problem we're causing or if it's a Windows/driver issue.

Comment 1

[Andreas Pehrson [:pehrsons]]

kinetik, can you make anything of this?

I saw no interesting other threads in the crash report in comment 0. My only interesting observation is that the crash rate was higher in 62 and then saw a drop in 63. Since then it's been steady.

I'll also note bug 1332789 which is similarly crashing in CDeviceEnumerator, though in a plugins process.

Comment 2

[Matthew Gregan [:kinetik]]

There's also bug 1507182 with CDeviceEnumerator related crashes, but in DestroyHWndNotificationThread and with more useful stacks.

Of the 20 or so crash stacks I looked at, only one seemed to have any media/playback related threads around. Quite a few had a PlayEventSound thread from the widget code, but that code looks fairly simple and may not be implicated. Possibly crashing during audio init/deinit, maybe racy?

The crashes seem to be a mix of content process and unclassified ("process type" field blank, non-e10s?).

We can register an IMMNotificationClient (which has a OnPropertyValueChanged callback) from dom/media/AudioNotificationSender.cpp, dom/plugins/ipc/PluginUtilsWin.cpp, and two places in cubeb: wasapi_collection_notification_client which only landed in 67 via bug 1527659, and wasapi_endpoint_notification_client which we stopped registering in content processes back in 61 via bug 1427011 to avoid mystery crashes in CAudioSessionControl::QueueStreamSwitch.

Not much to go on here, but based on the above perhaps we should focus on the two AudioNotification cases in Gecko...

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Some of these crashes are e5e5 UAF crashes, marking as security bug (probably will be sec-high given we're passing a UAF address to some OS function)

Comment 4

[Daniel Veditz [:dveditz]]

Looking at the comments it sounds like a lot of folks have this happen when they change their audio device (plugin/unplug headphones, connect a bluetooth headset). Going to call this 'sec-moderate' for now as it seems to flow from user action and not what the website is up to.

Comment 5

[Andrew McCreight [:mccr8]]

We have Windows ASan builds now, so somebody could try out the things suggested with those. Chris Peterson had an issue a while ago with similar steps that came up when he was using Windows ASan that was fixed, so it might help.

Comment 6

[Andrew McCreight [:mccr8]]

There are still a decent volume of UAF crashes with this signature. 42 in the last two weeks on current versions.

For example: bp-b5d444a8-d26a-4abd-9063-77b750230323

Comment 7

[Andreas Pehrson [:pehrsons]]

I note there is a very high correlation between these crashes and "A-Volute" modules. Either SS3DevProps.dll signed by "A-Volute" (example) or AudioDevProps2.dll signed by "A-Volute SAS" (example).

"SS3" seems to be Sonic Studio 3 (released in 2018, shortly before this bug was filed).

"AudioDevProps2" seems to come from Nahimic which is SteelSeries branded.

Seems like a plugin/driver issue to me.

Comment 8

[Greg Stoll :gstoll]

We've already had to block some Nahimic DLLs before in bug 1804023, FWIW.

Comment 9

[Greg Stoll :gstoll]

Nice catch with those module correlations. Unfortunately, neither of those apps will work on my laptop since I don't have the right kind of hardware, I guess.

Given that

• this is a relatively common crash and possibly security-related
• we've had to block Nahimic DLLs before

I'm going to investigate just blocking the two DLLs mentioned above.

Comment 10

[Greg Stoll :gstoll]

SS3DevProps.dll checklist:
Windows DLLBlocklist request form

1. How were we aware of the problem?
   Crash reports.

2. What is a suspicious product causing the problem?
   Sonic Studio 3

3. Is the product downloadable? If so, do we have a local repro?
   It is downloadable, but it won't load on my system because I don't have the right hardware.

4. Which OS versions does the problem occur on?
   Happens on Windows 10, so we can't limit this to older versions.

5. Which process types does the problem occur on?
   Just the browser process

6. What is the maximum version of the module in the crash reports?
   The DLL doesn't seem to have a version recorded in our third-party-module pings.

7. Is the issue fixed by a newer version of the product?
   Unknown since we don't have a version number.

8. Do we have data about the module in the third-party-module ping?
   Yes

9. Do we know how the module is loaded?
   It gets loaded through COM, so it should be safe to block.

10. Describe your conclusion.
    We should block all versions of SS3DevProps.dll in the browser process.

Comment 11

[Greg Stoll :gstoll]

AudioDevProps2.dll checklist:
Windows DLLBlocklist request form

1. How were we aware of the problem?
   Crash reports

2. What is a suspicious product causing the problem?
   Nahimic

3. Is the product downloadable? If so, do we have a local repro?
   It is downloadable, but it won't load on my system because I don't have the right hardware.

4. Which OS versions does the problem occur on?
   Happens on Windows 10, so we can't limit this to older versions.

5. Which process types does the problem occur on?
   Just the browser process

6. What is the maximum version of the module in the crash reports?
   2.6.14.0

7. Is the issue fixed by a newer version of the product?
   We have pings from later versions of the DLL but no crashes, so this is possible.

8. Do we have data about the module in the third-party-module ping?
   Yes

9. Do we know how the module is loaded?
   It gets loaded through COM, so it should be safe to block.

10. Describe your conclusion.
    We should block versions less than or equal to 2.6.14.0 of AudioDevProps2.dll in the browser process.

Comment 12

[Greg Stoll :gstoll]

Attachment: Bug 1540883 - block a few problematic DLLs r=gsvelto

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

block a few problematic DLLs r=gsvelto
https://hg.mozilla.org/integration/autoland/rev/bd028b83ef49c923964ff675bf53a51e967cc72b
https://hg.mozilla.org/mozilla-central/rev/bd028b83ef49

Comment 14

[Greg Stoll :gstoll]

Attachment: Bug 1540883 - block a few problematic DLLs r=gsvelto

Original Revision: https://phabricator.services.mozilla.com/D174502

Comment 15

[Phabricator Automation]

Uplift Approval Request

• Code covered by automated testing: no
• String changes made/needed: no
• Fix verified in Nightly: no
• Explanation of risk level: Just blocking a few third-party audio DLLs that have caused problems
• Steps to reproduce for manual QE testing: n/a
• User impact if declined: exposure to a possible security bug
• Risk associated with taking this patch: low
• Is Android affected?: no
• Needs manual QE test: no

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9327370 [details]
Bug 1540883 - block a few problematic DLLs r=gsvelto

This was approved for 102.11esr in Phabricator, but we had some hiccups with using Lando to push it. Adding the approval here for posterity and pushing it manually.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/ae06ad7a4dc6