1935079 - Crash in [@ __delayLoadHelper2 | _tailMerge_ole32.dll | (anonymous namespace)::get_default_endpoint((anonymous namespace)::com_ptr<T>&, __MIDL___MIDL_itf_mmdeviceapi_0000_0000_0001, __MIDL___MIDL_itf_mmdeviceapi_0000_0000_0002)]

Status: VERIFIED FIXED

(Core :: Audio/Video: cubeb, defect, P3)

Description

[Bob Owen (:bobowen)]

I found this while monitoring the USER_RESTRICTED roll-out.
In Fx133 on crash-stats there are currently 29 crashes from 5 clients, all of which have USER_RESTRICTED.
However, looking back to previous versions we have crashes with both content sandbox level 7 and 8, so it's not clear that this is directly related.
We currently have a rule to allow loads from system32 for the webcodec encoding, so also not clear why it is failing to load.
Loading ole32.dll is not something we want to be doing in the content process.

Crash report: https://crash-stats.mozilla.org/report/index/3a5265ee-fd6a-4fa3-b380-7af090241122

Reason:

FACILITY_VISUALCPP / ERROR_MOD_NOT_FOUND

Top 10 frames:

0  KERNELBASE.dll  RaiseException
1  xul.dll  __delayLoadHelper2(ImgDelayDescr const*, long long (**)())  /builds/worker/workspace/obj-build/toolkit/library/build/D:/a/_work/1/s/src/vctools/delayimp/delayhlp.cpp:301
2  xul.dll  _tailMerge_ole32.dll
3  xul.dll  (anonymous namespace)::get_default_endpoint((anonymous namespace)::com_ptr<IM...  media/libcubeb/src/cubeb_wasapi.cpp:1635
4  xul.dll  wasapi_init(cubeb**, char const*)  media/libcubeb/src/cubeb_wasapi.cpp:1731
5  xul.dll  cubeb_init(cubeb**, char const*, char const*)  media/libcubeb/src/cubeb.c:276
6  xul.dll  mozilla::CubebUtils::GetCubebUnlocked::<lambda_6>::operator()() const  dom/media/CubebUtils.cpp:606
6  xul.dll  mozilla::mscom::EnsureMTA::EnsureMTA(mozilla::CubebUtils::GetCubebUnlocked::<...  ipc/mscom/EnsureMTA.h:66
6  xul.dll  mozilla::CubebUtils::GetCubebUnlocked()  dom/media/CubebUtils.cpp:603
7  xul.dll  mozilla::CubebUtils::InitPreferredSampleRate()  dom/media/CubebUtils.cpp:380

Comment 1

[Jim Mathies [:jimm]]

Maybe Mathew has some ideas. Doesn't look serious though.

Comment 2

[Matthew Gregan [:kinetik]]

These crashes are the result of initializing libcubeb directly in the content process. The default configuration uses remoting (AudioIPC) to deal with this, so these crashes are produced by systems with the media.cubeb.sandbox pref explicitly set to false.

Confirmed with a local test that media.cubeb.sandbox = false produces the same crash stack when opening an audio stream. This pref only exists for debugging purposes and no longer functions with the default content sandbox restrictions, so I think the pref should be removed. It may be useful to retain some other way to disable audio remoting for debugging, but it would need to be tied to lowering the content sandbox level.

Comment 3

[Matthew Gregan [:kinetik]]

Attachment: Bug 1935079 - Ignore `media.cubeb.sandbox` when content sandbox is enabled. r?#cubeb-reviewers

Comment 4

[Bob Owen (:bobowen)]

Thanks for the quick fix kinetik.
I did try flipping that pref, but for me it just broke sound rather than crashed.
To be honest I thought that we would already break with the pref flipped, which is possibly correct in a small number of cases given that we already see this crash even without USER_RESTRICTED.

We're probably only at about 8% roll-out.
Difficult to predict with such small numbers, but it does mean that we could easily see this increase more than 10x.
We're also seeing bug 1929333 despite attempting to block the DLL, so the blocking isn't working all the time.
Given this I'm going to hold the roll-out at 10% for Fx133, so I can continue to monitor.

It would be good to get this uplifted to Beta though, so I can attempt to roll-out again in Fx134.

Comment 5

[Bob Owen (:bobowen)]

Forgot to mention that I noticed you have to flip the crash graph settings to "Like Match" to pick up the crashes for some reason.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1c7e2d98ff28
Ignore `media.cubeb.sandbox` when content sandbox is enabled.  r=cubeb-reviewers,pehrsons

Comment 7

[Matthew Gregan [:kinetik]]

Attachment: Bug 1935079 - Ignore `media.cubeb.sandbox` when content sandbox is enabled.

Original Revision: https://phabricator.services.mozilla.com/D231167

Comment 8

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: Crashes for users with audio sandboxing disabled
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Test Firefox audio with "media.cubeb.sandbox" pref set to both "true" and "false" (must restart Firefox after changing pref)
• Risk associated with taking this patch: very low
• Explanation of risk level: only affect users with non-default pref setting
• String changes made/needed: N/A
• Is Android affected?: no

Comment 9

[Sandor Molnar[:smolnar]]

Backed out for causing build bustages @ CubebUtils.cpp

Backout link: https://hg.mozilla.org/integration/autoland/rev/ecebc25e83f658854a8170f80102208b92bdc664

Push with failures

Failure log -> ERROR - /builds/worker/checkouts/gecko/dom/media/CubebUtils.cpp:305:27: error: use of undeclared identifier 'IsContentSandboxEnabled'

Comment 10

[Matthew Gregan [:kinetik]]

This just needs a MOZ_SANDBOX ifdef check added, I'll update the patch shortly.

Comment 11

[Matthew Gregan [:kinetik]]

Attachment: Bug 1935079 - Ignore `media.cubeb.sandbox` when content sandbox is enabled.

Original Revision: https://phabricator.services.mozilla.com/D231167

Comment 12

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: crashes for users with audio sandboxing disabled
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: Test audio with "media.cubeb.sandbox" pref set to "true" and "false" (requires Firefox restart)
• Risk associated with taking this patch: low
• Explanation of risk level: only affects users with non-default "media.cubeb.sandbox" pref
• String changes made/needed: N/A
• Is Android affected?: no

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fb9339c19565
Ignore `media.cubeb.sandbox` when content sandbox is enabled.  r=cubeb-reviewers,pehrsons

Comment 14

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/fb9339c19565

Comment 15

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/8c6ffdb97001

Comment 16

[Monica Chiorean]

I was able to reproduce the crash on Win11x64 using FF build 135.0a1(20241204092616) and having pref 'media.cubeb.sandbox'=false, while loading reddit.com.
Verified as fixed on Win11x64/Mac 12.6 using FF builds 135.0a1(20241212212433)and 134.b9.