1866595 - Crashes with Apple's AudioDSP.component driver on stack, on macOS 12 and up

Status: REOPENED

(Core :: Audio/Video: cubeb, defect)

Description

[BugBot [:suhaib / :marco/ :calixte]]

Crash report: https://crash-stats.mozilla.org/report/index/892d0d9d-d9f1-4fae-a1eb-d155a0231123

Reason: EXC_BAD_ACCESS / KERN_PROTECTION_FAILURE

Top 10 frames of crashing thread:

0  AudioDSP  AudioDSP@0x24ce5c  
1  AudioDSP  AudioDSP@0x38568c  
2  AudioDSP  AudioDSP@0x37fa98  
3  AudioDSP  AudioDSP@0x4bd8a8  
4  AudioToolboxCore  APComponent::newInstance  
5  AudioToolboxCore  instantiate  
6  AudioToolboxCore  __AudioComponentInstanceNew_block_invoke  
7  AudioToolboxCore  Synchronously  
8  AudioToolboxCore  AudioComponentInstanceNew  
9  XUL  cubeb_coreaudio::backend::create_typed_audiounit  third_party/rust/cubeb-coreaudio/src/backend/mod.rs:1148

By querying Nightly crashes reported within the last 2 months, here are some insights about the signature:

• First crash report: 2023-11-23
• Process type: Parent
• Is startup crash: No
• Has user comments: No
• Is null crash: No

By analyzing the backtrace, the regression may have been introduced by a patch [1] to fix Bug 1670633.

[1] https://hg.mozilla.org/mozilla-central/rev?node=be8e9a8eaa80

:pehrsons, since you are the author of the potential regressor, could you please take a look?

Comment 1

[Andreas Pehrson [:pehrsons]]

We saw the same crash signature on cubeb's CI (Github Actions, also running on MacOS 12). There were two crash signatures within AudioDSP. This one from creating a vpio AudioUnit, and another completely inside platform code, under a HALObject::PropertiesChanged callback.

These seem like platform bugs to me. I tried linearizing all accesses threading wise to no avail. Gecko's API use (leading to the crashes reported here) is already single threaded. I have not been able to reproduce any crashes locally on MacOS 14, which makes me think this could be MacOS 12 specific. I have an older Mac (10.15). I'll upgrade that to 12 and try reproducing again. I'll also try to move upstream CI to MacOS 13 to see if we get different results.

Comment 2

[Steven Michaud [:smichaud] (Retired)]

Here's a broader search for crashes that are probably all the same Apple bug. Note that these too are almost entirely on macOS 12.

https://crash-stats.mozilla.org/search/?modules_in_stack=~audiodsp&platform=Mac%20OS%20X&date=%3E%3D2023-09-10T19%3A40%3A00.000Z&date=%3C2023-12-10T19%3A40%3A00.000Z&_facets=signature&_facets=platform_version&_facets=cpu_arch&_facets=proto_signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Comment 3

[Andreas Pehrson [:pehrsons]]

I have been able to reproduce locally on MacOS 12 FWIW, and have filed a bug with Apple on this matter.

Comment 4

[Steven Michaud [:smichaud] (Retired)]

What are your steps to reproduce?

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:karlt, could you have a look please?

For more information, please visit BugBot documentation.

Comment 6

[Donal Meehan [:dmeehan]]

(In reply to Andreas Pehrson [:pehrsons] from comment #3)

I have been able to reproduce locally on MacOS 12 FWIW, and have filed a bug with Apple on this matter.

:persons did you get any updates on this?
Anything that can be done for Fx122 on our side, or will the bug be closed later?

Comment 7

[Andreas Pehrson [:pehrsons]]

No concrete updates from Apple as of yet. We're monitoring the crash rate, and have the option to disable the functionality that triggers this.

Comment 8

[Mikail]

Hi,

Is there any known workaround to this issue? I'm trying to reproduce and get details about another crash/bug with signature objc_release | CACFString::~CACFString (https://crash-stats.mozilla.org/report/index/4b8ec151-2939-4dd4-8b8f-49ad20231227, introduced somewhere in 122.0) using mozregression but I'm unable to due to the AudioDSP@0x26bf72 crashes happening since 122.0a1 (this ticket, crash report is https://crash-stats.mozilla.org/report/index/9b0e1e67-aba9-4f14-8884-31bdf0231227).

Additional details for this ticket (#1866595), reproduction steps on my end are:

1. Go to a meeting link in Google Meet (https://meet.google.com/XXX-XXXX-XXX, generate one using a Google account)
2. When firefox asks for microphone permissions, grant it
3. Firefox crashes

Thanks!

Comment 9

[Mikail]

It seems like another issue could be related with AudioDSP (HALObject::PropertiesChanged), but this time with the signature objc_msgSend | CFStringCompare: https://crash-stats.mozilla.org/report/index/e0c02116-f919-4fd8-a5fc-5c6aa0231228

In the case of this crash, the steps to reproduce are:

1. Go to a meeting link in Google Meet, no need to click "Join" (https://meet.google.com/XXX-XXXX-XXX, generate one using a Google account)
2. Plug headphones to the 3.5mm jack port
3. Unplug the headphones after a few seconds

Only tried to reproduce this on MacOS 12.7.1.

The crash AudioDSP@0x26bf72 may also happen sometimes at step 1), if it does, retrying usually works.

It looks like there are a lot of issues regarding audio, thus it would be quite possible that the crash signature objc_release | CACFString::~CACFString (https://crash-stats.mozilla.org/report/index/4b8ec151-2939-4dd4-8b8f-49ad20231227) is also caused by AudioDSP. That signature is trickier to reproduce:

1. Go to a meeting link in Google Meet, and click "Join" (https://meet.google.com/XXX-XXXX-XXX)
2. Plug Apple AirPods (2nd gen in my case) into the ears so they connect to the Macbook (most likely any Bluetooth audio device would work, but I don't have any to test with)
3. Close the site (CMD+W)
4. Remove the AirPods and put them in their case (the later is important)
5. Open a new tab (CMD+T), type a website URL (e.g., google.com) and visit it
6. Browser should crash

If the browser doesn't crash, close the tab, and try opening a new tab and visit a website again.

Comment 10

[Donal Meehan [:dmeehan]]

:andreas we are at the midpoint of Fx122 beta, could you consider disabling for Fx122 and then we can keep an eye of Fx123?

Comment 11

[Andreas Pehrson [:pehrsons]]

(In reply to Mikail from comment #8)

Hi,

Is there any known workaround to this issue? I'm trying to reproduce and get details about another crash/bug with signature objc_release | CACFString::~CACFString (https://crash-stats.mozilla.org/report/index/4b8ec151-2939-4dd4-8b8f-49ad20231227, introduced somewhere in 122.0) using mozregression but I'm unable to due to the AudioDSP@0x26bf72 crashes happening since 122.0a1 (this ticket, crash report is https://crash-stats.mozilla.org/report/index/9b0e1e67-aba9-4f14-8884-31bdf0231227).

Additional details for this ticket (#1866595), reproduction steps on my end are:

1. Go to a meeting link in Google Meet (https://meet.google.com/XXX-XXXX-XXX, generate one using a Google account)
2. When firefox asks for microphone permissions, grant it
3. Firefox crashes

Thanks!

Hi Mikail, thanks for reporting on this issue. I'm actively trying to understand the failure mode here and come up with a workaround.
With what frequency do these steps reproduce a crash? What input and output device are you using to reproduce? And does using other devices (like builtin mic+speakers) affect the frequency?

Comment 12

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1866595 - Disable VOICE input on MacOS 12. r?#cubeb-reviewers

This works around some platform bugs related to the VoiceProcessingIO audio
unit on MacOS 12.

Comment 13

[Andreas Pehrson [:pehrsons]]

We'll disable the fix for bug 1670633 on MacOS 12, which should fix the crashes. It will make MacOS 12 users susceptible to the "robotic voice" drift issue again unfortunately.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e3fea180f63a
Disable VOICE input on MacOS 12. r=cubeb-reviewers,chunmin

Comment 15

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/e3fea180f63a

Comment 16

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1866595 - Disable VOICE input on MacOS 12. r?#cubeb-reviewers

This works around some platform bugs related to the VoiceProcessingIO audio
unit on MacOS 12.

Original Revision: https://phabricator.services.mozilla.com/D197659

Comment 17

[Phabricator Automation]

Uplift Approval Request

• User impact if declined: Various crashes on MacOS 12 when starting audio (microphone) capture
• String changes made/needed: None
• Needs manual QE test: no
• Is Android affected?: no
• Explanation of risk level: Restoring to pre-regression behavior for MacOS 12 only
• Risk associated with taking this patch: Low
• Fix verified in Nightly: no
• Code covered by automated testing: no
• Steps to reproduce for manual QE testing: We'll monitor crash-stats

Comment 18

[Mikail]

(In reply to Andreas Pehrson [:pehrsons] from comment #11)

(In reply to Mikail from comment #8)

Hi,

Is there any known workaround to this issue? I'm trying to reproduce and get details about another crash/bug with signature objc_release | CACFString::~CACFString (https://crash-stats.mozilla.org/report/index/4b8ec151-2939-4dd4-8b8f-49ad20231227, introduced somewhere in 122.0) using mozregression but I'm unable to due to the AudioDSP@0x26bf72 crashes happening since 122.0a1 (this ticket, crash report is https://crash-stats.mozilla.org/report/index/9b0e1e67-aba9-4f14-8884-31bdf0231227).

Additional details for this ticket (#1866595), reproduction steps on my end are:

1. Go to a meeting link in Google Meet (https://meet.google.com/XXX-XXXX-XXX, generate one using a Google account)
2. When firefox asks for microphone permissions, grant it
3. Firefox crashes

Thanks!

Hi Mikail, thanks for reporting on this issue. I'm actively trying to understand the failure mode here and come up with a workaround.
With what frequency do these steps reproduce a crash? What input and output device are you using to reproduce? And does using other devices (like builtin mic+speakers) affect the frequency?

Hi Andreas,

I'm unfortunately now unable to reproduce AudioDSP@0x26bf72 with those steps (apart from one random crash in 122.0b6 that I'm unable to reproduce, I sent it to https://crash-stats.mozilla.org/report/index/a313b112-106c-43eb-b3f3-e72e40240105). I tried 122.0b1, 122.0b2, 122.0b3, 122.0b5, and 122.0b6, no luck with those reproduction steps.
I did upgrade the OS version from 12.7.1 to 12.7.2 since my original message but I have doubts that it has relations with the issue being non-producible.

I also tried switching input/output modes by mixing them around in the MacOS audio settings but none of the combinations caused firefox to crash (I did a test matrix for input: [built-in, airpods] and output: [built-in, airpods]).

I checked if objc_release | CACFString::~CACFString is still reproducible, at the moment I'm still able to reproduce (but it isn't the ticket's original crash report):

[...] objc_release | CACFString::~CACFString (https://crash-stats.mozilla.org/report/index/4b8ec151-2939-4dd4-8b8f-49ad20231227) [...]

1. Go to a meeting link in Google Meet, and click "Join" (https://meet.google.com/XXX-XXXX-XXX)
2. Plug Apple AirPods (2nd gen in my case) into the ears so they connect to the Macbook (most likely any Bluetooth audio device would work, but I don't have any to test with)
3. Close the site (CMD+W)
4. Remove the AirPods and put them in their case (the later is important)
5. Open a new tab (CMD+T), type a website URL (e.g., google.com) and visit it
6. Browser should crash

If the browser doesn't crash, close the tab, and try opening a new tab and visit a website again.

Today's crash report was https://crash-stats.mozilla.org/report/index/bc83d43b-8f30-4dd9-bbbe-26a750240105 (firefox 122.0b6).

It is currently unclear on my end if 122.0b6 already reverted the root cause of the AudioDSP crashes and thus whether objc_release | CACFString::~CACFString crashes are actually related or separate issues?

Comment 19

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/2e9b153efbb6

Comment 21

[Andreas Pehrson [:pehrsons]]

(In reply to Mikail from comment #18)

It is currently unclear on my end if 122.0b6 already reverted the root cause of the AudioDSP crashes and thus whether objc_release | CACFString::~CACFString crashes are actually related or separate issues?

It didn't, but 122.0b7 will. If you can reproduce either of the crashes there please file a new bug.

Comment 22

[BugBot [:suhaib / :marco/ :calixte]]

Copying crash signatures from duplicate bugs.

Comment 23

[Steven Michaud [:smichaud] (Retired)]

These are still around, under a different signature, on macOS 13 and 14.

Comment 24

[Oana Botisan, Desktop QA]

I can still reproduce this issue on latest Nightly 127.0a1 on macOS 12.

• Crash signature: objc_msgSend | HALC_ShellDevice::_GetPropertyData
• Crash ID: 58dea11f-bd62-4236-9b6b-2aca90240423

Comment 25

[Andreas Pehrson [:pehrsons]]

(In reply to Oana Botisan, Desktop QA from comment #24)

I can still reproduce this issue on latest Nightly 127.0a1 on macOS 12.

• Crash signature: objc_msgSend | HALC_ShellDevice::_GetPropertyData
• Crash ID: 58dea11f-bd62-4236-9b6b-2aca90240423

Thanks. Bug 1893106 will take care of this.

Comment 26

[Steven Michaud [:smichaud] (Retired)]

The patch for bug 1893106 didn't fix these crashes.

https://crash-stats.mozilla.org/search/?modules_in_stack=~audiodsp&version=128.0b&platform=Mac%20OS%20X&date=%3E%3D2024-06-08T15%3A29%3A00.000Z&date=%3C2024-07-08T15%3A29%3A00.000Z&_facets=signature&_facets=platform_version&_facets=version&_facets=cpu_arch&_facets=proto_signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-proto_signature