1874789 - Apple Studio Display microphone extremely distorted

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect, P2)

Description

[Ben Limmer]

Attachment: ff_122_0b9.mp3

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:122.0) Gecko/20100101 Firefox/122.0

Steps to reproduce:

I noticed this on Google Meet, but the issue is reproducible for me on any site that records audio.

Steps to reproduce:

1. Use Firefox beta (122.0b9 in my case)
2. Visit https://online-voice-recorder.com . The issue isn't specific to this site, but it's easy to show the problem.
3. Click record and use the Apple Studio Display microphone.
4. Record some audio.
5. Play back the audio in the browser, or download the file.

This issue happens consistently in Firefox Beta, even in "safe mode" and a clean profile. I've also reinstalled the app (overwrite /Applications/Firefox.app) and the issue persists.

This issue does not happen:

• In the release version of Firefox (121.0.1)
• When using other input devices (Macbook Pro microphone or Bose QC 45 headphones)
• In Google Chrome (120.0.6099.216)
• In other applications (e.g., Zoom, Loom, etc)

Actual results:

Upon playback, the audio is extremely distorted.

Expected results:

The audio should be clear, as with other apps.

Comment 1

[Ben Limmer]

Attachment: ff_121_0_1.mp3

Comment 2

[Ben Limmer]

Attachment: ff_122_0b9_bose_headphones.mp3

Comment 3

[Ben Limmer]

Attachment: chrome_120_0_6099_216.mp3

Comment 4

[Ben Limmer]

Attachment: media_about_support.json

Comment 5

[Ben Limmer]

Attachment: media.json

Attaching media section of about:support

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 7

[Ben Limmer]

Now that 122.0 hit stable, I re-tested this and still have the same issue. So this does appear to be a regression between 121.0 and 122.0.

Comment 8

[Andreas Pehrson [:pehrsons]]

Thanks for the report! Could you help me by gathering some logs, please?

• Go to about:logging, select the WebRTC preset, click Start Logging
• In another tab reproduce the issue
• Back on about:logging, click Stop Logging
• A new tab will open with the Firefox Profiler. On the top right, find the button to upload the profile, and include hidden threads. Then share the link here.

I'm marking this as a regression from bug 1670633 because it seems most likely, but we'll update as needed later.

Comment 9

[Mike E.]

Attachment: Firefox 2024-01-31 11.13 profile.json.gz

Comment 10

[Mike E.]

I have the same issue, here is the output of my Firefox Profiler
https://bugzilla.mozilla.org/attachment.cgi?id=9377447

(In reply to Andreas Pehrson [:pehrsons] from comment #8)

Thanks for the report! Could you help me by gathering some logs, please?

• Go to about:logging, select the WebRTC preset, click Start Logging
• In another tab reproduce the issue
• Back on about:logging, click Stop Logging
• A new tab will open with the Firefox Profiler. On the top right, find the button to upload the profile, and include hidden threads. Then share the link here.

I'm marking this as a regression from bug 1670633 because it seems most likely, but we'll update as needed later.

Comment 11

[Ben Limmer]

Here's the requested profiler info: https://share.firefox.dev/3vY7so6 . Thanks!

Comment 12

[Andreas Pehrson [:pehrsons]]

I could not reproduce this with the virtual blackhole-16ch device, but my hunch is that this is related to us configuring the output device for 8ch, and that it's the same as the input device.
I have a speculative fix on try https://treeherder.mozilla.org/jobs?repo=try&revision=9ca9c9ca4c1290da62fbc31eec122fcc2a5958f7. I'll post a link to the build when it's ready or you could go there and dig it out (click the B for a build config when it's green, and dig it out among the artifacts. You're looking for target.dmg)

Comment 13

[Mike E.]

I did find there is no issue when using a Logitec USB webcam mic. it seems specific to the Apple Studio Display built in mic.

Comment 14

[Ben Limmer]

I just tested the build from your link and the issue persists. Here are profiler logs from that build: https://share.firefox.dev/48Up1Ut

Comment 15

[Andreas Pehrson [:pehrsons]]

Thanks for the feedback. I'll have to attribute it to this specific microphone then, and not how voice processing interacts with the output. I'll try to get my hands on one of these but it could take a little while. If anyone is interested in looking into the code that sets this up themselves, I can give some pointers on where to start.

Comment 16

[Ben Limmer]

Although I'd be starting from "zero" with this codebase, I am a developer, so I might be able to help. Is there any additional debugging data I could provide to help narrow down the issue?

Comment 17

[Andreas Pehrson [:pehrsons]]

(In reply to Ben Limmer from comment #16)

Although I'd be starting from "zero" with this codebase, I am a developer, so I might be able to help. Is there any additional debugging data I could provide to help narrow down the issue?

Thanks, I'll give you some pointers.

We use libcubeb for audio, the macOS backend is cubeb-coreaudio-rs. You'll need rust to build it with cargo build. When running tests on macOS 14 there are some platform issues that may cause hangs so easiest to linearize them with RUST_TEST_THREADS=1 ./run_tests.sh.

As a starting point you can use the vpio-mono-output-logging branch. It's the basis for the build in comment 12, plus verbose logging enabled in the manual test_streams_tester test case. For playing around a bit, it's the easiest to use: cargo test test_stream_tester -- --ignored --nocapture.

I'd be most suspicious of where we request mono from the input or where we interpret input data.

Comment 18

[Ben Limmer]

Attachment: test-output.txt

I didn't go so far as putting breakpoints or stepping through the code, but I did run the test suite from your branch (see attached test output).

I also played ran the stream tester, but I'd need to dedicate some more time to figuring out how it works and reviewing the codebase. I'm not sure if any of this information here is helpful or not.

  cargo test test_stream_tester -- --ignored --nocapture
warning: lint `private_in_public` has been removed: replaced with another group of lints, see RFC <https://rust-lang.github.io/rfcs/2145-type-privacy.html> for more information
    --> src/backend/mod.rs:3752:24
     |
3752 | #[cfg_attr(test, allow(private_in_public))]
     |                        ^^^^^^^^^^^^^^^^^
     |
     = note: `#[warn(renamed_and_removed_lints)]` on by default

warning: function `test_disable_log` is never used
  --> src/backend/tests/utils.rs:17:8
   |
17 | pub fn test_disable_log() {
   |        ^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: `cubeb-coreaudio` (lib test) generated 2 warnings
    Finished test [unoptimized + debuginfo] target(s) in 0.02s
     Running unittests src/lib.rs (target/debug/deps/cubeb_coreaudio-a185629f92f4df5c)

running 1 test
commands:
	'q': quit
	'c': create a stream
	'd': destroy a stream
	's': start the created stream
	't': stop the created stream
	'r': register a device changed callback
	'l': set loopback (DUPLEX-only)
	'v': set volume
	'm': set input mute
	'p': set input processing
c
Select stream type:
1) Input 2) Output 3) In-Out Duplex 4) Back
1
Select input device:

   0: System default
   1: 00AF2102-97A5-4995-A66D-BB1500000003
	AudioObjectID: 111
	uid: iPhone Microphone
   2: AppleUSBAudioEngine:Apple Inc.:Studio Display:00008030-001019A21EB9802E:6,7
	AudioObjectID: 57
	uid: Studio Display Microphone
   3: BuiltInMicrophoneDevice
	AudioObjectID: 106
	uid: MacBook Pro Microphone
   4: com.loom.desktop.audio-device.device
	AudioObjectID: 85
	uid: LoomAudioDevice
2
mod.rs:2328: Use global latency 512 instead of the requested latency 4096.
mod.rs:2702: Input device ID: 57 (aggregate: false)
mod.rs:2872: (0x124b04380) Initializing input by device info: device_info { id: 57, flags: device_flags(DEV_INPUT) }
mod.rs:2890: (0x124b04380) Opening input side: rate 48000, channels 1, format Float32LE, layout FRONT_CENTER, prefs VOICE, latency in frames 512, voice processing false.
mod.rs:2920: (0x124b04380) Input hardware description: AudioStreamBasicDescription { mSampleRate: 48000.0, mFormatID: 1819304813, mFormatFlags: 9, mBytesPerPacket: 4, mFramesPerPacket: 1, mBytesPerFrame: 4, mChannelsPerFrame: 1, mBitsPerChannel: 32, mReserved: 0 }
mod.rs:1294: The buffer frame size of AudioUnit 0x92bc00ae for INPUT is already 512
mod.rs:3040: (0x124b04380) Input audiounit init with device 57 successfully.
cubeb_resampler_internal.h:521:Input and output sample-rate match, target rate of 48000Hzmod.rs:2393: (0x124b04380) Cubeb stream init successful.
Stream 0x124b04380 created.
commands:
	'q': quit
	'c': create a stream
	'd': destroy a stream
	's': start the created stream
	't': stop the created stream
	'r': register a device changed callback
	'l': set loopback (DUPLEX-only)
	'v': set volume
	'm': set input mute
	'p': set input processing
t
state: Stopped
mod.rs:4090: Cubeb stream (0x124b04380) stopped successfully.
Stream 0x124b04380 stopped.
commands:
	'q': quit
	'c': create a stream
	'd': destroy a stream
	's': start the created stream
	't': stop the created stream
	'r': register a device changed callback
	'l': set loopback (DUPLEX-only)
	'v': set volume
	'm': set input mute
	'p': set input processing
test backend::tests::manual::test_stream_tester has been running for over 60 seconds
q
Quit.
mod.rs:4039: Cubeb stream (0x124b04380) destroyed successful.
Stream 0x124b04380 destroyed.
test backend::tests::manual::test_stream_tester ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 198 filtered out; finished in 77.19s

Assertion failed: (logging_thread.get_id() == std::thread::id() && (g_cubeb_log_callback == cubeb_noop_log_callback || !g_cubeb_log_callback)), function ~cubeb_async_logger, file cubeb_log.cpp, line 146.

Sorry I haven't been of more help on this, I haven't had much free hacking time lately.

Comment 19

[Andreas Pehrson [:pehrsons]]

No worries, thanks for the help so far.
I have access to one of these displays now so I'll take a look shortly.

Comment 20

[Andreas Pehrson [:pehrsons]]

It appears that without the platform AGC the signal goes completely out of bounds. I had done some simple testing of the AGC before and it appeared to not be able to amplify the signal if it is weak, therefore I turned it off. It appears we need it to avoid clipping. I'll put up a fix.

Comment 21

[Andreas Pehrson [:pehrsons]]

Here are some test builds if you want to give it a spin before hitting Nightly: https://treeherder.mozilla.org/jobs?repo=try&revision=8ef7d10e47886739aca7469ac00412501fc27b69

Comment 22

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1874789 - Update cubeb-coreaudio-rs to d777208b31. r?cubeb-reviewers!

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ee5e8cc2e91d
Update cubeb-coreaudio-rs to d777208b31. r=cubeb-reviewers,supply-chain-reviewers,padenot

Comment 24

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/ee5e8cc2e91d

Comment 25

[Andreas Pehrson [:pehrsons]]

I hear reports of other input devices exaggeratingly amplifying low-volume background noises with this fix. So rather than impacting all devices with that we'd rather live with this clipping issue on the studio display only. I'll ask to have this backed out. I can instead land a pref to allow disabling the VoiceProcessingIO audio unit added in bug 1670633, which should revert behavior to what we had in 121.

Comment 26

[Ben Limmer]

Is there a way to flip the switch automatically based on the characteristics of the Studio Display? As a frequent user of Google Meet, this bug has been quite painful to work around. I can certainly flip the switch myself, but others might not discover this.

Comment 27

[Natalia Csoregi [:nataliaCs]]

Backed out as requested, for causing audio issues.

• backout: https://hg.mozilla.org/integration/autoland/rev/7e33bd78743f2406eaef5b58b833a41ab2205085

Comment 28

[Sandor Molnar[:smolnar]]

Backout merged to central:

https://hg.mozilla.org/mozilla-central/rev/7e33bd78743f

Comment 29

[Andreas Pehrson [:pehrsons]]

(In reply to Ben Limmer from comment #26)

Is there a way to flip the switch automatically based on the characteristics of the Studio Display? As a frequent user of Google Meet, this bug has been quite painful to work around. I can certainly flip the switch myself, but others might not discover this.

I have put a blocklist in the backend also.

Comment 30

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1874789 - Update cubeb-coreaudio-rs to d23ab55eab. r?#cubeb-reviewers!

This update introduces a blocklist function for using VPIO, and
makes the platform AGC settable by param.

Comment 31

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1874789 - Add a pref to allow disabling VOICE in cubeb. r?padenot!

Intended as an escape valve and diagnosis tool in case someone has a device
setup that is troublesome with VPIO on macOS.

Comment 32

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1874789 - Update cubeb-coreaudio-rs to d23ab55eab. r?#cubeb-reviewers!

This update introduces a blocklist function for using VPIO, and
makes the platform AGC settable by param.

Original Revision: https://phabricator.services.mozilla.com/D202404

Comment 33

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1874789 - Add a pref to allow disabling VOICE in cubeb. r?padenot!

Intended as an escape valve and diagnosis tool in case someone has a device
setup that is troublesome with VPIO on macOS.

Original Revision: https://phabricator.services.mozilla.com/D202405

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f0ab0afca52f
Update cubeb-coreaudio-rs to d23ab55eab. r=cubeb-reviewers,supply-chain-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/80e08d441c6c
Add a pref to allow disabling VOICE in cubeb. r=padenot

Comment 35

[Phabricator Automation]

Uplift Approval Request

• Is Android affected?: no
• Code covered by automated testing: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Fix verified in Nightly: no
• String changes made/needed: None
• User impact if declined: Users on macOS using Apple's Studio Display for both audio input and output will capture distorted audio input
• Risk associated with taking this patch: low
• Explanation of risk level: This blocks uses of Apple Studio Display as input and output from using the fix from bug 1670633, effectively using the behavior prior to that bug. Also adds a pref but without affecting default behavior.

Comment 36

[Sandor Molnar[:smolnar]]

Backed out for causing assertion failures at media.getusermedia.microphone.prefer_voice_stream_with_processing.enabled

Backout link: https://hg.mozilla.org/integration/autoland/rev/0f11d5aca9e02300b1aa70ff8c42fa6456d80ffb

Push with failures

Failure log -> Assertion failure: IsAtomic<bool>::value || NS_IsMainThread() (Non-atomic static pref 'media.getusermedia.microphone.prefer_voice_stream_with_processing.enabled' being accessed on background thread by getter)

Comment 37

[Paul Adenot (:padenot)]

Attachment: Bug 1874789 - Make the new pref atomic.

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a30f7bce38fe
Update cubeb-coreaudio-rs to d23ab55eab. r=cubeb-reviewers,supply-chain-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/ab4d27a3e7d9
Add a pref to allow disabling VOICE in cubeb. r=padenot
https://hg.mozilla.org/integration/autoland/rev/4c97b568c14a
Make the new pref atomic.

Comment 39

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/a30f7bce38fe
https://hg.mozilla.org/mozilla-central/rev/ab4d27a3e7d9
https://hg.mozilla.org/mozilla-central/rev/4c97b568c14a

Comment 40

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit BugBot documentation.

Comment 41

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/659336962cb4
https://hg.mozilla.org/releases/mozilla-beta/rev/b28d335f4dbc

Comment 42

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/d342f05c644b

Comment 44

[Ben Limmer]

Confirmed this is fixed on 124.0b8! Thank you