1749790 - Underruns on macOS when doing calls

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect, P2)

Description

[Paul Adenot (:padenot)]

I'm not sure why or what happens, but I've had a call (Google Meet, 3 people including myself), and there were audio glitches. I'm running big sur on a 2018 macbook pro.

Profiling, I saw the callback in the child being delayed (no calls for about half the callback interval time), but we don't have markers in the actual callback in the parent, so I'm not sure what's up. I also lost the profile somehow.

:jrmuizel also can repro, maybe he has a profile.

My build was from: https://hg.mozilla.org/mozilla-central/rev/8bc2581b2c7bb99a1138ece1f6e7bf80ff7f79d3, with audioipc enabled on macOS, but fairly old.

Comment 2

[Jeff Muizelaar [:jrmuizel]]

here's a profile that should include the time when the crackling was happening: https://share.firefox.dev/3qnOVfL

Comment 3

[Paul Adenot (:padenot)]

Same as I saw, AudioIPC Client Callback shows a clear irregularity of callbacks. Something is off, the ipc layer seem to delay some callbacks.

Comment 4

[Matthew Gregan [:kinetik]]

I'm still investigating this - I haven't been able to reproduce locally yet. Jeff's profile seems to be missing the AudioIPC Server * threads in the parent, although as noted we don't have markers in those threads so stacks may not add much debugging info for this issue.

Jeff and Paul - what audio hardware were you using during the call?

Bug 1750282 may be another instance of the same issue.

If it's reproducible in an --enable-debug build, `MOZ_LOG=timestamp,sync,cubeb:5,audioipc2::*:5" may give some clues for debugging.

Comment 5

[Jeff Muizelaar [:jrmuizel]]

I was seeing this on the built-in audio hardware on a 2019 MBP with macOS 10.15

Comment 6

[Paul Adenot (:padenot)]

I was using the TRRS plug of the macbook, for both microphone and headphones.

Comment 7

[Paul Adenot (:padenot)]

(In reply to Matthew Gregan [:kinetik] from comment #4)

I'm still investigating this - I haven't been able to reproduce locally yet. Jeff's profile seems to be missing the AudioIPC Server * threads in the parent, although as noted we don't have markers in those threads so stacks may not add much debugging info for this issue.

There's some ideas for adding markers in rust crates that end up in the Firefox Profiler marker view, using https://crates.io/crates/profiling or https://crates.io/crates/tracing, we should probably do it.

Comment 8

[Matthew Gregan [:kinetik]]

I've got a patch to add markers using the Rust API for Gecko's built-in markers, I'll finish that up very soon.

Comment 9

[Matthew Gregan [:kinetik]]

I can reproduce something like this in a debug build with trace logging enabled (but possibly because the logging overloads our RT budget, will verify), but not in a regular Nightly build so far.

Comment 10

[Paul Adenot (:padenot)]

Maybe the easiest to repro would be to play a sine wave on its own via Web Audio, because this is going to reduce the latency used for the audio stream (128 frames for output-only vs. 512 frames for input/output w/ audio input processing, on macOS), and make any glitch obvious and more likely to happen.

https://github.com/padenot/fx-profiler-audio-cb and https://blog.paul.cx/post/profiling-firefox-real-time-media-workloads/ might be useful to compare no-sandbox vs. sandboxed audio, or validate various fixes.

AudioIPC Server * threads should be included, because "audio" is present in the media profiler preset list, and the profiler is doing case-insensitive substring match.

I'm thinking that maybe we're not prioritizing a particular thread to real-time?

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1425788

Comment 12

[Matthew Gregan [:kinetik]]

(In reply to Paul Adenot (:padenot) from comment #10)

Maybe the easiest to repro would be to play a sine wave on its own via Web Audio, because this is going to reduce the latency used for the audio stream (128 frames for output-only vs. 512 frames for input/output w/ audio input processing, on macOS), and make any glitch obvious and more likely to happen.

https://github.com/padenot/fx-profiler-audio-cb and https://blog.paul.cx/post/profiling-firefox-real-time-media-workloads/ might be useful to compare no-sandbox vs. sandboxed audio, or validate various fixes.

Will experiment further with this, thanks for the pointers.

AudioIPC Server * threads should be included, because "audio" is present in the media profiler preset list, and the profiler is doing case-insensitive substring match.

Not sure what's going on here, the server threads show up for me.

I'm thinking that maybe we're not prioritizing a particular thread to real-time?

AudioIPC Server Callback and AudioIPC Client Callback are both running at real-time priority - they show up in Instruments (and ps -M) as priority 97.

Comment 13

[Pablo]

I am also hearing a crackling noise when testing with latest beta after i press the button to join a web conference, for 1 second and then it stops using Webex

https://cart.webex.com/sign-up-webex

-select the button "start a meeting"
-select join from browser
-click on green button "start meeting"
webex has its own music when a meeting starts, and for the first second it sounds crackling.

Comment 14

[Jim Mathies [:jimm]]

[Tracking Requested - why for this release]:
We need to disable audioipc for macos in 97 due to this.

Comment 16

[Jeff Muizelaar [:jrmuizel]]

Interestingly I'll hear this crackling when playing music in YouTube Music in Chrome and a muted video ad in Firefox. Pausing the muted video makes the crackling go away.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

The regressing feature was disabled in 97.0b7.

Comment 18

[Matthew Gregan [:kinetik]]

I have access to a machine where this is easily reproducible now, so hoping for quicker progress on this shortly.

Comment 19

[Matthew Gregan [:kinetik]]

Working on a potential fix - will post test builds soon.

Comment 20

[Matthew Gregan [:kinetik]]

There's an experimental build available at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TUMtc6EsSh-QJn-1IaIJLQ/runs/0/artifacts/public/build/target.dmg that improves the situation significantly for me. I'd love to get feedback on whether that build improves the situation for anyone seeing this issue.

Comment 21

[Matthew Gregan [:kinetik]]

Fixed by the landing of bug 1817043 and dependencies.