1878380 - Non-speaking Google Meet participant appears with frozen video for periods of time

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Haik Aftandilian [:haik]]

While testing Google Meet in a 3-person meeting using Nightly on macOS 14.2.1 (23C71) on an M2 MacBook Pro 16GB, I've noticed that the non-speaking participant's video is often frozen for periods of time. I have not experienced this in testing with Google Chrome. The participant I've noticed it on was using a virtual background. I most recently hit this on Nightly on Jan 31st, 2024.

Comment 1

[Andreas Pehrson [:pehrsons]]

(In reply to Haik Aftandilian [:haik] from comment #0)

While testing Google Meet in a 3-person meeting using Nightly on macOS 14.2.1 (23C71) on an M2 MacBook Pro 16GB, I've noticed that the non-speaking participant's video is often frozen for periods of time. I have not experienced this in testing with Google Chrome. The participant I've noticed it on was using a virtual background. I most recently hit this on Nightly on Jan 31st, 2024.

The way they do image processing involves main thread so I'd say we're either looking at a main thread freeze on the sender side, or a network- and/or our libwebrtc copy's interpretation-of-network issue. Did you see it on more than the one participant, and do you know if that participant saw the same thing in their self-view? Are you able to readily reproduce this?

Best would probably be to get a profile from about:logging (WebRTC preset) from the send side while this occurs.

Comment 2

[Andreas Pehrson [:pehrsons]]

I'll just note that I saw something like this locally (would impact the remote side also) when the default system output device changed.

Haik, if you try this with a virtual background, does it match your reported symptoms timing-wise?

Here's a profile.

Comment 3

[Andreas Pehrson [:pehrsons]]

This is not gecko changing the device -- gecko has some mitigations to keep realtime clocks and events ticking if device changes are slow. This is the cubeb backend changing the device under the hood.

Comment 4

[Andreas Pehrson [:pehrsons]]

I'll mark this as a regression from bug 1670633, because the VoiceProcessingIO AudioUnit introduced there is quite a bit slower to create and destroy compared to what we had before. It should be reproducible prior to that too however, with bluetooth devices for instances.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1670633

Comment 6

[Andreas Pehrson [:pehrsons]]

Gecko should already be getting notified by cubeb when it is about to do a device switch. We can use that to start the fallback system-clock driver, since we don't know whether the device switch will be slow (which it will on macOS in duplex since bug 1670633, and possibly on all platforms if one of the devices involved is using bluetooth).

I'll try to do this in the short term. Longer term we prefer to not have cubeb be smart (doing device switches automatically under the hood) but rather signal this kind of thing to its user so decisions can be made there.

Marking S3 because once it happens it only lasts a few seconds, and because it is limited to mac except in corner cases (bluetooth).
Moving back to a webrtc component since cubeb already has an event we can use, so the short-term fix will be in gecko.

Comment 7

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1878380 - Add AudioCallbackDriver unit test. r?padenot!

This adds a test that notifies an audio driver of a device change and waits for
a second without making data callbacks. Then it checks that something else was
iterating the graph during the wait.

Comment 8

[Haik Aftandilian [:haik]]

@pehrsons, I'll work on getting some profiles. (In reply to Andreas Pehrson [:pehrsons] from comment #2)

I'll just note that I saw something like this locally (would impact the remote side also) when the default system output device changed.

Haik, if you try this with a virtual background, does it match your reported symptoms timing-wise?

The affected person was using a virtual background. Could you clarify what you mean by "default system output device changed"? Is that referring to video output device and do you mean changed during the meeting? Thanks

I'll work on collecting profiles.

Comment 9

[Andreas Pehrson [:pehrsons]]

(In reply to Haik Aftandilian [:haik] from comment #8)

@pehrsons, I'll work on getting some profiles. (In reply to Andreas Pehrson [:pehrsons] from comment #2)

I'll just note that I saw something like this locally (would impact the remote side also) when the default system output device changed.

Haik, if you try this with a virtual background, does it match your reported symptoms timing-wise?

The affected person was using a virtual background. Could you clarify what you mean by "default system output device changed"? Is that referring to video output device and do you mean changed during the meeting? Thanks

I'll work on collecting profiles.

I mean the default system audio output device, sorry if that wasn't clear. You can change it manually on the Sound page of the system Settings app, on macOS. It also changes automatically when you plug in an output device, usually. Changing it with the mic and camera captured on meet, with a virtual background applied freezes the virtual background rendering for upwards of four seconds for me. See the profile in comment 2.

Comment 10

[Andreas Pehrson [:pehrsons]]

I have a fix lined up, and here are two profiles from the respective side of a call in meet. Both have virtual backgrounds applied.
Recent Nightly: https://share.firefox.dev/3wcdWjm
Local build with fix: https://share.firefox.dev/3UyNiLr

The latter spins up a fallback driver while switching devices, so that the MediaTrackGraph keeps spinning during the switch, and currentTime of media elements keeps incrementing too.

Comment 11

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1878380 - To AudioCallbackDriver add a ChangingDevice audio stream state. r?padenot!

The new state is set when we're notified by cubeb (which is implemented only in
the mac backend) that it is changing output device under the hood. While the new
state is set, any audio callbacks coming in will be ignored, since there is no
guarantee that they won't come in after the DeviceChanged callback, but before
the device switch starts. A fallback driver is started, if not already running,
at the same time as setting the ChangingDevice audio stream state, and is
changed by the fallback driver in its first data callback to Starting. Handover
from the fallback driver to the audio driver then continues as before, and is
triggered by the first audio callback coming in after the device switch.

Comment 12

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1878380 - To MockCubebStream add a way to notify of DeviceChanged sync. r?padenot!

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a3ac160c3048
To MockCubebStream add a way to notify of DeviceChanged sync. r=padenot
https://hg.mozilla.org/integration/autoland/rev/fb1c6e6b2549
Add AudioCallbackDriver unit test. r=padenot
https://hg.mozilla.org/integration/autoland/rev/b82c1a744158
To AudioCallbackDriver add a ChangingDevice audio stream state. r=padenot

Comment 14

[Cristian Tuns]

Backed out for causing gtest failures in GraphDriver.cpp

• Backout link
• Push with failures
• Failure Log
• Failure line: TEST-PAssertion failure: !InIteration(), at /builds/worker/checkouts/gecko/dom/media/GraphDriver.cpp:1056
  gtest TEST-UNEXPECTED-FAIL | gtest | test failed with return code 2147483651

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1670633

Comment 16

[Haik Aftandilian [:haik]]

(In reply to Haik Aftandilian [:haik] from comment #8)

I'll work on collecting profiles.

I haven't been able to reproduce this. I will keep testing with Nightly once the fix goes in.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ea91ac733b54
To MockCubebStream add a way to notify of DeviceChanged sync. r=padenot
https://hg.mozilla.org/integration/autoland/rev/1b63c78343df
Add AudioCallbackDriver unit test. r=padenot
https://hg.mozilla.org/integration/autoland/rev/f40c582148d3
To AudioCallbackDriver add a ChangingDevice audio stream state. r=padenot

Comment 18

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/ea91ac733b54
https://hg.mozilla.org/mozilla-central/rev/1b63c78343df
https://hg.mozilla.org/mozilla-central/rev/f40c582148d3

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:pehrsons, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox124 to wontfix.

For more information, please visit BugBot documentation.

Comment 20

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1878380 - To MockCubebStream add a way to notify of DeviceChanged sync. r?padenot!

Original Revision: https://phabricator.services.mozilla.com/D201201

Comment 21

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1878380 - Add AudioCallbackDriver unit test. r?padenot!

This adds a test that notifies an audio driver of a device change and waits for
a second without making data callbacks. Then it checks that something else was
iterating the graph during the wait.

Original Revision: https://phabricator.services.mozilla.com/D200938

Comment 22

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1878380 - To AudioCallbackDriver add a ChangingDevice audio stream state. r?padenot!

The new state is set when we're notified by cubeb (which is implemented only in
the mac backend) that it is changing output device under the hood. While the new
state is set, any audio callbacks coming in will be ignored, since there is no
guarantee that they won't come in after the DeviceChanged callback, but before
the device switch starts. A fallback driver is started, if not already running,
at the same time as setting the ChangingDevice audio stream state. The fallback
driver changes the audio stream state back to Starting in its first data
callback after a five millisecond grace period. Then handover to the audio
driver happens as it normally would: the first data callback from the audio
driver sets the audio stream state to Running, and the next fallback data
callback after that stops the fallback driver and tells the audio driver it can
drive the graph.

Original Revision: https://phabricator.services.mozilla.com/D201195

Comment 23

[Phabricator Automation]

Uplift Approval Request

• Risk associated with taking this patch: low-medium
• Explanation of risk level: The affected code is fairly complex, but we re-use a path that is well tested, and have a test exercising it in CI.
• User impact if declined: Google meet's self-view (the video which is sent to other participants) will freeze for a few seconds when changing default output device.
• String changes made/needed: None
• Is Android affected?: no
• Needs manual QE test: yes
• Code covered by automated testing: yes
• Steps to reproduce for manual QE testing: On macOS, open google meet, enable a video background, change default sound output device in system settings.
• Fix verified in Nightly: no

Comment 24

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/a0bf0522a7d6
https://hg.mozilla.org/releases/mozilla-beta/rev/7267784a2655
https://hg.mozilla.org/releases/mozilla-beta/rev/493a988dc9bd

Comment 25

[Catalin Sasca, Desktop Test Engineering [:csasca]]

We were able to reproduce freezes of around 2 seconds when changing audio outputs on Firefox Nightly 124.0a1 (2024-02-02).

On Firefox 125.0a1 (2024-02-26) while there are three or more participants sometimes we can some see slight stutters (a split second) when changing audio outputs but it is way smoother than the affected build. :jib, are those stutters expected? (Asking you as :pehrsons is in PTO until March 4th).

Comment 26

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

What do you mean "stutters"? In audio or video?

If you mean the video freezes have reduced from 2 seconds to a split second that seems like a win. But it's hard to say without more information

Comment 27

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Based on comment 10 I'd say a split second of video freeze when virtual background is used is expected.

Comment 28

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Yep, the stutters were encountered in video.

That being said we've verified that the ~2 second freezes were no longer encountered on Firefox 124.0b4 and Firefox 125.0a1 (2024-02-26). Tests were performed on two different macOS Arm devices running macOS 14.3.1 and 14.2.1.

Comment 29

[Paul Adenot (:padenot)]

Catalin, thanks for the thorough verification! Would you mind reproducing this, capturing stutters with the Firefox Profiler using the WebRTC preset, as shown in this video: https://blog.paul.cx/post/profiling-firefox-media-workloads/#the-media-preset (this uses the media preset, please use the webrtc preset).

This isn't expected, so thanks for noticing it. Let me know if I can help with the profiler.

Comment 30

[Alexandru Trif, Desktop Test Engineering [:atrif]]

(In reply to Paul Adenot (:padenot) from comment #29)

Catalin, thanks for the thorough verification! Would you mind reproducing this, capturing stutters with the Firefox Profiler using the WebRTC preset, as shown in this video: https://blog.paul.cx/post/profiling-firefox-media-workloads/#the-media-preset (this uses the media preset, please use the webrtc preset).

This isn't expected, so thanks for noticing it. Let me know if I can help with the profiler.

Hello! The split-second video stutter reproduces on macOS M1 mini 14.2.1 when I remove the jack headphones from the machine. Profiler link. If more information is needed please let us know. Thank you!

Comment 31

[Paul Adenot (:padenot)]

Brilliant, thanks! I'll let :pehrsons look at the profile when he's back next week. In light of the profile, this might be expected, but he'll know for sure.

Comment 32

[Andreas Pehrson [:pehrsons]]

That's expected, thanks. The "stutter" is a pause in video while we wait for the thread that runs the fallback driver to spawn. The fallback driver is what drives the graph (i.e. real-time video) until the audio driver is running again. The "stutter" I've seen has been <0.5s whereas the audio driver reconfig (this bug) can take multiple seconds.