1689741 - WebRTC - Automatic Ear Detection freezes realtime (unbuffered) video, confusing users

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

+++ This bug was initially created as a clone of Bug #1688981 +++

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36

Steps to reproduce:

1. Open one (or both) of the following webRTC test pages in Firefox with Airpods connected: https://webrtc.github.io/test-pages/src/single-audio/ or https://webrtc.github.io/test-pages/src/audio-and-video/
2. Remove your Airpods from your ears and put them back in their case, effectively disconnecting that audio source

Actual results:

1. Video freezes

Expected results:

1. Video should not freeze

Comment 1

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

When I did a mozRegression on Mac, it failed with "Unable to find enough data to bisect" here:

Tested mozilla-central build: 2020-01-03 (verdict: b)
Tested mozilla-central build: 2020-01-02 (verdict: g)

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=14ca7276922f986f08f803b28b6d0309f8a081c0&tochange=fc3002cb85dc4d2c44d8aec1246f0356097411e3

Also the detail behaviors are interesting. First the OS:

1. When I take the airpods out of my ears, macOS switches output to speakers, but airpod mic remains hot (tap them together)
2. When I put them back in my ears, macOS switches output back to airpods
3. When I put them in the case, the airPod microphone turns off

In the bad (b) build:

1. Video freezes as soon as I take the airPods out of my years (suggesting this is tied to output?)
2. If I put them back into my ears, the video resumes (!)
3. If I take them out again the video freezes again
4. If I put them in the case, the microphone track ends ("ended" printed on screen)

In the "good" (g) build:

1. Video keeps playing when I take the airPods out of my years, airpod mic remains hot (tap them together)
2. If I put them in the case, the microphone track ends ("ended" printed on screen)

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Interestingly, the video freezes and resumes exactly the same way in Safari (14.0.2 and 14.1). Chrome does not freeze. 🤷🏼‍♂️

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Turns out this is Automatic Ear Detection 😆

Alastor, is this expected behavior?

Seems confusing for real-time (unbuffered) media like a web conference. Maybe even a potential privacy issue if someone thinks they're muted in this case, when the other party can still see and hear them (verified with https://jsfiddle.net/jib1/r9x5tchq/show in two browser windows).

Should we perhaps limit this to non-realtime (as in buffered) media, like YouTube?

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(verified with https://jsfiddle.net/jib1/r9x5tchq/show in two browser windows)

Note it only pauses unmuted video elements, which was a clue. If I turn off auto ear detection they don't pause either.

I'm going to guess media controller here, either bug 1606588 or bug 1571493 from the patchlog.

Comment 5

[Andreas Pehrson [:pehrsons]]

The only downside I can really think of if we constrain MediaController pausing to only apply to non-MediaStream media providers is that the OS might automatically switch to speakers which start blaring something from an ongoing call or webrtc-based broadcast.

Comment 6

[Alastor Wu [:alwu]]

I don't have Airpods , so I'm not able to test that. But could you help me confirm that if the media element was paused by here? If so, we can simply add a constraint to avoid the media with real-time streaming won't be affected by media control.

(In reply to Andreas Pehrson [:pehrsons] from comment #5)

The only downside I can really think of if we constrain MediaController pausing to only apply to non-MediaStream media providers is that the OS might automatically switch to speakers which start blaring something from an ongoing call or webrtc-based broadcast.

Will Airpods automatically get disconnected from computer by Automatic Ear Detection? It sounds like that it just sends a media key singal to pause media, but it still keep connecting, which means the sound would still be sent to Airpods?

Comment 7

[Andreas Pehrson [:pehrsons]]

(In reply to Alastor Wu [:alwu] from comment #6)

Will Airpods automatically get disconnected from computer by Automatic Ear Detection? It sounds like that it just sends a media key singal to pause media, but it still keep connecting, which means the sound would still be sent to Airpods?

Could very well be. ni? jib to check that and the confirmation.

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Alastor Wu [:alwu] from comment #6)

I don't have Airpods , so I'm not able to test that. But could you help me confirm that if the media element was paused by here?

Confirmed in the debugger.

If so, we can simply add a constraint to avoid the media with real-time streaming won't be affected by media control.

WFM, but see below.

(In reply to Andreas Pehrson [:pehrsons] from comment #5)

The only downside I can really think of if we constrain MediaController pausing to only apply to non-MediaStream media providers is that the OS might automatically switch to speakers which start blaring something from an ongoing call or webrtc-based broadcast.

This is what happens in Chrome, which appears to have this exception for MediaStreams AFAICT. That is: ear detection still affects Youtube in Chrome, but not the demo links in comment 0 which causes audio redirected to speakers, thanks to macOS.

It's bad for those local-loop demos, and causes microphone feedback.

But demos aside, and for real WebRTC calls, this doesn't seem terrible at all. I mean I clearly can't "pause" my other participants from speaking, and having them muted by this action seems surprising. — I think I would still expect to hear them talking if I take off my AirPods mid-call. That others can hear them now even seems like a feature. I think that's the right action, and is what would happen if I finished putting them in the case anyway, or if I was using and disconnecting any other brand of headsets. If that's undesirable, I can mute the tab.

Will Airpods automatically get disconnected from computer by Automatic Ear Detection? It sounds like that it just sends a media key singal to pause media, but it still keep connecting, which means the sound would still be sent to Airpods?

The AirPods overall remain connected, and the AirPod microphone is still active, but macOS instantly redirects sound output to the MacBook Pro Speakers. E.g. With comment 0 in Chrome, if I remove the AirPods and click them together, I hear clicking sounds coming out of the laptop speakers.

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I'm marking this S2 since it represents a regression in behavior from the view of an end-user (i.e. the introduction of a desirable feature extended too far to cover areas where it is not desirable).

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

[Tracking Requested - why for this release]: Not a major issue, just causing some confusion for AirPod users in web conferencing.

Comment 11

[Julien Cristau [:jcristau]]

S3 and longstanding issue, doesn't sound like I need to track for 87.

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

we can simply add a constraint to avoid the media with real-time streaming won't be affected by media control.

Sounds like this is what we need to do (I assume you mean a condition, not a constraint). Where exactly would this go?

Let me know if you need help testing this.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Would it be possible to leave other volume controls alone, by narrowly only defeating the video pausing? This seems like it would achieve parity with Chrome here.

Comment 14

[Alastor Wu [:alwu]]

Sorry, I will start working this in next Fx release. But I'm not sure how to determine if the input stream is from real time media or not, (eg. input media stream can be captured from another media element), do we have any way to distinguish them?

Comment 15

[Paul Adenot (:padenot)]

(In reply to Alastor Wu [:alwu] from comment #14)
(eg. input media stream can be captured from another media element)

In this case, for all intents and purposes, it's considered real-time.

Comment 16

[Alastor Wu [:alwu]]

Okay, thank you!

Comment 17

[Alastor Wu [:alwu]]

Attachment: Bug 1689741 - not controlling the real-time media via the media control.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3bfc9bc50e19
not controlling the real-time media via the media control. r=padenot

Comment 19

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/3bfc9bc50e19

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:alwu, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.