880312 - [meta] Let the user switch between input devices during a WebRTC call

Status: NEW

(Core :: WebRTC: Audio/Video, enhancement)

Description

[Florian Quèze [:florian]]

It would be useful to let users switch between the built-in microphone and a headset without having to stop and restart a call.

Example use case: the user is called; accepts the call (it uses the built-in microphone which is the only available microphone at the time); the sound is poor (echo/background noise/whatever...); the user plugs a headset in. At this point it would be nice to be able to switch to it without stopping the call.

I'm not sure this needs to be exposed in the web platform, but I think it would be nice if the device could be changed during a call from the browser chrome.

Comment 1

[Jason Smith [:jsmith]]

This sounds quite similar to some of the context behind bug 827146.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

The "right" way to do this is to make a new getUserMedia call and change devices (and ask the user).  We'd also need an API to know when there are devices to change to.  Chrome will likely hide this, and if you have an active audio capture will auto-switch to headset (if available) or internal (if headset is removed).  Whether we should or not is an open question.

Note also that this requires us to be able to either handle renegotiation, OR to allow some form of indivisible remove/addstream (to form an ad-hoc replaceStream()).  I.e.

   removeStream(oldstream);
   addStream(newstream);

If this doesn't change the number of audio and video streams, we *could* interpret that as "replaceStream" - perhaps.  Likely not spec compliant, but we don't have onnegotiationeeded yet

Comment 3

[Florian Quèze [:florian]]

(In reply to Randell Jesup [:jesup] from comment #2)
> The "right" way to do this is to make a new getUserMedia call and change
> devices (and ask the user).

The idea I had in mind was to show the list of devices in the panel that appears when clicking the green camera icon that indicates a device is currently being shared, and let the user change the device there (the UI would be similar to what's displayed when the permission prompts lets the user select the device before granting access).

I don't think that requires asking the user, as it's the user that would initiate the change.

Comment 4

[Justin Dolske [:Dolske]]

From a webpage's point-of-view, would this require any API/compatibility change? That is, can the browser seamlessly switch to different stream (which might have a different resolution, fps, sample rate, etc)? [I suppose, in the worst case, the browser could resample to mimic the original stream, but that would be a mess.]

Comment 5

[Jared Wein [:jaws] (please needinfo? me)]

Can we get some more attention behind this bug. I've hit it before, and it just came up in a comment on HN (https://news.ycombinator.com/item?id=8965934).

Comment 6

[Eric Rescorla (:ekr)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #5)
> Can we get some more attention behind this bug. I've hit it before, and it
> just came up in a comment on HN
> (https://news.ycombinator.com/item?id=8965934).

The most difficult part of this is the UX design, so presumably the
priority of that work is a question for Chad.

Comment 7

[Florian Quèze [:florian]]

(In reply to Eric Rescorla (:ekr) from comment #6)
> (In reply to Jared Wein [:jaws] (please needinfo? me) from comment #5)
> > Can we get some more attention behind this bug. I've hit it before, and it
> > just came up in a comment on HN
> > (https://news.ycombinator.com/item?id=8965934).
> 
> The most difficult part of this is the UX design

Really? I thought the UX here would be just a dropdown in the device sharing panel, similar to what has been designed (but not implemented) to change the shared window during a call.

Are the platform pieces to support this already in place?

Comment 8

[Eric Rescorla (:ekr)]

(In reply to Florian Quèze [:florian] [:flo] from comment #7)
> (In reply to Eric Rescorla (:ekr) from comment #6)
> > (In reply to Jared Wein [:jaws] (please needinfo? me) from comment #5)
> > > Can we get some more attention behind this bug. I've hit it before, and it
> > > just came up in a comment on HN
> > > (https://news.ycombinator.com/item?id=8965934).
> > 
> > The most difficult part of this is the UX design
> 
> Really? I thought the UX here would be just a dropdown in the device sharing
> panel, similar to what has been designed (but not implemented) to change the
> shared window during a call.

OK, but has it been implemented? I don't see any designs.


> Are the platform pieces to support this already in place?

There's some code for this but I don't know how well it works or
whether it has been busted by the SDParta landing:

https://dxr.mozilla.org/mozilla-central/source/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#2034

Comment 9

[Jared Wein [:jaws] (please needinfo? me)]

Yeah, the UX here should not be a blocker. I don't see any questions that the pre-existing initial selection doorhanger and the screensharing work doesn't already answer.

Comment 10

[Maire Reavy [:mreavy]]

We have a choice to make about how we solve this:

1) We can rely on renegotiation.  Renegotiation is about to go up for review and we're planning to land it in Fx38.  If so, this is handled by the app, and no additional platform work is needed -- though device enumeration and preference would let the app present the device options in a better way.

2) We can use replaceTrack when the browser supports it.  Firefox currently supports this.  This option requires no support on the other side of the call (e.g. Chrome or older Firefox versions).  This would benefit from our adding device enumeration and preference, but it's not required.  We could just use the general gUM prompt. 

NOTE: It is possible to combine 1 and 2 (depending on the capabilities of the browsers involved.)

3) We can allow the user to change the device outside the app.  This would work with all apps if you are running Firefox, but would require UX design and implementation as well as platform support for changing the source of a track at the request of the UX.  This also implies that the method of initiating this change is outside of the app (not a button within the app).  So you would need to do this via the doorhanger.

Those are our options.  Which do we choose?

Comment 11

[Eric Rescorla (:ekr)]

My $.02:

1. We should not rely on renegotiation. We should expose replaceTrack to the
JS so that apps can change track inputs.

2. We should have an in-chrome way to change the input device which simply
swaps the input without bothering the application. This adds value to apps
without requiring them to explicitly support device changing.

Renegotiation is nice in future, but you shouldn't need to renegotiate to
change input devices.

Comment 12

[Martin Thomson [:mt:]]

I agree with ekr.  For applications, replaceTrack is the right API, and we already implement that.

For users, we should offer the following options:
 - stop sharing
 - switch to another device (which should ultimately include a "null" device as we've discussed elsewhere)

The only risk with an in-chrome mechanism is that it can cause breakages.  For instance, switching from a device with a hardware encoder to one without could cause downstream issues.  I wouldn't concern myself with that right now, because I don't believe that any of our platforms will encounter it just yet.

Comment 13

[Maire Reavy [:mreavy]]

I'm turning this into a meta.  Jesup will do bug 1129258.  I'm looking for an owner for bug 1129254 (on the UI side).  The UI changes we need are fairly small and contained. (Basically we just need to copy the dropdowns for camera and mic from the permission prompt doorhanger to the in-call doorhanger so the user can change inputs during a call.)

Comment 14

[Forest]

While you folks are thinking about how to do this, please make it work for audio output devices as well. When I plug in and switch to my USB headset, I don't want my caller's voice to keep playing through my PC speakers.  (A speakerphone-like experience is okay for some calls, but definitely not okay for others.)

Comment 15

[Eric Rescorla (:ekr)]

(In reply to Forest from comment #14)
> While you folks are thinking about how to do this, please make it work for
> audio output devices as well. When I plug in and switch to my USB headset, I
> don't want my caller's voice to keep playing through my PC speakers.  (A
> speakerphone-like experience is okay for some calls, but definitely not okay
> for others.)

This should actually work by default (and does in my tests). Note that
WebRTC audio output is the same as ordinary audio output in the browser,
so this isn't a WebRTC issue.

Comment 16

[Forest]

(In reply to Eric Rescorla (:ekr) from comment #15)
> (In reply to Forest from comment #14)
> > While you folks are thinking about how to do this, please make it work for
> > audio output devices as well. When I plug in and switch to my USB headset, I
> > don't want my caller's voice to keep playing through my PC speakers.  (A
> > speakerphone-like experience is okay for some calls, but definitely not okay
> > for others.)
> 
> This should actually work by default (and does in my tests). Note that
> WebRTC audio output is the same as ordinary audio output in the browser,
> so this isn't a WebRTC issue.

It didn't work for me when I tried Firefox Hello today. I'm on linux with ALSA, not PulseAudio.

Comment 17

[Eric Rescorla (:ekr)]

(In reply to Forest from comment #16)
> (In reply to Eric Rescorla (:ekr) from comment #15)
> > (In reply to Forest from comment #14)
> > > While you folks are thinking about how to do this, please make it work for
> > > audio output devices as well. When I plug in and switch to my USB headset, I
> > > don't want my caller's voice to keep playing through my PC speakers.  (A
> > > speakerphone-like experience is okay for some calls, but definitely not okay
> > > for others.)
> > 
> > This should actually work by default (and does in my tests). Note that
> > WebRTC audio output is the same as ordinary audio output in the browser,
> > so this isn't a WebRTC issue.
> 
> It didn't work for me when I tried Firefox Hello today. I'm on linux with
> ALSA, not PulseAudio.

Does it work with an HTML5 video?

Comment 18

[Forest]

> Does it work with an HTML5 video?

I'd be happy to try, if you can tell me how to select Firefox's audio output device for HTML5 videos.

My point is that always using the system default output device mostly works for things like web videos, but for conversations, people often want to plug in and use a headset.  Sometimes that's a totally separate device from the system default.  Every desktop voip application I've ever seen offers output and input device selection because the need is so very common, but I haven't found a way to do it in voip web apps running in firefox.

Comment 19

[Eric Rescorla (:ekr)]

(In reply to Forest from comment #18)
> > Does it work with an HTML5 video?
> 
> I'd be happy to try, if you can tell me how to select Firefox's audio output
> device for HTML5 videos.
> 
> My point is that always using the system default output device mostly works
> for things like web videos, but for conversations, people often want to plug
> in and use a headset.

Right, and what I am saying is that at least on my Mac, when you plug in
a headset, that becomes the system default and the audio gets routed there.


> Sometimes that's a totally separate device from the
> system default.  Every desktop voip application I've ever seen offers output
> and input device selection because the need is so very common, but I haven't
> found a way to do it in voip web apps running in firefox.

Well, I certainly want to be able to do this for videos as well.

Regardless, the first question is whether or not HTML5 videos behave differently
from WebRTC. If there is a UI affordance to change the output device for those,
then it ought to work for WebRTC. If you think there should be one but there's
not, file a separate bug.

There is also work in progress on audio output selection via JS, but that's
not done yet and is different from the kind of in-chrome affordance that
is the topic of this bug.

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Eric Rescorla (:ekr) from comment #19)
> (In reply to Forest from comment #18)
> > > Does it work with an HTML5 video?
> > 
> > I'd be happy to try, if you can tell me how to select Firefox's audio output
> > device for HTML5 videos.
> > 
> > My point is that always using the system default output device mostly works
> > for things like web videos, but for conversations, people often want to plug
> > in and use a headset.
> 
> Right, and what I am saying is that at least on my Mac, when you plug in
> a headset, that becomes the system default and the audio gets routed there.

Mac and (to a slightly lesser extent) windows switch automatically, at least for the way we use the audio channels.  In fact, it's hard to get Mac *not* to switch.

> > Sometimes that's a totally separate device from the
> > system default.  Every desktop voip application I've ever seen offers output
> > and input device selection because the need is so very common, but I haven't
> > found a way to do it in voip web apps running in firefox.
> 
> Well, I certainly want to be able to do this for videos as well.

Linux under Pulse/ALSA generally doesn't switch by default; you have to switch manually using the system UI.

> Regardless, the first question is whether or not HTML5 videos behave
> differently
> from WebRTC. If there is a UI affordance to change the output device for
> those,
> then it ought to work for WebRTC. If you think there should be one but
> there's
> not, file a separate bug.

There isn't a UI in Firefox; it depends on you using the system UI (similar to Mac/Windows).  

> There is also work in progress on audio output selection via JS, but that's
> not done yet and is different from the kind of in-chrome affordance that
> is the topic of this bug.

It's arguable that we should provide in-chrome browser switching.  There's some utility, but it adds more rarely-used options to the main UI, and for most cases system switching is easy.  The main case that doesn't handle is where you want some audio to go one place, and browser to go another place.  (Ignoring the issue of the differences in auto-switching between OSes.)

Comment 21

[Forest]

> Linux under Pulse/ALSA generally doesn't switch by default; you have to
> switch manually using the system UI.

Yep. And if you're not using Pulse, you not only have to switch it at the system level (which in many cases means editing a cryptic .asoundconf file), but also have to restart firefox in order for the change to be picked up. It's a pain for the users who know how to do it, and basically inaccessible for those who don't.

Comment 22

[Mathieu Hofman]

When looking at redoing the in-browser UI for device switching, can we also consider how to let the web application take control of this in a user friendly way?
I've filed Bug 1212996 with some ideas on how with could work.

Comment 24

[Dan Minor [:dminor]]

Clearing five year old needinfo as part of triage.

Comment 25

[Ben Wong]

As videoconferencing via WebRTC has become much more common since this bug was opened, I believe it should be considered a higher priority. Working around this bug is possible, but three different people I have spoken to have cited it as the reason “Firefox does not work”. They were not computer power users. If they had asked me, I could have told them how to go to their operating system settings. But they figured out a different solution on their own: don't use Firefox. 😥