1723703 - [Regression] device selection appears 3 times on the creation of a web call in some cases

Status: VERIFIED FIXED

(Firefox :: Site Permissions, defect, P2)

Description

[Gustavo Homem]

Attachment: firefox-issue.mp4

There has been a regression regarding the device selection dialog which affects at least:

• Jitsi
• Flock

and makes the device dialog selection appear:

• 3 times for Flock, one of which only includes the audio device
• 2 times for Jitsi, both with the webcam and audio device

Not affected: whereby

It appears that the problem was introduced with Firefox 90. I have searched for similar issues here in the bugtracker but didn't find any.

Attached to this bug report is a video where the problem is reproduced.

Comment 1

[Gustavo Homem]

The Flock technical support team tested this on Firefox 90 running on Windows and they were unable to reproduce the issue. Could this be Linux specific?

Comment 2

[Gustavo Homem]

I have another confirmation that this does not affect Firefox 90.0.2 on Windows. Confirmed to affect Linux by multiple users (Ubuntu 20.04).

Comment 3

[Emma Zühlcke [:emz]]

I can reproduce on Windows 10 and Linux on Nightly 92.0a1 (2021-08-03) , if I select a microphone which is not selected by default.
Tested on https://meet.jit.si/

mozregression points to Bug 1443294:

 5:14.72 INFO: Last good revision: 1c164041529d4b336d630fe829bb3533411d0e98
 5:14.72 INFO: First bad revision: 6a31940f60b784e5658dc172753d4eb84a617eb0
 5:14.72 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=1c164041529d4b336d630fe829bb3533411d0e98&tochange=6a31940f60b784e5658dc172753d4eb84a617eb0

jib, since you worked on Bug 1443294, could you take a look? Is this change in behavior expected? I remember that we had a discussion about possible breakage.

Comment 4

[Gustavo Homem]

The second confirmation I got on Windows came from a laptop that has a single audio device and a single camera. On the other hand, the Ubuntu computers often use dockstations and therefore select USB-C audio and webcams which are not the internal devices (so perhaps they are not seen as the default devices).

So, thank you Paul for the clarifications.

Comment 5

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I see what's happening. Firefox grants per-device permission, and bug 1443294 regressed a really basic case (needs two cameras or two mics):

const stream1 = await navigator.mediaDevices.getUserMedia({video: true});
const stream2 = await navigator.mediaDevices.getUserMedia({video: true});

If you pick something other than the default choice in the first prompt, then you get prompted again, with the same default choice as before (the one you didn't pick).

Is this wrong? Probably, because

• chances are high the user didn't like the default,
• same results from same input seems least surprising to the app (especially since other browsers don't have choices in their prompts), and
• fewer prompts!

If you're here about a demo site, we're sorry about this regression and you can stop reading.

If you're here about a professional WebRTC site, please read on:

Pro sites should be managing devices for their users between visits, like this:

const stream = await navigator.mediaDevices.getUserMedia({
  video: {deviceId: localStorage.camId},
  audio: {deviceId: localStorage.micId}
});
localStorage.camId = stream.getVideoTracks()[0].getSettings().deviceId;
localStorage.micId = stream.getAudioTracks()[0].getSettings().deviceId;

If they do this, then this bug shouldn't happen.

What may be happening, is we've seen sites that provoke permission prompts early:

await navigator.mediaDevices.getUserMedia({video: true}); // get the prompt over with

But this assumes the browser implements all-device permissions, which doesn't work well in Firefox. It's also a bad idea in other browsers since this may turn on the wrong camera briefly, causing its hardware light to go on for 3-10 seconds, surprising users.

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

So how to fix it. I'd rather not back out bug 1443294, since the <90 behavior combined with the new permission grace periods, was interfering with device switching.

So I think we need to improve the heuristics a bit.

The simplest band-aid might be to reinstate the old behavior only when no constraints are passed for a kind. But it wouldn't fix sites that did e.g.:

const stream1 = await navigator.mediaDevices.getUserMedia({video: {width: 1280, height: 720}});
const stream2 = await navigator.mediaDevices.getUserMedia({video: {width: 1280, height: 720}});

If the user chose a much lower resolution camera in the first prompt, should the second call also return it? What if instead the calls were:

const stream1 = await navigator.mediaDevices.getUserMedia({video: {width: 640, height: 480}});
const stream2 = await navigator.mediaDevices.getUserMedia({video: {width: 1280, height: 720}});

Should the second call still return the same low-resolution camera as the first?

There's no good answer down this path, since there's an inherent conflict between the app controlling selection and the user.

A more aggressive band-aid might be to reinstate the old behavior only when no inherent constraints (deviceId, facingMode, groupId) are passed for a kind. This might be the most pragmatic solution, since implicit choosing of multiple cameras based on non-inherent device properties like resolution isn't very practical.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

"old behavior" = immediately return the first device on the constrained list the site already has permission to, even if it's not at the top.

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #5)

What may be happening, is we've seen sites that provoke permission prompts early:

await navigator.mediaDevices.getUserMedia({video: true}); // get the prompt over with

But this assumes the browser implements all-device permissions, which doesn't work well in Firefox. It's also a bad idea in other browsers since this may turn on the wrong camera briefly, causing its hardware light to go on for 3-10 seconds, surprising users.

I forgot to spell out the workaround for this:

await sameProWayOfGettingCameraWithLocalStorage({video: true}); // get the prompt over with

This avoids asking for the wrong camera and avoids throwing away a user-choice, which avoids double-prompts, and gets you a fix immediately.

Comment 9

[Emma Zühlcke [:emz]]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #6)

So how to fix it. I'd rather not back out bug 1443294, since the <90 behavior combined with the new permission grace periods, was interfering with device switching.

Thanks for looking into this!
I agree that we shouldn't back out Bug 1443294. I'm a bit worried that, by changing the prompt heuristics, we're making it even more difficult for websites to predict FF behavior.

A more aggressive band-aid might be to reinstate the old behavior only when no inherent constraints (deviceId, facingMode, groupId) are passed for a kind. This might be the most pragmatic solution, since implicit choosing of multiple cameras based on non-inherent device properties like resolution isn't very practical.

This seems like a good compromise. In what sense do you think it's aggressive? Could this have other side effects / potentially lead to more breakage?

Would you like to take the bug?

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I don't think it's too aggressive given the trade-offs. But any heuristic will have failures.

To recap the fix: we're discussing letting permission interfere with the device selection algorithm again to some degree, which will have side effects again (especially with grace periods). Levels of aggressiveness of interference (low to high):

1. only interfere with unconstrained requests
2. only interfere with ideal (non-required) constraints on non-inherent device properties
3. interfere with all ideal (non-required) constraints
4. ignore app constraints entirely

(#3 is Firefox <=89, and #4 would be a spec violation, so those are only listed for context).

With #2

the side effect would be sites trying to choose cameras based on non-inherent device properties. I haven't seen any sites do this in practice.

E.g. sites might expect getUserMedia({video: {width: 1920}}) to consistently return a 1920 capable external camera over an internal one without that capability. But if the site already has permission to the internal one, but not the external one, they'd get the internal one (no prompt), and would have to force the 1920 requirement with getUserMedia({video: {width: {min: 1920}}}) to get the external one (with prompt).

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I think a reason sites don't attempt resolution-based camera picking is that cameras may be pointing at different things.

Comment 12

[Gustavo Homem]

Thanks for the efforts. Would proposal #2 fix the situation for Flock and Jitsi?

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1723703 - let permission interfere with device selection once again, unless inherent constraints are present. r?pbz

Comment 14

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: JitsiAsksForWrongCamera.png

(In reply to Gustavo Homem from comment #12)

Would proposal #2 fix the situation for Flock and Jitsi?

It won't fix Jitsi, because Jitsi explicitly asks for different devices than it already has. See this screenshot of the Jitsi lobby with my Logitech BRIO on, yet it's prompting for my FaceTime camera...

Logging reveals Jitsi asks for video: {deviceId: "[id of my FaceTime camera]"} which != cameraTrack.getSettings().deviceId.

The bug here seems to be Jitsi assuming it has gotten its ideal devices, which is not to spec.

I'd recommend Jitsi update its deviceId memory with track.getSettings().deviceId here. Nils, can Jitsi fix this?

Comment 15

[Gustavo Homem]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #14)

Created attachment 9238549 [details]
JitsiAsksForWrongCamera.png

(In reply to Gustavo Homem from comment #12)

Would proposal #2 fix the situation for Flock and Jitsi?

It won't fix Jitsi (I don't have access to Flock),

I can test Flock if there is a binary pre-release (Linux x64) available with this fix.

Comment 16

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Gustavo Homem from comment #15)

I can test Flock if there is a binary pre-release (Linux x64) available with this fix.

Thanks Gustavo! Linux is here. Other platforms here (Under "Artifacts" tab of the relevant Build).

Comment 17

[Gustavo Homem]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #16)

(In reply to Gustavo Homem from comment #15)

I can test Flock if there is a binary pre-release (Linux x64) available with this fix.

Thanks Gustavo! Linux is here. Other platforms here (Under "Artifacts" tab of the relevant Build).

Not fixed for Flock either. Tested with the nightly you linked above and the behaviour was the same.

Comment 18

[Gustavo Homem]

Given the situation of more than one website broken, would it be feasible to have the version 89 behaviour back and warn website owners of a future change? Or at least provide a preference to enable the old behaviour temporarily, so that teams can work smoothly until website providers have time to adapt?

Comment 19

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

would it be feasible to have the version 89 behaviour back

The problem is the 89 behavior interfered with the permission grace periods we added in 90 (see bug 1443294 comment 2). So if we back out bug 1443294, we might break camera and microphone switching on some sites.

Comment 20

[Gustavo Homem]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #19)

would it be feasible to have the version 89 behaviour back

The problem is the 89 behavior interfered with the permission grace periods we added in 90 (see bug 1443294 comment 2). So if we back out bug 1443294, we might break camera and microphone switching on some sites.

Sure, but a scenario with multiple devices is now broken on different sites as well.

How about a preference which is off by default to activate that old behaviour ON until a solution that fits all is found?

Comment 21

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

How about a preference which is off by default to activate that old behaviour ON until a solution that fits all is found?

There's already a checkbox in the prompt that users can use to avoid this: "✅ Remember this decision" (without side-effects), so a user-configurable pref seems unnecessary.

We have a contact with Jitsi. We're looking for one with Flock.

Comment 22

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I was able to add a workaround for Flock (build). With this fix, Flock only prompts once for me.

Flock is using deviceId: "default" which is a non-spec Chrome-only construct that was confusing the heuristic added in this patch.

It's worth noting however, that Flock's device management appears broken: it won't remember my device choices for next visit, even if I use their picker. Once they fix this, I recommend they follow the same advice given to Jitsi in comment 14, or the symptom may recur.

Comment 23

[Gustavo Homem]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #22)

I was able to add a workaround for Flock (build). With this fix, Flock only prompts once for me.

Great. If there is a binary build I could test and confirm.

Flock is using deviceId: "default" which is a non-spec Chrome-only construct that was confusing the heuristic added in this patch.

It's worth noting however, that Flock's device management appears broken: it won't remember my device choices for next visit, even if I use their picker. Once they fix this, I recommend they follow the same advice given to Jitsi in comment 14, or the symptom may recur.

I can talk to them once this is fixed.

Comment 24

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Sorry for the late reply. The builds are in the link in comment 23 under B of your platform, in the Artifacts sub-tab. The linux build is here.

Comment 25

[Gustavo Homem]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #24)

Sorry for the late reply. The builds are in the link in comment 23 under B of your platform, in the Artifacts sub-tab. The linux build is here.

I confirm that the build you indicated fixes the problem for Flock. Would be great if this could be released.

Thank you very much.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d9179bfe1a83
let permission interfere with device selection once again, unless inherent constraints are present. r=pbz,johannh

Comment 27

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/d9179bfe1a83

Comment 28

[Monica Chiorean]

We were not able to reproduce the issue on Nightly and Beta 91.0 on Win10, Ubuntu20.4 and Mac 11.5.2.
Can you please confirm issue is fixed on Beta 94 on your side?

Thank you.

Comment 29

[Gustavo Homem]

Hi Monica,

I had already confirmed in the comment above that I tested for indicated build and it works.

Thanks! :-)

Comment 30

[Monica Chiorean]

(In reply to Gustavo Homem from comment #29)

Hi Monica,

I had already confirmed in the comment above that I tested for indicated build and it works.

Thanks! :-)

Hi Gustavo,

I know you already checked on Nightly 94, but just to be sure that it works as expected, could you please double check on Beta 94 as well?

Thank you so much.

Comment 31

[Gustavo Homem]

(In reply to Monica Chiorean from comment #30)

(In reply to Gustavo Homem from comment #29)

Hi Monica,

I had already confirmed in the comment above that I tested for indicated build and it works.

Thanks! :-)

Hi Gustavo,

I know you already checked on Nightly 94, but just to be sure that it works as expected, could you please double check on Beta 94 as well?

Thank you so much.

Hi Monica,

I have checked on firefox-94.0b6.tar.bz2 downloaded just now and it indeed fixes the problem for Flock.

Thank you for your efforts.

Comment 32

[Monica Chiorean]

Closing based on previous comment 31.