1577199 - Evaluate Feature policy allow/block "speaker-selection" in WebRTC API

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, enhancement, P2)

Description

[Thomas Nguyen (:tnguyen)]

We seem have not supported "speaker" FP at the moment. If we are going to ship FP with the support of "speaker", we should evaluate to adapt the change in WebRTC API including enumarateDevices. getSinkDevice. etc..

Comment 1

[Michael Froman [:mjf]]

:jib, any thoughts here?

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

This is a TODO (later) item identified in bug 1548087.

Bug 1548087 filters out "videoinput" and "audioinput" devices from enumerateDevices() in iframes without "camera" and "microphone" feature policy respectively, but not "audiooutput" devices, because we don't support "speaker" feature policy.

If you run with media.setsinkid.enabled set to true (it is off by default) then enumerateDevices() will still expose those "audiooutput" devices in iframes by default.

Comment 3

[Sagar]

Tried this just now on meet.google.com with the pref media.setsinkid.enabled set to true on MacOS 10.15.3 (19D76) with Firefox Nightly 76.0a1 (2020-03-29) (64-bit)

It does not work for WebRTC during the call but does work for the 'Test' audio bit (likely a local file)

What's the status?

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Looks like "speaker" is an experimental feature that can already be turned on with dom.security.featurePolicy.experimental.enabled.

Andrea, if we want to make "speaker" non-experimental, would we simply move it to be under sSupportedFeatures, or is there more to it?

Comment 5

[Andrea Marchesini [:baku]]

It seems that 'speaker'' is not implemented yet:
https://searchfox.org/mozilla-central/rev/3446310d6cc5c85cde16a82eccf560e9b71a3d44/dom/security/featurepolicy/FeaturePolicyUtils.cpp#51
We should check if the code is correctly written (if written) and if we have enough tests.

See this: https://searchfox.org/mozilla-central/rev/3446310d6cc5c85cde16a82eccf560e9b71a3d44/dom/media/MediaManager.cpp#3186-3187

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Yes my question is in the context of fixing that in bug 1630289.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

This has been renamed to "speaker-selection" in the spec. There's an old patch in bug 1493982 to add it, but it might need updating to apply to the recentltly landed selectAudioOutput API.

As far as enumerateDevices goes, the new selectAudioOutput model should ensure "audiooutput" devices cannot appear ahead of that API succeeding, so no extra work should be needed there.

Comment 8

[Karl Tomlinson (:karlt)]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #7)

As far as enumerateDevices goes, the new selectAudioOutput model should ensure "audiooutput" devices cannot appear ahead of that API succeeding, so no extra work should be needed there.

Yes, checking the permissions policy for selectAudioOutput() would be sufficient to block speakers exposed via that method. However, we'll still need a mechanism to ensure that implicit consent via getUserMedia() microphones does not bypass the "speaker-selection" permissions policy.

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

/me wishes there was a way to 👍 in bugzilla

Comment 10

[Karl Tomlinson (:karlt)]

https://treeherder.mozilla.org/jobs?repo=try&revision=7977d10aa326e3613fef96640296ca958266dfba

Comment 11

[Karl Tomlinson (:karlt)]

Attachment: Bug 1577199 pass error message as well as name to top-level test window r?jib

This works better with typical errors where the name does not contain much
information.

Depends on D119606

Comment 12

[Karl Tomlinson (:karlt)]

Attachment: Bug 1577199 test speaker-selection permissions policy on selectAudioOutput() r?jib

Depends on D119862

Comment 13

[Karl Tomlinson (:karlt)]

Attachment: Bug 1577199 test speaker-selection permissions policy on setSinkId() r?jib

Depends on D119863

Comment 14

[Karl Tomlinson (:karlt)]

Attachment: Bug 1577199 Test speaker-selection permissions policy on enumerateDevices() r?jib

Mozilla will usually run enumerateDevices-permissions-policy.https.html
without media.navigator.streams.fake because fake input devices are in
different groups to the output devices. The exception is Mozilla's Mac test
machines because they have no real audio input device. With fake input
devices, there are no exposed output devices to test.

Depends on D119864

Comment 15

[Karl Tomlinson (:karlt)]

Attachment: Bug 1577199 add speaker-selection feature policy r?ckerschb

Depends on D119865

Comment 16

[Karl Tomlinson (:karlt)]

Attachment: Bug 1577199 apply speaker-selection feature policy r?jib

Depends on D119866

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2cb3d60e3b53
pass error message as well as name to top-level test window r=jib
https://hg.mozilla.org/integration/autoland/rev/3c29964b3efe
test speaker-selection permissions policy on selectAudioOutput() r=jib
https://hg.mozilla.org/integration/autoland/rev/f1c0e04d0abd
test speaker-selection permissions policy on setSinkId() r=jib
https://hg.mozilla.org/integration/autoland/rev/3506db2039ae
Test speaker-selection permissions policy on enumerateDevices() r=jib
https://hg.mozilla.org/integration/autoland/rev/ea3d6fe31e76
add speaker-selection feature policy r=ckerschb
https://hg.mozilla.org/integration/autoland/rev/2ef9aaac893a
apply speaker-selection feature policy r=jib

Comment 18

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/29862 for changes under testing/web-platform/tests

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/2cb3d60e3b53
https://hg.mozilla.org/mozilla-central/rev/3c29964b3efe
https://hg.mozilla.org/mozilla-central/rev/f1c0e04d0abd
https://hg.mozilla.org/mozilla-central/rev/3506db2039ae
https://hg.mozilla.org/mozilla-central/rev/ea3d6fe31e76
https://hg.mozilla.org/mozilla-central/rev/2ef9aaac893a

Comment 20

[Hamish Willee]

FYI, docs work for this can be tracked in https://github.com/mdn/content/issues/7746#issuecomment-898177902 .

All I have done so far is browser compat.

1. Running the enumerate devices test here https://wpt.live/audio-output/ indicates that the first test passes on FF 90.1.3 for android. Why would that be since the permission is added in FF92 according to this issue?

2. Do we know of any other implementations? Same test fails for Chrome for Android v93.

3. My understanding is that if this permission restricts access to speakers. Specifically

• MediaDevices.enumerateDevices() will not complete the promise unless the permission is granted
• MediaDevices.getUserMedia() will not offer the list of devices unless the permission is granted.
• Something undocumented called selectAudioOutput (perhaps a replacement for getUserMedia()?) won't pop up device options until the permission is granted.

Is that correct?

4. There also seems to be a restriction on HTMLMediaElement.setSinkId(). The docs say "DOMException No permission to use the requested device" - is this the permission talked about? (i.e. no docs change required here other than to list the specific permission required?

Comment 21

[Karl Tomlinson (:karlt)]

(In reply to Hamish Willee from comment #20)

Note that "audiooutput" devices, selectAudioOutput(), and setSinkId() features in Firefox are all behind the "media.setsinkid.enabled", which is default-disabled.

Even with the pref enabled, the Android implementation is incomplete.

All I have done so far is browser compat.

1. Running the enumerate devices test here https://wpt.live/audio-output/ indicates that the first test passes on FF 90.1.3 for android. Why would that be since the permission is added in FF92 according to this issue?

"enumerateDevices() returns no audiooutput devices before permission grant" with enumerateDevices-with-selectAudioOutput.https.html and Firefox 90.1.3 because the audiooutput feature is disabled.

2. Do we know of any other implementations? Same test fails for Chrome for Android v93.

I don't know of any other implementations.

Chrome doesn't support the relevant features on Android.

Chrome does have some kind of setSinkId() implementation on desktop, but some differences with the spec mean that it difficult to tell from the permissions policy tests added here whether or not it is checking permissions policy.
There is a console message "Unrecognized feature: 'speaker-selection'." implying that Chrome does not implement this.

Webkit doesn't implement the relevant features.

3. My understanding is that if this permission restricts access to speakers. Specifically

• MediaDevices.enumerateDevices() will not complete the promise unless the permission is granted

enumerateDevices() does complete without the permission grant, but returns no audiooutput devices.

• MediaDevices.getUserMedia() will not offer the list of devices unless the permission is granted.

getUserMedia() offers only camera and microphone devices and so is mostly unaffected by this speaker-selection permission policy.

• Something undocumented called selectAudioOutput (perhaps a replacement for getUserMedia()?) won't pop up device options until the permission is granted.

selectAudioOutput() will reject without a popup when disabled through permissions policy.

4. There also seems to be a restriction on HTMLMediaElement.setSinkId(). The docs say "DOMException No permission to use the requested device" - is this the permission talked about? (i.e. no docs change required here other than to list the specific permission required?

There are a number of different errors that can occur during setSinkId(). The docs are unclear at the moment. I guess this is one of the permissions. "NotAllowedError" would be more specific than "DOMException".

Comment 22

[Hamish Willee]

Thanks Karl! I'm sure I'll have other questions, but for now, selectAudioOutput() is undocumented. I'll add it to the BCD and/or experimental pages. Do you know what FF version it first appeared in?

Comment 23

[Hamish Willee]

In addition to the version of firefox that selectAudioOutput() first appeared in ^^^ , do we know whether it is in Chrome or Webkit. Does not appear to be on search, but wanted to check.
FYI,

• draft docs for selectAudioOutput() in draft at https://github.com/mdn/content/pull/7962
• first part of docs for the feature policy went in as https://github.com/mdn/content/pull/7875

Review welcome.

Comment 24

[Hamish Willee]

Ignore that. Found the bug that indicates it went in FF88.

Comment 25

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 26

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged by jgraham