1729889 - Screen selection broken using WebAPI with Firefox 93+

Status: RESOLVED WONTFIX

(Core :: WebRTC: Audio/Video, defect)

Description

[Ruben Esteve]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36

Steps to reproduce:

The following code snippet doesn't work to select screen device.
navigator.mediaDevices.getUserMedia({ video: { mediaSource: 'screen' } })

Apparently the constraint object field is case sensitive (MediaSource).
The following does work:
navigator.mediaDevices.getUserMedia({ video: { MediaSource: 'screen' } })

Actual results:

Console error:
Uncaught (in promise) DOMException: Display capture requires transient activation from a user gesture.

Expected results:

Screen selection window pops up.

Comment 1

[Ruben Esteve]

Steps to reproduce:

The following code snippet doesn't work to select screen device.
navigator.mediaDevices.getUserMedia({ video: { mediaSource: 'screen' } })

Actual results:

Console error:
Uncaught (in promise) DOMException: Display capture requires transient activation from a user gesture.

Expected results:

Screen selection window pops up.

Comment 2

[Ruben Esteve]

Last comment stands. As a matter of fact there was no code to make it working, but the constraint was not processed when using a non-existing field MediaSource. Therefore default camera device came up instead.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::DOM: Core & HTML' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 4

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Base on hg log history, I think WebRTC: Audio/Video might be a proper component, feel free to bounce it back if I was wrong.

Comment 5

[Michael Froman [:mjf]]

I can confirm this in 94, and also that it does not happen in 91 or 92.

Comment 6

[Michael Froman [:mjf]]

mozregression produces: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=3778bee8e0f8081a70782163fc54d238ebee21a1&tochange=3d86317782d298b98be58646114bebba21611014

Bug 1705289 - Karl, is it expected that different case gives different results?

Comment 7

[Karl Tomlinson (:karlt)]

Thank you for the clear report.

This was an expected behavior change to align the behavior of the legacy experimental mediaSource: 'screen' API with that of the standard-track getDisplayMedia() API. getDisplayMedia() requires a recent user gesture (transient activation) to provide some evidence that the user is actively engaged with the document at the time the request for display media is presented, which is a requirement introduced due to the potentially sensitive nature of the data exposed.

This is consistent with the behavior intended for Safari. Does your use case have a solution that works with WebKit browsers?

(In reply to Michael Froman [:mjf] from comment #6)

Bug 1705289 - Karl, is it expected that different case gives different results?

Yes, the dictionary member keys are case-sensitive. See comment 2, which corrected comment 0.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1705289

Comment 9

[Ruben Esteve]

Thanks for the clarification Karl.
The problem for us is that we are executing that code snippet directly instead of it being triggered from an interaction with the UI.
We'll try to work around it, but is there any Firefox flag that could disable this new restriction?

Comment 10

[Karl Tomlinson (:karlt)]

(In reply to Ruben Esteve from comment #9)

is there any Firefox flag that could disable this new restriction?

There is no option for the user to bypass the transient activation check, sorry.
(Test frameworks using webdriver typically use a click for this, but I assume that is not what you are wanting.)

Comment 11

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Karl, should the state of the bug be moved to INVALID? Since it is an expected behavior, can the regression flag be removed?

Comment 12

[Ruben Esteve]

I guess this bug can be closed then. We'll have to find a solution in our test harness along the lines of what's been suggested (using a click or something). Thanks.

Comment 13

[Karl Tomlinson (:karlt)]

I think the regression flag is appropriate as the bug is describing what appears to be a regression from some perspective.

Technically I guess INVALID is specified because intended behavior is not considered a bug. However I'd prefer WONTFIX for a valid report on a breaking change in behavior with an explicit decision not to fix. (We might still reconsider this decision if a serious break in a common use case were to come to light.)

For testing, we also have SpecialPowers.wrap(document).notifyUserGestureActivation(). I don't know exactly how our test harness enables SpecialPowers

Comment 14

[Ruben Esteve]

Karl, is there any way we can also use SpecialPowers? Is it a publicly available library?

Comment 15

[Karl Tomlinson (:karlt)]

SpecialPowers is loaded as an extension. The code is open source and so I assume it could be extracted and used in your Firefox testing, but I'm not aware of recent cases of people doing this.

I suspect WebDriver/geckodriver/Marionette would provide a more versatile, and probably better supported, testing framework, which would include click support, making SpecialPowers unnecessary for this purpose.

At the other end of options, the "dom.user_activation.transient.timeout" pref might be set to a large number so that a single click from a user starting a test might be sufficient for all getUserMedia() calls occurring in a single document. Another user interaction, however, would be required for each document loaded.