1321221 - Implement getDisplayMedia for screen capture to comply with spec changes

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

As discussed in bug 1313758 comment 8, we currently implement an outdated draft [2] of the spec.

Comment 1

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

In the latest Screen Capture draft (Draft 27), the constraint is reduced to a simple boolean, app won't be able to indicate the surface type, then we have only two choices:

1) Let user choose the surface type; then let user choose the window / screen / applicationto share.
2) Always provide one type of surface type.

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

In hindsight, I think an obstacle to implementing the spec here is that our current implementation allows passing in constraints to downscale the resolution and decimate frame rates from the very first frame. Constraints do not filter the selection list one bit.

We consider this a useful feature, since desktops tend to be of extremely large resolution, and their high frame rate to be overkill for most WebRTC use.

The spec on the other hand disallows constraints on getDisplayMedia, except on subsequent calls to track.applyConstraints presumably.

This presumably means JS would need to expect the full-size full-frame-rate stream first, and then issue

    await track.applyConstraints({width, height, frameRate})

on the video track before attaching it to a sink.

Martin, does this match your intent? Do you have any guidance on how to implement this in a performant manner? Should we be concerned that this may cause capture to fail in situations where returning the unscaled result might fail where a downscaled version might not?

Comment 3

[Martin Thomson [:mt:]]

Can you take that as an issue to the screen sharing repo.  The intent behind removing the constraints was primarily to avoid the constraints being used to guide selection.  No one had a use case that they could articulate for other constraints, so removing them entirely was easier.  Adding them back isn't impossible.

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Filed https://github.com/w3c/mediacapture-screen-share/issues/49

Comment 5

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #2)
> The spec on the other hand disallows constraints on getDisplayMedia, except
> on subsequent calls to track.applyConstraints presumably.

This was resolved at the April interim. See https://github.com/w3c/mediacapture-screen-share/issues/49#issuecomment-381556471

Just waiting on a PR to update the spec.

Comment 8

[wilhelm.wanecek]

In light of google's (fairly recent) announcement to implement getDisplay media (Chromestatus: https://www.chromestatus.com/feature/6744724455030784 Discussion: https://bugs.chromium.org/p/chromium/issues/detail?id=326740#c35), it would be _fantastic _ if Firefox was able to align their implementation sooner rather than later :)

Is this still only blocked on the PR to update the spec? :) Is there anyone who is willing to devote some time and effort to unblocking this? I'm afraid I'm completely inexperienced in spec-writing.

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I'm slated to work on getDisplayMedia after I add the resizeMode constraint. Given the timing and recent interest, and depending on spec progress, I may start them in parallel.

I'm working with Suhas to finish off the spec, from the issues resolved at the April w3c meeting, including the issue in comment 4. The best overview of that remaining spec work is probably the slides [1]. Suhas and I just discussed this yesterday, and he will be providing PRs.

[1] https://docs.google.com/presentation/d/1sboQqMc5O1-oZ7w0ZY1VUyhY8JG0VxOzU8kAQitoHJU/edit#slide=id.g2c521efd37_0_20

Comment 10

[wilhelm.wanecek]

Great news! Thanks for the update :) Is any extra effort required to coordinate with the Chrome and Edge teams, e.g. in regards to the resizeMode and/or changes to the spec?

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I'm pushing this to the front of my queue.

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I'm working with Suhas to finish off the spec.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Intent to implement and ship: https://groups.google.com/forum/#!topic/mozilla.dev.platform/20EhEy_ahKg

Comment 14

[Luca Barbato]

You might mention this issue in https://developer.mozilla.org/en-US/docs/Web/API/Screen_Capture_API/Using_Screen_Capture#Simple_screen_capture since apparently even the current 65.0b4 does not seem to support the features presented.

Comment 15

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Thanks Luca for noticing that. I see browser compatibility info is missing for getDisplayMedia [1], that's where it would go.

We're targeting 66 with this feature, hopefully in Nightly by Friday.

Note you can get screensharing working today in Firefox using the adapter.js polyfill, as mentioned on that MDN page.

Here's an example: https://jsfiddle.net/jib1/q75yb8pf/

Hope that helps!

[1] https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia

Comment 16

[wilhelm.wanecek]

Exciting! Will the implementation include capture of system audio? I'm asking because Chromium has asserted that they will (unfortunately) not do so in their initial release [1].

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=326740#c105

Comment 17

[Luca Barbato]

I'm using the polyfill with the legacy method (getUserMedia() + constraint), my key interest is in getting the tab-capture.

Currently firefox works fine for capturing the whole screen and a single window.

I'm looking forward to see 66 :)

Comment 18

[Eric Shepherd [:sheppy]]

:jib -- Compatibility data has been done for getDisplayMedia() but has not been pulled to the production MDN server yet. Should be soon.

I'm actually torn between wanting system audio capture or just mic capture. Having the option of either (or both!) would be best. But personally for my own use cases mic capture is the most important thing.

Comment 19

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1321221 - Implement getDisplayMedia() and update related wpt tests. r?pehrsons,smaug

Comment 20

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Sorry Luca and Wilhelm, this release will be an API update only, and won't introduce system audio capture or tab capture.

Comment 21

[wilhelm.wanecek]

That's unfortunate, thanks for letting us know Jan-Ivar. This leads me to some follow-up questions:
1. Will getDisplayMedia throw if `{ audio: true }` is included in the constraints, or will it instead choose e.g. microphone input?
2. Will there be any way to feature-check whether audio-capture is supported by the browser without calling getDisplayMedia?
3. What's the timeline for deprecation of the existing API?

Comment 22

[Luca Barbato]

Capturing from microphone and mixing it to the video is already possible with the current API even if that will cause you to pop up two separate requests.

Tab capturing (and even better "self tab" capturing) is not currently possible (the next best thing, capturing the window, works though)

Comment 23

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to wilhelm.wanecek from comment #21)
> 1. Will getDisplayMedia throw if `{ audio: true }` is included in the
> constraints, or will it instead choose e.g. microphone input?

Neither. {audio: true} is ignorable in the spec (and {video: true} is implicit), which means:

 {video: true, audio: true} = 1 video track, 0 or 1 audio track.
 {audio: true}              = 1 video track, 0 or 1 audio track.
 {video: true}              = 1 video track, 0 audio tracks.
 {}                         = 1 video track, 0 audio tracks.

In Firefox you will get 0 audio tracks (for now at least).

> 2. Will there be any way to feature-check whether audio-capture is supported
> by the browser without calling getDisplayMedia?

Unfortunately no. Note there's no guarantee you'll get an audio track. This is still primarily a screen-sharing API, with audio a secondary and optional property only. It requires both browser support and the end user choosing to share a surface that has audio, and it being technically feasible to capture audio from that window. No commitments here atm.

> 3. What's the timeline for deprecation of the existing API?

Good question. We will probably keep it for a release or two, but it's behavior is changing immediately to align with getDisplayMedia(), which means JS loses the ability to choose. Instead, users will see a combined window and screen list. Application sharing is deprecated. See bug 1474376.

Comment 24

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1321221 - Delay getDisplayMedia() promise until first frame arrives, to pass wpt. r?pehrsons

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/680b0fa72580
Implement getDisplayMedia() and update related wpt tests. r=pehrsons,smaug
https://hg.mozilla.org/integration/autoland/rev/654a36f62f5a
Delay getDisplayMedia() promise until first frame arrives, to pass wpt. r=ng

Comment 26

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14685 for changes under testing/web-platform/tests

Comment 27

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

Comment 28

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/680b0fa72580
https://hg.mozilla.org/mozilla-central/rev/654a36f62f5a

Comment 29

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 30

[Eric Shepherd [:sheppy]]

These docs were finished a while ago, and BCD update is in place so compat tables are accurate for Firefox.

Comment 31

[Kohei Yoshino [:kohei]]

Bug 1391439 is now deployed. You can try it out on the Create New Attachment page of Bugzilla 😄