Implement getDisplayMedia for screen capture to comply with spec changes

RESOLVED FIXED in Firefox 66

Status

()

defect
P2
normal
Rank:
17
RESOLVED FIXED
2 years ago
a month ago

People

(Reporter: mchiang, Assigned: jib)

Tracking

(Blocks 2 bugs, {dev-doc-complete})

unspecified
mozilla66
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox66 fixed)

Details

(URL)

Attachments

(2 attachments)

As discussed in bug 1313758 comment 8, we currently implement an outdated draft [2] of the spec.

Updated

2 years ago
Rank: 25
Priority: -- → P2
In the latest Screen Capture draft (Draft 27), the constraint is reduced to a simple boolean, app won't be able to indicate the surface type, then we have only two choices:

1) Let user choose the surface type; then let user choose the window / screen / applicationto share.
2) Always provide one type of surface type.
(Assignee)

Updated

2 years ago
Blocks: 1396510
(Assignee)

Comment 2

2 years ago
In hindsight, I think an obstacle to implementing the spec here is that our current implementation allows passing in constraints to downscale the resolution and decimate frame rates from the very first frame. Constraints do not filter the selection list one bit.

We consider this a useful feature, since desktops tend to be of extremely large resolution, and their high frame rate to be overkill for most WebRTC use.

The spec on the other hand disallows constraints on getDisplayMedia, except on subsequent calls to track.applyConstraints presumably.

This presumably means JS would need to expect the full-size full-frame-rate stream first, and then issue

    await track.applyConstraints({width, height, frameRate})

on the video track before attaching it to a sink.

Martin, does this match your intent? Do you have any guidance on how to implement this in a performant manner? Should we be concerned that this may cause capture to fail in situations where returning the unscaled result might fail where a downscaled version might not?
Flags: needinfo?(martin.thomson)
Can you take that as an issue to the screen sharing repo.  The intent behind removing the constraints was primarily to avoid the constraints being used to guide selection.  No one had a use case that they could articulate for other constraints, so removing them entirely was easier.  Adding them back isn't impossible.
Flags: needinfo?(martin.thomson)
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3
(Assignee)

Updated

2 years ago
Duplicate of this bug: 1400224
(Assignee)

Comment 7

11 months ago
(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #2)
> The spec on the other hand disallows constraints on getDisplayMedia, except
> on subsequent calls to track.applyConstraints presumably.

This was resolved at the April interim. See https://github.com/w3c/mediacapture-screen-share/issues/49#issuecomment-381556471

Just waiting on a PR to update the spec.

Updated

10 months ago
Blocks: 1303919

Comment 8

10 months ago
In light of google's (fairly recent) announcement to implement getDisplay media (Chromestatus: https://www.chromestatus.com/feature/6744724455030784 Discussion: https://bugs.chromium.org/p/chromium/issues/detail?id=326740#c35), it would be _fantastic _ if Firefox was able to align their implementation sooner rather than later :)

Is this still only blocked on the PR to update the spec? :) Is there anyone who is willing to devote some time and effort to unblocking this? I'm afraid I'm completely inexperienced in spec-writing.
Flags: needinfo?(jib)
(Assignee)

Comment 9

10 months ago
I'm slated to work on getDisplayMedia after I add the resizeMode constraint. Given the timing and recent interest, and depending on spec progress, I may start them in parallel.

I'm working with Suhas to finish off the spec, from the issues resolved at the April w3c meeting, including the issue in comment 4. The best overview of that remaining spec work is probably the slides [1]. Suhas and I just discussed this yesterday, and he will be providing PRs.

[1] https://docs.google.com/presentation/d/1sboQqMc5O1-oZ7w0ZY1VUyhY8JG0VxOzU8kAQitoHJU/edit#slide=id.g2c521efd37_0_20
Flags: needinfo?(jib)
(Assignee)

Updated

10 months ago
Assignee: nobody → jib
(Assignee)

Updated

10 months ago
Rank: 25 → 17
Priority: P3 → P2

Comment 10

10 months ago
Great news! Thanks for the update :) Is any extra effort required to coordinate with the Chrome and Edge teams, e.g. in regards to the resizeMode and/or changes to the spec?
(Assignee)

Comment 11

10 months ago
I'm pushing this to the front of my queue.
Keywords: dev-doc-needed
(Assignee)

Comment 12

10 months ago
I'm working with Suhas to finish off the spec.

Updated

7 months ago
Blocks: 1497559

Comment 14

4 months ago
You might mention this issue in https://developer.mozilla.org/en-US/docs/Web/API/Screen_Capture_API/Using_Screen_Capture#Simple_screen_capture since apparently even the current 65.0b4 does not seem to support the features presented.
(Assignee)

Comment 15

4 months ago
Thanks Luca for noticing that. I see browser compatibility info is missing for getDisplayMedia [1], that's where it would go.

We're targeting 66 with this feature, hopefully in Nightly by Friday.

Note you can get screensharing working today in Firefox using the adapter.js polyfill, as mentioned on that MDN page.

Here's an example: https://jsfiddle.net/jib1/q75yb8pf/

Hope that helps!

[1] https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia
Depends on: 1474376

Comment 16

4 months ago
Exciting! Will the implementation include capture of system audio? I'm asking because Chromium has asserted that they will (unfortunately) not do so in their initial release [1].

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=326740#c105

Comment 17

4 months ago
I'm using the polyfill with the legacy method (getUserMedia() + constraint), my key interest is in getting the tab-capture.

Currently firefox works fine for capturing the whole screen and a single window.

I'm looking forward to see 66 :)
:jib -- Compatibility data has been done for getDisplayMedia() but has not been pulled to the production MDN server yet. Should be soon.

I'm actually torn between wanting system audio capture or just mic capture. Having the option of either (or both!) would be best. But personally for my own use cases mic capture is the most important thing.
(Assignee)

Updated

4 months ago
Depends on: 1515548
(Assignee)

Comment 20

4 months ago
Sorry Luca and Wilhelm, this release will be an API update only, and won't introduce system audio capture or tab capture.

Comment 21

4 months ago
That's unfortunate, thanks for letting us know Jan-Ivar. This leads me to some follow-up questions:
1. Will getDisplayMedia throw if `{ audio: true }` is included in the constraints, or will it instead choose e.g. microphone input?
2. Will there be any way to feature-check whether audio-capture is supported by the browser without calling getDisplayMedia?
3. What's the timeline for deprecation of the existing API?

Comment 22

4 months ago
Capturing from microphone and mixing it to the video is already possible with the current API even if that will cause you to pop up two separate requests.

Tab capturing (and even better "self tab" capturing) is not currently possible (the next best thing, capturing the window, works though)
(Assignee)

Comment 23

4 months ago
(In reply to wilhelm.wanecek from comment #21)
> 1. Will getDisplayMedia throw if `{ audio: true }` is included in the
> constraints, or will it instead choose e.g. microphone input?

Neither. {audio: true} is ignorable in the spec (and {video: true} is implicit), which means:

 {video: true, audio: true} = 1 video track, 0 or 1 audio track.
 {audio: true}              = 1 video track, 0 or 1 audio track.
 {video: true}              = 1 video track, 0 audio tracks.
 {}                         = 1 video track, 0 audio tracks.

In Firefox you will get 0 audio tracks (for now at least).

> 2. Will there be any way to feature-check whether audio-capture is supported
> by the browser without calling getDisplayMedia?

Unfortunately no. Note there's no guarantee you'll get an audio track. This is still primarily a screen-sharing API, with audio a secondary and optional property only. It requires both browser support and the end user choosing to share a surface that has audio, and it being technically feasible to capture audio from that window. No commitments here atm.

> 3. What's the timeline for deprecation of the existing API?

Good question. We will probably keep it for a release or two, but it's behavior is changing immediately to align with getDisplayMedia(), which means JS loses the ability to choose. Instead, users will see a combined window and screen list. Application sharing is deprecated. See bug 1474376.

Comment 25

4 months ago
Pushed by jbruaroey@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/680b0fa72580
Implement getDisplayMedia() and update related wpt tests. r=pehrsons,smaug
https://hg.mozilla.org/integration/autoland/rev/654a36f62f5a
Delay getDisplayMedia() promise until first frame arrives, to pass wpt. r=ng
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14685 for changes under testing/web-platform/tests
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

Comment 28

4 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/680b0fa72580
https://hg.mozilla.org/mozilla-central/rev/654a36f62f5a
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Upstream PR merged
(Assignee)

Updated

3 months ago
Depends on: 1523958

These docs were finished a while ago, and BCD update is in place so compat tables are accurate for Firefox.

Bug 1391439 is now deployed. You can try it out on the Create New Attachment page of Bugzilla 😄

You need to log in before you can comment on or make changes to this bug.