1303919 - Implement chrome.desktopCapture

Status: RESOLVED WONTFIX

(WebExtensions :: Compatibility, enhancement, P3)

Description

[Shane Caraveo (:mixedpuppy)]

https://developer.chrome.com/extensions/desktopCapture

Placeholder for chrome.desktopCapture

Comment 1

[Andy McKay]

See also: https://w3c.github.io/mediacapture-screen-share/

Comment 2

[Bob Silverberg [:bsilverberg]]

Attachment: getUserMedia dialog in Firefox

Comment 3

[Bob Silverberg [:bsilverberg]]

Attachment: desktopCapture.chooseDesktopMedia dialog in Chrome

Comment 4

[Bob Silverberg [:bsilverberg]]

I don’t think this API does what we thought it did. It is used to open a dialog to indicate which type of media one wants to share. It isn’t used to take any sort of screen shots, or to actually capture anything. It just opens a dialog which then provides a stream which can be passed into a call to Chrome’s getUserMedia() API.

In Firefox we already have a WebAPI that does this, called  MediaDevices.getUserMedia() [1], which is also used to open a similar dialog box. I have attached screen shots of the dialog in Chrome and in Firefox.

Given that we already have a WebAPI that does this, it doesn’t seem necessary to also implement this as a WebExtensions API. Andy, what do you think?

[1] https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia

Comment 5

[Andy McKay]

Agreed.

Comment 6

[Rob Wu [:robwu]]

In case others end up here (e.g. https://github.com/muaz-khan/Firefox-Extensions/issues/14)

The significant part of the desktopCapture API is not the fact that it has a UI, but that it offers the ability to capture the screen. Starting from Firefox 52, an add-on is no longer necessary to allow extensions to use screen sharing. Any web page can *request* access to the screen (whether the user approves is a different matter).

See https://wiki.mozilla.org/Screensharing

Comment 7

[Mike Conca [:mconca]]

Despite the existence of a similar WebAPI as described in comment 4, the lack of desktopCapture is proving to be a barrier for Chrome developers porting to Firefox. We also have additional Chrome market data showing that usage of this API is increasing.

Nov, 2016 - 56th most popular API, used by 1355 extensions
Aug, 2017 - 50th most popular API, used by 2126 extensions
Nov, 2017 - 45th most popular API, used by 2338 extensions

desktopCapture is the only API in the top 100 Chrome API that Firefox does not support. The 2338 extensions that use it represent approximately 1.4M Chrome users.

I am reopening this bug and putting it into the WebExtensions backlog (P3) for future implementation.

Comment 8

[Shane Caraveo (:mixedpuppy)]

I wonder if chrome.desktopCapture can just wrap the existing webapi.

Comment 9

[Rob Wu [:robwu]]

I wonder whether Firefox should really implement chrome.desktopCapture.

The chrome.desktopCapture API's sole purpose is to obtain a token so that the getUserMedia API can access the screen, e.g. as shown at https://stackoverflow.com/questions/24760955/show-video-from-chrome-desktopcapture-choosedesktopmedia-in-a-tab/24768054#24768054

In Firefox, the desktopCapture API is not needed for that. Firefox's getUserMedia API supports screen capturing without extensions:

navigator.mediaDevices.getUserMedia({
  video: {
    mediaSource: 'screen' // or 'window' or 'application'.
  }
}); // Returns a Promise<MediaStream>

The only potential benefit of chrome.desktopCapture over Firefox's getUserMedia is that the desktopCapture API takes a list of "screen", "window", "tab" as input, so that the user can select any of these sources in a UI, whereas Firefox's getUserMedia requires the application to decide and request only one of these sources.

Anyone who wants to build a cross-browser screen sharing application already needs to account for the difference between browsers. In this regard, Firefox is already more user-friendlier because users do not need to install a separate extension for screen sharing, and developer-friendlier because devs don't need two separate components to implement screen sharing.


As an add-on reviewer, I have seen multiple screen sharing add-ons that transitioned from legacy add-ons to WebExtensions as follows:
BEFORE: web page -> content script -> Addon SDK main script modifies prefs -> reply to content script -> reply to web page (e.g. https://github.com/muaz-khan/Firefox-Extensions/tree/master/enable-screen-capturing).
AFTER : web page -> content script -> reply to web page (the extension is does nothing else, because web pages can capture the screen without extension APIs).
So clearly, these applications and extensions can do fine without the desktopCapture API.


chrome.desktopCapture may eventually be obsolete too, because there is an effort to standardize how screen capture is handled in WebRTC:
"Firefox use an API that's much closer to the w3c spec, not sure if it's 100% compliant or not. We want to move over to this API but we're not actively working on it yet."
https://bugs.chromium.org/p/chromium/issues/detail?id=326740#c20



So far, I framed this comment in the context of "Why make an extension API if web applications can do without it?".
So, let's now view this from the other perspective: "What can extensions do with this API that can't be done on the web?".
Mainly the ability to capture the screen from a hidden background page. If this is the goal, then we'd better start with properly supporting the getUserMedia API from background pages, because that is the prerequisite for media capturing to work.
For gUM in extensions, see bug 1394062, bug 1278100, bug 1363854 (and maybe others, I didn't perform an exhaustive search).
Maybe there is already an appropriate UI for selecting the media source when these bugs get fixed.

Comment 10

[Caitlin Neiman [:caitmuenster]]

Hi all, this has been added to the agenda for the February 13, 2018 WebExtensions APIs triage. Anyone who is interested in this bug is welcome to join the meeting. 

Here’s a quick overview of what to expect at the triage: 

* We normally spend 5 minutes per bug
* The more information in the bug, the better
* The goal of the triage is to give a general thumbs up or thumbs down on a proposal; we won't be going deep into implementation details

Relevant Links: 

* Wiki for the meeting: https://wiki.mozilla.org/WebExtensions/Triage#Next_Meeting
* Meeting agenda: https://docs.google.com/document/d/1lB8M4WhF2xRrf4A5DlUNkKKiiuxfpxfx6CfHLv-FoAM/edit#
* Vision doc for WebExtensions: https://wiki.mozilla.org/WebExtensions/Vision

Comment 11

[Mike Conca [:mconca]]

Chrome recently announced an intent to implement for the WebRTC API GetDisplayMedia.
https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/j7k2nI_9nng

Note that this is an intent to implement, not ship, so timing is unclear. Nevertheless, it seems like desktopCapture will likely be deprecated and polyfilled to the new API, once it is available.  It would be very convenient for Chrome developers if WebExtensions could also polyfill desktopCapture to either the existing getUserMedia() API, or the Firefox implementation of GetDisplayMedia (bug 1321221).

A Firefox polyfill depends on finding a way to show the current WebRTC permission dialog when the active tab is not the requestor (popup or background).

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

This is a web standards issue, and Mozilla should drive convergence around the web standard. That means explicitly not supporting non-standard alternatives, as that risks hurting the web. I'm slated to work on bug 1321221, so I ask that we wait for that, and not implement this.

AFAIK there's nothing web developers cannot do in Firefox with screen sharing today (we should probably never support desktop capture from a background page due to the inherent security risks). 

Short-term, I suggest we go the other way and shim getDisplayMedia in adapter.js for both Firefox (using getUserMedia) and Chrome (using desktopCapture). See [1].

[1] https://github.com/webrtc/adapter/issues/830#issuecomment-403044133

Comment 13

[Nils Ohlmeier [:drno]]

(In reply to Mike Conca [:mconca] (Denver, CO, USA UTC-6) from comment #7)
> Despite the existence of a similar WebAPI as described in comment 4, the
> lack of desktopCapture is proving to be a barrier for Chrome developers
> porting to Firefox. We also have additional Chrome market data showing that
> usage of this API is increasing.
> 
> Nov, 2016 - 56th most popular API, used by 1355 extensions
> Aug, 2017 - 50th most popular API, used by 2126 extensions
> Nov, 2017 - 45th most popular API, used by 2338 extensions
> 
> desktopCapture is the only API in the top 100 Chrome API that Firefox does
> not support. The 2338 extensions that use it represent approximately 1.4M
> Chrome users.

Just to clarify for people who are not as familiar with the context:
- Chrome up till today still requires an add-on for any given web page to enable screen sharing. 
- In Firefox on the other hand no such add-on is any longer required.

So the numbers above only tell us how many pages/services want to offer the screen sharing feature in Chrome, not extensions offering general features across the web via desktopCapture. And since most of these extensions are only enabling screen sharing for one side users might even have to install several of these add-ons to enable screen sharing for different services.

With Chrome recent announcement of implementing getDisplayMedia(), which sounds to me will no longer require a web extension to enable getDisplayMedia() for a given page/service, I'm expecting the number of extensions in the Chrome market to drop significantly.

Comment 14

[Mike Conca [:mconca]]

:jib and I talked this morning and I agree with closing this feature request. Providing a web standard implementation of getDisplayMedia() to go along with getUserMedia() is the preferred path.  In addition to providing a shim to adapter.js, we could also add a polyfill from getDisplayMedia() to desktopCapture in https://github.com/mozilla/webextension-polyfill.