1396510 - Webextensions API to enable/disable browser sharing

Status: UNCONFIRMED

(WebExtensions :: General, enhancement, P5)

Description

[Andrei Skrykalov]

Attachment: Screenshot_1.png

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36

Steps to reproduce:

1)
var constraints = { video: { mediaSource: "browser" } };
navigator.mediaDevices.getUserMedia(constraints)
  .then(stream => video.srcObject = stream)
  .catch(function(err) {console.log(err.message)});

2)
set option in about:config
media.getusermedia.browser.enabled: true and 1)

3) In the new browser, I can not change the settings from "about: config" via web extension, there is no corresponding API


Actual results:

1) "The request is not allowed by the user agent or the platform in the current context."
2) Incorrect selection window (see Screenshot_1.png)
3) There is no way to capturethe stream from the active tab, changing the configuration through the extension, as it was before the introduction of web extensions in Firefox 57


Expected results:

I can get stream with the active tab. I see the correct tab selection window.
I can do this without using an extension, or API for web extensions allows me to put the necessary options

Comment 1

[Andrew Swan [:aswan]]

Nothing in this report is very clear to me.  Are you looking for a way for webextensions to set the media.getusermedia.browser.enabled preference?

Comment 2

[Andrei Skrykalov]

(In reply to undefined from comment #undefined)
> Yes, that's right! In addition, when you request a video from an active tab, the appearing popup behaves incorrectly. It has an empty dropdown

Comment 3

[Andrei Skrykalov]

(In reply to Andrew Swan [:aswan] from comment #1)
> Nothing in this report is very clear to me.  Are you looking for a way for
> webextensions to set the media.getusermedia.browser.enabled preference?

Yes, that's right! In addition, when you request a video from an active tab, the appearing popup behaves incorrectly. It has an empty dropdown

Comment 4

[Andrew Swan [:aswan]]

Okay, updating the title of this bug for the webextensions request (this would be an obvious complement to browser.privacy.network.peerConnectionEnabled though probably not under network).

Florian, can you take a look at the screenshot?  Is this a known issue or should we clone this into a new bug for the empty dropdown?

Comment 5

[Florian Quèze [:florian]]

This is about enabling browser sharing, ie. a way to get a video stream of the currently selected tab. This feature was added in our platform for Firefox Hello, and never exposed to the web due to very serious privacy issues. This doesn't have UI, so the broken drop down isn't really a surprise. I'm not sure if exposing browser sharing to WebExtensions is reasonable in terms of security; I doubt it, but jib will know better.

Jan-Ivar, should we consider exposing a browser sharing API to WebExtension? And if not, should we remove the browsing sharing code from MediaManager.cpp?

Comment 6

[Andrei Skrykalov]

Yes, that is what we need: get a video stream of the currently selected tab. In Google Chrome we use chrome.tabCapture.capture, and we try to implement the same functionality in Firefox.

Comment 7

[Andrew Swan [:aswan]]

Wait, I'm confused.  I can't read Russian but the dialog in the screenshot looks like bugzilla.mozilla.org is the origin from which getUserMedia() is being called, not an extension page (which would start with moz-extension://)
In any case, a request to add that capability (whether it is exposed to the web or just to extensions) should be put in a new bug, probably in component Core:WebRTC.

Comment 8

[Florian Quèze [:florian]]

(In reply to Andrew Swan [:aswan] from comment #7)
> Wait, I'm confused.  I can't read Russian but the dialog in the screenshot
> looks like bugzilla.mozilla.org is the origin from which getUserMedia() is
> being called, not an extension page (which would start with moz-extension://)

Likely because for the purpose of the screenshot, the code provided in step '1)' of the steps to reproduce has been pasted in the devtools console, and so ran in the context of the page devtools was inspecting (bugzilla in this case).

Comment 9

[Andrei Skrykalov]

New bug report:
https://bugzilla.mozilla.org/show_bug.cgi?id=1397784

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Martin, do you have any concerns with (re)enabling tab-sharing for add-ons (now web extensions)?

Andrei, does the chrome.tabCapture.capture API have any requirements of use or limitations in what tabs you may capture?

Comment 11

[Andrei Skrykalov]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #10)
> Martin, do you have any concerns with (re)enabling tab-sharing for add-ons
> (now web extensions)?
> 
> Andrei, does the chrome.tabCapture.capture API have any requirements of use
> or limitations in what tabs you may capture?

Yes, the activeTab permission required - https://developer.chrome.com/extensions/activeTab. And an extension with the activeTab permission only obtains access to a tab in response to an explicit user gesture:
Executing a browser action
Executing a page action
Executing a context menu item
Executing a keyboard shortcut from the commands API
Accepting a suggestion from the omnibox API

Comment 12

[Martin Thomson [:mt:]]

I think that this needs to follow the conclusion here: https://bugzilla.mozilla.org/show_bug.cgi?id=1007817#c20 (and later comments).  Providing this API for parity is sensible, but it probably needs to be a fill on top of https://w3c.github.io/mediacapture-screen-share/ with all that entails.  Unlike in the general web case, I think that we discuss whether webextensions get to narrow the selection that is presented, and - if the selection narrows to just one, as in this case - whether we offer the "remember my choice" checkbox.  We might even consider some improvements to the UX for this case.  However, treating this like any other screen sharing request is a sensible starting place.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Makes sense. FWIW Firefox Hello used to pass in a windowID using the browserWindow constraint to pick a tab. This worked because Firefox Hello controlled the tab selection process. Absent that constraint, the API returns the asking tab. [1] 

Short of some short-term exception for same-tab access, it sounds like this is blocking on new preview, or event selection UX. Even same-tab access would require some permission UX with new language that doesn't exist today.

I therefore agree with comment 12 that this probably pushes a solution out far enough that we should address bug 1321221 first, to avoid extending functionality to a non-spec API.

[1] https://jsfiddle.net/jib1/uax4gom5/ with "media.navigator.permission.disabled" = false in about:config

Comment 14

[Caitlin Neiman [:caitmuenster]]

Hi Andrei, this has been added to the agenda for the February 13, 2018 WebExtensions APIs triage. Would you be able to join us? 

Here’s a quick overview of what to expect at the triage: 

* We normally spend 5 minutes per bug
* The more information in the bug, the better
* The goal of the triage is to give a general thumbs up or thumbs down on a proposal; we won't be going deep into implementation details

Relevant Links: 

* Wiki for the meeting: https://wiki.mozilla.org/WebExtensions/Triage#Next_Meeting
* Meeting agenda: https://docs.google.com/document/d/1lB8M4WhF2xRrf4A5DlUNkKKiiuxfpxfx6CfHLv-FoAM/edit#
* Vision doc for WebExtensions: https://wiki.mozilla.org/WebExtensions/Vision

Comment 16

[Rob Wu [:robwu]]

It's already possible to capture the whole window from a web page these days, with https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia

It is already possible to capture a screenshot of a tab with https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/Tabs/captureVisibleTab

There is a feature request for the tabCapture API to stream a specific tab at bug 1391223. It doesn't look like there is much activity in it (any more?).

Comment 17

[Shaurya (Teleparty)]

(In reply to Rob Wu [:robwu] from comment #16)

It's already possible to capture the whole window from a web page these days, with https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia

It is already possible to capture a screenshot of a tab with https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/Tabs/captureVisibleTab

There is a feature request for the tabCapture API to stream a specific tab at bug 1391223. It doesn't look like there is much activity in it (any more?).

I think there's still a ton of interest from both users and developers in this feature, especially given the prevalence of screen recording extensions that can record a specific tab in remote work environments.

For example Chrome has a huge number of tab recorder extensions, including Loom (4m users) and Screencastify (6m users) that can record tab video AND audio:
https://chrome.google.com/webstore/detail/screencastify-screen-vide/mmeijimgabbpbgpdklnllpncmdofkcpn?hl=en
https://chrome.google.com/webstore/detail/loom-%E2%80%93-free-screen-record/liecbddmkiiihnedobmlmillhodjkdmb?hl=en

Neither has a Firefox extension, primarily because tab audio capture on Firefox is impossible. The tabcapture WebExtensions API is unimplemented as you noted. However, your suggested workaround -- the MediaDevices.getDisplayMedia() API -- to capture a window is insufficient because the getDisplayMedia API still lacks support for Audio capture on Firefox.

These Chrome developers and others obviously want to port their extensions over to Firefox, but are simply unable to do so because there is currently no path to tab audio capture on Firefox. The easiest migration path for these engineers would be for Firefox to implement the tabCapture webExtensions API since most of these developers make use of the extension API on Chrome, but even if they were willing to refactor their extensions to use the getDisplayMedia API they cannot due to the lack of Audio capture support on Firefox.

Implementing audio capture in the getDisplayMedia API also paves the way for tab-sharing in screen-sharing applications, something hundreds of millions of video conferencing users would appreciate as well.

---

Also worth noting:

Firefox implemented tab Video capture support in the getDisplayMedia API back in March 2019 in Firefox 66, but still does not support Audio capture. Both Chrome and Edge have been supporting the optional audio capture flag in the MediaDevices.getDisplayMedia() API for years.

Firefox Hello innovated by introducing video and audio tab-sharing primitives back in Firefox 45 in March 2016. This was way ahead of its time.

It seems like there is some old code (currently disabled) to support tab audio capture back from the Firefox Hello days. If this code can be reused either to support tab audio capture in the webextensions API or the getDisplayMedia API I think that would be go a long ways to making Firefox more fully featured in tab screen-recording applications.

https://bugzilla.mozilla.org/show_bug.cgi?id=1541425
https://bugzilla.mozilla.org/show_bug.cgi?id=1541425
https://bugzilla.mozilla.org/show_bug.cgi?id=1651145

Comment 18

[Shaurya (Teleparty)]

Following up on this.

Is it feasible to reenable tab audio capture from the Firefox Hello codebase to add tab capture to the getDisplayMedia API?