1637791 - Show a highlight around any windows shared over WebRTC, or around the edge of the screen when sharing a screen.

Status: NEW

(Firefox :: Site Permissions, enhancement, P3)

Description

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

For bonus points, ideally, these highlights would not be captured by the WebRTC desktop capture layer. That should be a fun challenge.

Comment 1

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Hey dminor,

It looks like there's a ::SetExcludedWindow capability in some of the WebRTC screen capturing stuff, but it looks to me as if it's only implemented for macOS... looking through the Chromium source code, I don't see them using it at all over there, too.

It's likely that we'll need to implement this highlight using a special transparent OS-window, and if we don't want it to be captured during a screen capture, I presume we'd want to use something like ::SetExcludedWindow.

Hey dminor, suppose I were to try to implement ::SetExcludedWindow for Windows... do you know if there'd be any difficulty in getting that change landed in the third-party webrtc code?

Comment 2

[Dan Minor [:dminor]]

Hi Mike, In this case, since it's already implemented for one platform, I suspect your chances of getting it landed upstream for a different platform are pretty good. The desktop capture code seems to change pretty slowly, so I expect your patch for Firefox would apply upstream without too many difficulties. I have a webrtc.org account, I can help get it up for review once done, although I might need your help addressing any review comments from upstream.

Comment 3

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - Add a chrome-only mousethrough window.open feature argument. r?nika!

Comment 4

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - Make dependent popup windows have parent popup level by default. r?nika!

Depends on D75404

Comment 5

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - [WIP] Add prototype sharing highlight window

Depends on D75406

Comment 6

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Indicators mockup

Comment 7

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - Make nsIMediaManagerService responsible for knowing how to resolve a window or screen to a raw ID. r?jib!

Depends on D80688

Comment 8

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - Make webrtcUI.jsm use nsIMediaManagerService to resolve windows to raw IDs. r?pbz!

Depends on D80689

Comment 9

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - Cocoa window hack

Depends on D75408

Comment 10

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

So the WIP implementation here is a prototype that uses a transparent DOM window to draw the highlight. This appears to mostly work, however there's a bit of a performance penalty on the graphics side of things - since the widget layer isn't drawing the highlight, we can't take advantage of the OS compositor to cheaply composite it, which means that the highlights add overhead. Not a good thing when sharing over WebRTC (which is already pretty intensive from the OS compositor's standpoint).

After consulting with mstange, I'm going to change tack here a little and try having the widget layer draw the highlights. The general idea is to add properties to the nsIAppWindow and nsIScreen interfaces to draw the highlights.

Comment 11

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Here's a demo from shorlander on how the highlight should be layered to maximize visibility across a variety of backgrounds: http://stephenhorlander.com/sharing-highlight/sharing-highlight.html

Comment 12

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - Stub out showSharingHighlight methods for nsIAppWindow and nsIScreen. r?mstange!

Depends on D80690

Comment 13

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Attachment: Bug 1637791 - Implement showSharingHighlight for nsIAppWindow and nsIScreen for macOS. r?mstange!

Depends on D81576

Comment 14

[Paul Zühlcke [:pbz]]

This seems like a nice feature. Mike, are you still working on this? I'm assuming it's backlog for now.

Comment 15

[Mike Conley (:mconley) (:⚙️) (PTO: May 24 - June 2nd)]

Yeah, this is backlogged, thanks.