Make it possible to match an nsIMediaDevice to an nsIScreen
Categories
(Core :: WebRTC, task)
Tracking
()
People
(Reporter: mconley, Unassigned)
References
(Blocks 2 open bugs)
Details
Attachments
(3 files)
Right now, it's possible to map a nsIMediaDevice to a browser window. We do this by matching the raw device ID to the window ID exposed here: https://searchfox.org/mozilla-central/rev/2c1092dc68c63f7bad6da6a03c5883a5ab5ff2ca/dom/interfaces/base/nsIDOMWindowUtils.idl#2071
In order to pull off bug 1637791 for the screen case, we'll need to know which screen the user is sharing. We have the nsIScreenManager, which lets us get at nsIScreen's... but we need to be able to match those to nsIMediaDevice.
I think perhaps the nsIDOMWindowUtils API was a good start, but probably not the right design for doing this kind of matching. I suspect we should add a method to nsIMediaDevice to match against browser windows or screens.
|
Reporter | |
Comment 1•4 years ago
|
||
Or perhaps for nsIMediaDevice's that represent screens, they can have a reference to the associated nsIScreen, and nsIMediaDevice's that represent browser windows, they can have a reference to the associated browser DOM window.
|
|
Reporter | |
Comment 2•4 years ago
|
||
Hey jib,
How difficult do you think it would be to add a reference to the associated browser window / nsIScreen for "scary" nsIMediaDevice types?
|
||
Comment 3•4 years ago
|
||
I'd have to look tomorrow. Sharing the ni? with dminor in case he knows already.
|
|
Reporter | |
Comment 5•4 years ago
|
||
Having the nsIScreen / browser DOM window be a reference on the nsIMediaDevice probably isn't the right model because we need nsIMediaDevice to work in the content process (where there's no direct access to nsIScreen / browser DOM windows).
What we currently do for browser windows is that we ask the widget layer to give us the native window handle, and then cast that to the right format and compare it against the rawID for the device. We have to go through this kind of circuitous path because the third-party WebRTC layer gets access to the window / screen through its own mechanism outside of Gecko, and that's unlikely to change.
So my current idea is to have two methods on nsIMediaDevice: matchScreen(in nsIScreen) and matchDOMWindow(in mozIDOMWindow). This will make the MediaDevice class responsible for knowing how to match an nsIScreen or mozIDOMWindow to the underlying rawId, which probably makes more sense then where I stashed it on the nsIDOMWindowUtils interface.
Objections to that approach, jib?
|
|
Reporter | |
Comment 6•4 years ago
|
||
Bah, I forgot that we serialize this nsIMediaDevice structure up and send it to the parent as a JS Object. This approach isn't going to work as-is. Let me think about this more.
|
|
Reporter | |
Comment 7•4 years ago
|
||
How about instead of making it the responsibility of the nsIMediaDevice to be able to resolve to a DOM window or a nsIScreen, we make it the responsibility of the nsIMediaManagerService?
Would it be okay if I add a new method to nsIMediaManagerService, like rawIdMatchesWindow(in AString rawId, in mozIDOMWindowProxy aWindow), and rawIdMatchesScreen(in AString rawId, in nsIScreen aScreen)?
|
|
Reporter | |
Comment 8•4 years ago
|
||
Hm... part of the challenge here also is that the X11 backend used on Linux doesn't seem to have a proper notion of Screen IDs. It will return "-1" (the kFullDesktopScreenId ID) when capturing. I think this means that unless we patch the X11 backend, that we won't be able to differentiate between captures on different screens on Linux.
|
|
Reporter | |
Comment 9•4 years ago
|
||
It does appear that recent-ish patches in the WebRTC repository include support for enumerating and setting the screen source on the Linux X11 screen capture backend when using randr: https://webrtc.googlesource.com/src/+/e952b78c283f5e54376561f7c7dd1460143c6112%5E%21/
Hey dminor, how often do we pull webrtc.org updates into m-c?
|
|
Reporter | |
Comment 10•4 years ago
|
||
(That last update to the webrtc code has the screen source ID be the name Atom of the XRRMonitorInfo representing the screen)
|
||
Comment 11•4 years ago
|
||
We're planning to do a webrtc.org update starting second half of 2020.
I had looked at pulling that particular changeset in early because it would help with multiple monitor support on Linux, but it would be painful because it adds Xrandr to the build, and our build system integration of gn is less than pleasant (see https://firefox-source-docs.mozilla.org/build/buildsystem/gn.html). If you're willing to fight with it, I'm happy to review a patch, but my impression was that it was better to wait on a full update to take it.
|
|
||
Comment 12•4 years ago
|
||
(In reply to Mike Conley (:mconley) (:⚙️) (Extremely busy) from comment #7)
Would it be okay if I add a new method to nsIMediaManagerService, like
rawIdMatchesWindow(in AString rawId, in mozIDOMWindowProxy aWindow), andrawIdMatchesScreen(in AString rawId, in nsIScreen aScreen)?
Good question. That might work. Will you be calling it from the main process? If so, you may need to short-circuit things to avoid opening MediaManager with its media thread in the master process (or does nsIMediaManagerService implicitly create it? It shouldn't need to run there anymore, unless you run Firefox as single-process, but it may be running there already. Sadly, I'm actually not sure since a lot of this was written pre-e10s).
Something like a getScreenFromRawId that returns a screen ref if one is found or null otherwise, might be preferable as an API, in case IPC machinery were to ever be involved e.g. like is the case is for sanitizeDeviceIds.
|
|
Reporter | |
Comment 13•4 years ago
|
||
|
|
Reporter | |
Comment 14•4 years ago
|
||
Depends on D80685
|
|
Reporter | |
Comment 15•4 years ago
|
||
Depends on D80686
|
||
Updated•4 years ago
|
Description
•