1559011 - Capturing some windows via getDisplayMedia doesn't work properly on Windows

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Bryan Ferguson]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

1. Browse to a page that uses the getDisplayMedia API in Firefox on Windows (Win8 or newer). A sample that displays captured frames while capturing makes the problem easier to observe (e.g. https://screen-record.glitch.me ), or at least one that allows local playback of captured frames later (e.g. https://webrtc.github.io/samples/src/content/getusermedia/getdisplaymedia ).
2. Press the appropriate button to choose a source to capture, and from the 'Select Window or Screen' dropdown, choose a window from an app in one of these categories:
   a) Firefox (the window you're recording from or another)
   b) Microsoft Office apps (e.g. Excel, Word, PowerPoint, OneNote, Outlook)
   c) Electron apps (e.g. Visual Studio Code, Atom, another app from http://electronjs.org/apps )
   d) A Chromium-based browser (e.g. Chrome, Opera, Edge preview)
3. If capturing a window other than the one you're recording from, and using a sample/test that displays captured frames while capturing, resize & move the window being captured if necessary to see the captured frames.
4. If capturing a Firefox or Office app window, make some changes to the window content (e.g. change selection, scroll, zoom, or make edits to the address bar or document region)
5. Observe the captured frames (after stopping any recording if necessary).

Actual results:

The window contents are captured properly (the captured frames accurately represent all the window contents).

Expected results:

The window contents aren't captured properly. In the case of a Firefox window, the captured frames may not be representative of the initial window content, and don't reflect any subsequent changes. In the case of an Office app window, changes to the document region / viewport aren't reflected in the captured frames. In the case of a window from an Electron app or another Chromium-based browser, the window client area is black in the captured frames.

Comment 1

[Bryan Ferguson]

The WebRTC native code library supports 2 high-level approaches to window capture on Windows: capturing the screen & cropping it to the window bounds when some conditions are met, or attempting to capture just the window contents otherwise (e.g. if occluded by an unrelated window). Using the former approach conditionally with a fallback to the latter currently requires calling CroppingWindowCapturer::CreateCapturer (as Chromium-based browsers now do) rather than DesktopCapturer::CreateWindowCapturer (as Firefox currently does).

The contents of the affected app windows may be captured properly using the cropping approach (which is used on Win8+ when the captured window isn't occluded by unrelated windows).

This WebRTC bug tracks a potential change to capture the affected app windows properly in the fallback approach on Win8.1+: https://crbug.com/webrtc/10734 . That said, there may be some benefit to Firefox using the cropping approach when possible: it'd fix this issue in the interim in some cases, and may mitigate the perf impact of any fix for that bug (by making it less likely that the impacted path is used).

Note that the Firefox sharing indicator window at the top of the stream would prevent use of the cropping approach when the captured window is behind it unless it's allowed to be ignored / included in the captured frames by calling SetExcludedWindow. Chromium-based browsers appear to intend to allow the sharing notification bar to be included in the captured frames (as in the whole-screen capture case), but don't currently (details in https://crbug.com/973245 ). That said, they allow hiding the notification bar (with it still shown in the taskbar / window switcher), unlike the Firefox sharing indicator. Perhaps a means of hiding or unpinning the Firefox sharing indicator should be added, but that perhaps shouldn't block allowing it to be included in captured window frames in the meantime (such that the more-reliable / -performant cropping approach could be used).

Comment 2

[Dan Minor [:dminor]]

Thank you for your report! The black client area problem while capturing Chrome / Electron apps sounds like a known issue (Bug 1398716). The Office app window may also be a known issue, but I was unable to find a bug for it. We definitely need to investigate that and the other problems you mentioned.

Comment 3

[Bryan Ferguson]

This bug should no longer repro as written if Firefox updates to WebRTC M77 or newer, where this bug is fixed: https://crbug.com/webrtc/10734 . That fix degrades window capture performance on Windows unless using the cropping window capturer, which could be mitigated somewhat as follows:

1. Call CroppingWindowCapturer::CreateCapturer or the DesktopCaptureOptions::set_allow_cropping_window_capturer(true) (new in WebRTC M78) to allow use of the faster cropping capturer in some cases (when the window is foremost / non-occluded). A downside is possible inadvertent inclusion of unrelated UI in the captured frames (e.g. https://crbug.com/webrtc/10835 ).
2. Optionally call DesktopCapturer::SetExcludedWindow with the sharing indicator window handle to allow use of the faster cropping capturer in more cases (when the window is occluded only by the sharing indicator). A downside is inclusion of the sharing indicator window in the captured frames (especially if it can't be hidden or freely moved).

Comment 4

[Andreas Pehrson [:pehrsons]]

Dan, perhaps this would be a good candidate for cherry-picking?

Comment 5

[Dan Minor [:dminor]]

Yes, offhand it does look like it will apply cleanly.

https://webrtc.googlesource.com/src/+/f89110d67902e787f6745ad2b52f7f09fc808512%5E%21/#F0

Comment 6

[Dan Minor [:dminor]]

I've filed Bug 1586071 to handle switching to the CroppingWindowCapturer. I noticed some strangeness (distorted capture, like there was an incorrect video stride) if I moved a shared window near to the sharing indicator, so I think we would need to try DesktopCapturer::SetExcludedWindow and see if that helps. I've noticed some problems with cropping in general while investigating this bug, so perhaps we just have a bug somewhere in the cropping code in the version of webrtc.org we're currently using.

Comment 7

[Dan Minor [:dminor]]

Attachment: Bug 1559011 - Try using PW_RENDERFULLCONTENT for Windows window capture; r=pehrsons

This is a partial cherrypick of https://webrtc.googlesource.com/src/+/f89110d67902e787f6745ad2b52f7f09fc808512.
The cropping changes in that revision are problematic on our version of webrtc.org and
result in distorted video, which looks as though there is a stride problem. This takes
the change to try to use PW_RENDERFULLCONTENT and to fall back to the current code if
that fails. This fixes capturing Chrome windows and allows Firefox to properly capture
its own window.

Using PW_RENDERFULLCONTENT can adversely affect performance. Using the
CroppingWindowCapturer can avoid using the WindowCapturer in some circumstances and so
result in better performance. Bug 1586071 tracks switching to the
CroppingWindowCapturer.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/734f05da6e8e
Try using PW_RENDERFULLCONTENT for Windows window capture; r=pehrsons

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/734f05da6e8e

Comment 11

[Maria Berlinger [:mberlinger], Services QA]

The issue is verified fixed using Fx71.0b5 on Windows 10, macOS and Ubuntu 18.04. The capture of skype and chrome windows works without issues now, correctly displaying the content.
However, the issue is still reproducible on Windows 7 using Fx 71.0b5 AND latest nightly Fx72.0a1. Is there something that can be done for this?

Comment 12

[Dan Minor [:dminor]]

(In reply to Maria Berlinger [:maria_berlinger], Release Desktop QA from comment #11)

The issue is verified fixed using Fx71.0b5 on Windows 10, macOS and Ubuntu 18.04. The capture of skype and chrome windows works without issues now, correctly displaying the content.
However, the issue is still reproducible on Windows 7 using Fx 71.0b5 AND latest nightly Fx72.0a1. Is there something that can be done for this?

The fix uses the PW_RENDERFULLCONTENT flag to PrintWindow which is only available on Windows 8.1 and higher, so unfortunately we would expect this to remain broken on Windows 7. Thank you for your help with testing this.

Comment 13

[Maria Berlinger [:mberlinger], Services QA]

Based on the above comment, marking this as verified fixed.