1916707 - The screen share stream of getUserMedia / getDisplayMedia Test Page shows an incorrect discoloration on a system with HDR monitor

Status: NEW

(Core :: WebRTC: Audio/Video, defect)

Description

[Daniel Bodea [:danibodea]]

Attachment: mozregression error.png

Note

• The screen share video stream of getUserMedia / getDisplayMedia Test Page shows an incorrect discoloration on a system with HDR monitor.

Found in

• Beta v131.0a1

Affected versions

• Nightly v131.0b1

Tested platforms

• Affected platforms: Windows 10 + HDR monitor
• Unaffected platforms: ?

Steps to reproduce

1. Load https://storage.googleapis.com/desktop_test_assets/TestCases/ScreenShare/ShareScreen.html
2. Click on Start Capture
3. Choose any screen and allow permission.

Expected result

• Correct coloration of video stream is observed.

Actual result

• Incorrect coloration of video stream is observed.

Regression range

• First bad: Tested mozilla-central build: 2024-04-15 (verdict: b)
• Last good: Tested mozilla-central build: 2024-04-14 (verdict: g)
• an exact regressor could not be performed due to an error of mozregression (see attached image).

Additional notes

• This issue appears to be specific to my system. It could not be reproduced on a different Windows 10.
• It might be related to the HDR monitor on my system, like the reporter bug 1912887.

Comment 1

[Andreas Pehrson [:pehrsons]]

With one monitor in HDR and another in sRGB, do you see it:

• when capturing the HDR monitor and rendering on the sRGB monitor, or
• when capturing the sRGB monitor and rendering on the HDR monitor, or
• both

? To figure whether it is a capture or a render problem.

If bug 1888181 is the culprit, then https://hg.mozilla.org/mozilla-central/rev/6b214d143973 looks interesting. The only commit in desktop_capture/win in that update, and HDR capture related. Context: https://issues.chromium.org/issues/40787684.

Could you capture WebRTC logging profiles before and after the regression? Covering the capture setup and a second or so with live capture.
Steps:

• Go to about:logging
• Select the WebRTC Preset
• Click Start Logging
• In another tab, e.g. on https://mozilla.github.io/webrtc-landing/gum_test.html, reproduce the issue.
• Back on about:logging, click Stop Logging.
• In the new tab that appears with the Firefox Profiler, please upload the profile with hidden threads included, and share the links here.

That should tell us what backend is in use, which might give us a clue. The chromium issue linked above mentions the HDR monitor not being available for capture for them (using the dxgi backend), but also that it was not a problem in Firefox (not immediately sure what backend you end up using), prior to the regression.

Comment 2

[Daniel Bodea [:danibodea]]

Unfortunately, I cannot use 2 monitors that are set in HDR and respectively, non-HRD mode. If I turn on HDR, the secondary monitor simply stops working (is not recognized by windows). If I turn Off HRD, then I can use both monitors. I have to mention that the primary monitor is HRD compatible, while the secondary monitor is not.

If I connect both monitors and I turn off HDR mode and run these steps to reproduce, then the issue no longer occurs (the streaming of the screen share displays the correct coloration), So we can deduce that this issue only occurs in HDR more.

WebRTC logging capture when issue IS NOT reproducing (in latest Nightly v132.0a1 while using 2 monitors in non-HDR mode):
https://share.firefox.dev/4eG4g1n

WebRTC logging capture when issue IS reproducing (in latest Nightly v132.0a1 while using 1 monitor is HRD mode):
https://share.firefox.dev/47JUpp2

WebRTC logging capture when issue IS NOT reproducing (in Nightly 126.0a1 (2024-04-14) while using 1 monitor in HRD mode):
https://share.firefox.dev/4gDXrzd

Please let me know if further help is necessary. Thank you.

Comment 3

[Jim Mathies [:jimm]]

Hey Andreas, follow up ping, mind taking a look at these profiles?

Comment 4

[Andreas Pehrson [:pehrsons]]

Sorry for losing track of this. I had a look but the HDR-in-126-no-repro profile doesn't seem to be capturing the screen. I see it enumerating cameras but not much more. No frames are flowing on the Cameras IPC thread in the content process. The other two are fine but this one is important. Could you give that another shot?

Comment 5

[Jim Mathies [:jimm]]

Disabling gfx.webrender.overlay-vp-auto-hdr and restarting fixes this. I think we found the initial cause.

Bug 1882689 - Enable NVIDIA RTX Video TrueHDR in Nightly r=gfx-reviewers,nical
Differential Revision: https://phabricator.services.mozilla.com/D205028

Comment 6

[Jim Mathies [:jimm]]

Attachment: photo of the bug

Comment 7

[Andreas Pehrson [:pehrsons]]

An interesting finding there is that with the "infinity mirror" effect, the discoloration does not increase for each layer. That suggests the capture code is fine, and that the issue is in rendering, which Jim's regression range also suggests. Daniel, could you confirm flipping this pref also fixes the issue for you?

Sotaro, do you have an idea if anything could be done here?

Comment 8

[Jim Mathies [:jimm]]

Daniel, can you confirm you are using a newer nvidia card that supports TrueHDR as well? Thanks.

Comment 9

[Andreas Pehrson [:pehrsons]]

(In reply to Andreas Pehrson [:pehrsons] from comment #7)

An interesting finding there is that with the "infinity mirror" effect, the discoloration does not increase for each layer. That suggests the capture code is fine, and that the issue is in rendering, which Jim's regression range also suggests. Daniel, could you confirm flipping this pref also fixes the issue for you?

Jim pointed out this is incorrect. The chrome and buttons in the second layer are yellowish white even though the yellowish white is no more yellow than in the first layer.

padenot suggested we forward incorrect color space information. This is likely, we make assumptions on color space being BT601 for video capture, and we drop the ICC profile info (links below) we get from the backend when we convert from ARGB to I420 with libyuv.

Sotaro, do you have an idea if anything could be done here?

Maybe you have an idea of how we'd best map the icc_profile bytes to the color space enum?

Comment 10

[Jim Mathies [:jimm]]

Chrome also experiences this issue when capturing windows of Firefox and other applications. However when capturing Chrome windows the problem doesn't reproduce.

Comment 11

[Jim Mathies [:jimm]]

Crazy testing matrix here, but this is what I'm seeing -
Firefox

gfx.webrender.overlay-vp-auto-hdr=false

Firefox sharing window - no issue
Second Firefox window - no issue
Other application window - no issue
Screen - no issue

gfx.webrender.overlay-vp-auto-hdr=true
media.webrtc.capture.allow-directx=true

Firefox sharing window - no issue
Second Firefox window - no issue
Second Firefox instance window - issue present / no issue (hit and miss)
Other application window - no issue
Screen - issue present

gfx.webrender.overlay-vp-auto-hdr=true
media.webrtc.capture.allow-directx=false

Firefox sharing window - no issue
Second Firefox window - no issue
Second Firefox instance window - no issue
Other application window - no issue
Screen - issue present

Chrome

Chrome sharing window - no issue
Second Chrome window - issue present
Other application window - no issue
Screen - issue present

Comment 12

[Andreas Pehrson [:pehrsons]]

Same issue reported on Chromium: https://issues.chromium.org/issues/347991512
It notes the WGC screen capture backend doesn't face this issue. When Jim tried it he still fell back to the directx backend. There are a number of runtime guards preventing the use of the WGC backend, so presumably he hit something there

Re comment 9, the directx backend doesn't set the icc_profile, but it's the only means in the desktop capture framework to signal color space on frames from the backend.

padenot also noted OBS Studio in their windows screen capture backend makes some assumptions on the color space. See e.g. https://github.com/obsproject/obs-studio/blob/80ea1b14fbf09eca19f56f701ab836c3a60b1da5/libobs-d3d11/d3d11-duplicator.cpp#L237 Perhaps there's something we could do in the backend here. Best if someone who can reproduce hammers it out.

I still don't understand how Daniel and Jim found different regression ranges. There may be more than one failure path here.

Comment 13

[Daniel Bodea [:danibodea]]

In my case, disabling gfx.webrender.overlay-vp-auto-hdr and restarting does not fix the issue.
Also, the discoloration does increase for each layer of the "mirror effect.
The issue only occurs when the HDR is enabled in Windows 10 settings (Display / Windows HD Color / Use HDR).
BUT
This issue I am reproducing now also occurs in Chrome and I no longer have the NVIDIA graphics card I had in the past (nvidia geforce gtx 1050).
I am currently using a AMD RX580 8GB:
GPU #1
Active Yes
Description Radeon RX 580 Series
Vendor ID 0x1002
Device ID 0x67df
Driver Version 31.0.21912.14
Driver Date 2-20-2024
Drivers aticfx64 aticfx64 aticfx64 amdxc64 aticfx32 aticfx32 aticfx32 amdxc32 atiumd64 atidxx64 atidxx64 atiumdag atidxx32 atidxx32 atiumdva atiumd6a
Subsys ID e3661da2
RAM 8192

Please let me know if I can help you further.

Comment 14

[Sotaro Ikeda [:sotaro]]

I could reproduce the problem with my Win11 PC.

It seemed that there were 2 points of regression.

At first, I got Bug 1882689. But pref gfx.webrender.overlay-vp-auto-hdr = false, did not address the problem for me.

./mach mozregression -a https://storage.googleapis.com/desktop_test_assets/TestCases/ScreenShare/ShareScreen.html

8:43.55 INFO: No more integration revisions, bisection finished.
8:43.55 INFO: Last good revision: 64f24a096feca3751d8bf9b25cec34afb9d63a5e
8:43.55 INFO: First bad revision: d01a6b91c042bf0f4d48bec38e1c2d50469fb247
8:43.55 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=64f24a096feca3751d8bf9b25cec34afb9d63a5e&tochange=d01a6b91c042bf0f4d48bec38e1c2d50469fb247

Then next, I tried the following, and it showed Bug 1888181

./mach mozregression --pref gfx.webrender.overlay-vp-auto-hdr:false -a https://storage.googleapis.com/desktop_test_assets/TestCases/ScreenShare/ShareScreen.html

7:05.81 INFO: No more integration revisions, bisection finished.
7:05.81 INFO: Last good revision: 99005ed04752410a3b1d0bfba3d8200868bdf9bb
7:05.81 INFO: First bad revision: aa91bdc8664a9357975076b64efa04a94e0d44cf
7:05.81 INFO: Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=99005ed04752410a3b1d0bfba3d8200868bdf9bb&tochange=aa91bdc8664a9357975076b64efa04a94e0d44cf

Comment 15

[Sotaro Ikeda [:sotaro]]

Sotaro, do you have an idea if anything could be done here?

Maybe you have an idea of how we'd best map the icc_profile bytes to the color space enum?

In chromium, ICCProfile::GetColorSpace() is used to get color space. It is called in DesktopCaptureDevice::Core::OnCaptureResult(). And color conversion is basically done by ColorTransform

IIRC, gecko seems not have a class that could be used generally for color conversion with GPU/CPU. For Image data color conversion with ICC, qcms_transform_data() seems to be used.

And when ICC is applied to video, it should not use video overlay. Since its color conversion might not be supported by video overlay(D3D11VideoProcessor). And when video overlay is not used, super resolution is also not applied.

Comment 16

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #15)

IIRC, gecko seems not have a class that could be used generally for color conversion with GPU/CPU. For Image data color conversion with ICC, qcms_transform_data() seems to be used.

Kelsey, :jrmuizel, :aosmond, is there other ways to transform color with ICC in gecko?

Comment 17

[Andreas Pehrson [:pehrsons]]

It seems like DXGI1.5 added HDR support, and a new API in IDXGIOutput5::DuplicateOutput1 (we use the older IDXGIOutput1 version from DXGI1.2) to allow setting the capturing app's supported formats.

The docs say:

The pSupportedFormats array should only contain display scan-out formats. See Format Support for Direct3D Feature Level 11.0 Hardware for required scan-out formats at each feature level. If the current fullscreen buffer format is not contained in the pSupportedFormats array, DXGI will pick one of the supplied formats and convert the fullscreen buffer to that format before returning from IDXGIOutputDuplication::AcquireNextFrame. The list of supported formats should always contain DXGI_FORMAT_B8G8R8A8_UNORM, as this is the most common format for the desktop.

Using this when supported and forcing DXGI_FORMAT_B8G8R8A8_UNORM seems like the easiest path forward, as DXGI should handle conversion to what we expect.

Comment 18

[Jeff Muizelaar [:jrmuizel]]

qcms only supports transforming data on the CPU.

Comment 19

[Sotaro Ikeda [:sotaro]]

From Kelsey in gfx-chat,

---

webgl does cspace conversions on the gpu here: https://searchfox.org/mozilla-central/source/dom/canvas/WebGLContext.cpp#1029-1106

Comment 20

[Andreas Pehrson [:pehrsons]]

We confirmed here that for SDR, according to IDXGIOutut6::GetDesc1, DXGI_OUTPUT_DESC1::ColorSpace is DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709. For HDR it's DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020.

---

Test build at https://treeherder.mozilla.org/jobs?repo=try&revision=924a874345baeb594a3dfb1effc01f513c077881
Log output for both cases:

• SDR: (dxgi_output_duplicator.cc:115): IDXGIOutput6::GetDesc1 return DXGI_OUTPUT_DESC1: Name: \\.\DISPLAY1 Coordinates: [left=0, top=0, right=3840, bottom=2160] AttachedToDesktop: 1 Rotation: 0 Monitor: 10001 BitsPerColor: 10 ColorSpace: DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709 RedPrimary: 0.639648, 0.330078 GreenPrimary: 0.299805, 0.599609 BluePrimary: 0.150391, 0.0595703 WhitePoint: 0.313477, 0.329102 MinLuminance: 0.5 MaxLuminance: 270 MaxFullFrameLuminance: 270
• HDR: (dxgi_output_duplicator.cc:115): IDXGIOutput6::GetDesc1 return DXGI_OUTPUT_DESC1: Name: \\.\DISPLAY1 Coordinates: [left=0, top=0, right=3840, bottom=2160] AttachedToDesktop: 1 Rotation: 0 Monitor: 10001 BitsPerColor: 8 ColorSpace: DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 RedPrimary: 0.683914, 0.312024 GreenPrimary: 0.288168, 0.653056 BluePrimary: 0.155387, 0.0512352 WhitePoint: 0.345701, 0.358535 MinLuminance: 0 MaxLuminance: 370 MaxFullFrameLuminance: 370