2057751 - [Wayland][HDR] Hardware decoded videos HDR metadata aren’t passed to compositor causing lack of luminance/washed colors

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect)

Description

[Louis-Philippe Manette [:louisphilippe]]

Attachment: firefox-153.0-hdr-metadata.patch

I'm running KDE Wayland Plasma 6.7.3 in HDR mode with an AMD RX 9060 XT on Gentoo (FF 152.0.6), and I ran into the following issues: PQ HDR videos look washed out because of having SDR luminance.

I looked into the code and found that MaxCLL/MaxFALL and MDCV HDR metadata aren't passed to wp_image_description_creator_params_v1_create in WaylandSurface::EnableColorManagementLocked, causing the washed-out colors.

It seems at least that KWin compositor is leveling the light to SDR content if these informations are missing.

I made a local fix to wire HDRMetadata through DMABufSurfaceYUV down to the WaylandSurface.
The fix retrieves HDRMetadata in FFmpeg decoder for VAAPI and Vulkan w/o direct-export (CreateImageVAAPI/CreateImageVulkan) and passes it to wayland compositor.

The fix can be tested with any PQ HDR video on youtube .

For instance
CLLI+MDCV
https://www.youtube.com/watch?v=LXb3EKWsInQ&t=67s
-> Notably the clouds color at the beginning

MDCV (trailer Avatar)
Specifically this video doesn't have CLLI but only MDCV
https://www.youtube.com/watch?v=Ej_kZi6qX9w&t=1s

Comment 1

[Louis-Philippe Manette [:louisphilippe]]

Attachment: about:config

Comment 2

[Louis-Philippe Manette [:louisphilippe]]

Attachment: about:support

Comment 3

[Louis-Philippe Manette [:louisphilippe]]

Attachment: firefox-153.0-hdr-metadata-v2.patch

Comment 4

[Brad Werth [:bradwerth]]

Thank you! I'll take this, reproduce the problem and confirm the fix, and get it landed.

Comment 5

[Louis-Philippe Manette [:louisphilippe]]

Thank you!
Would it help if I upload the patches in Phabricator?

By the way, I'm currently trying to extend the patch to software decoding and direct-export (still work in progress).
I noticed that HDR videos in software decoding and in vulkan+direct-export modes are rendered in 8 bits NV12, just like SDR ones. This causes visible banding on HDR content.
This seems specific to these paths, as VAAPI and Vulkan without direct-export correctly handle HDR in 10-bit (P010).
Is this fallback to NV12 a known limitation or expected behavior?

Comment 6

[Louis-Philippe Manette [:louisphilippe]]

I just checked my about:support and realized that DMABUF_SURFACE_EXPORT is actually blocklisted on my system (Blocklisted; failure code FEATURE_FAILURE_BROKEN_DRIVER).

So I probably wasn't actually testing true direct-export. I assume that since the patch covers Vulkan, direct-export is probably already covered and works correctly in 10-bit (P010).

My question about the 8-bit NV12 banding limitation only applies to the software decoding path!

Comment 7

[Runar]

Getting this too, just giving a reply to support that the bug is there, on a completely different setup, so it's not just your card (and annoyance). 7900XTX, NobaraOS (Fedora based), Plasma Wayland with HDR on, Firefox 152 from RPM. Monitor is an MSI MAG 321UP QD-OLED, EDID says 456 peak / 254 max avg, KWin set to 450 peak and 110 SDR white.

Both the clips you linked wash out for me in Firefox (in addition to a lot of other test videos). Same streams in mpv on the same monitor look fine, looks vibrant and nice there. Spent a while chasing this before I found your bug, so, stuff that made no difference: fullscreen vs windowed, scale 1.0 vs my usual 1.55, Flatpak vs RPM build. All identical.

The useful one though — with gfx.wayland.hdr=false and gfx.color_management.hdr=true the video plays as correct SDR. Looks right, no washing. So decoding and Firefox's own tone mapping seems fine? it might look like just the handoff to the compositor is the problem?

ffprobe on LXb3EKWsInQ (format 701, av01.0.13M.10), metadata is definitely there in the stream:

side_data_type=Mastering display metadata
min_luminance=1/10000
max_luminance=10000000/10000
side_data_type=Content light level metadata
max_content=1100
max_average=180

and mpv's wayland log on the same clip:

[vo/gpu-next/wayland] Compositor supports setting mastering display primaries.
[vo/gpu-next/wayland] target: min_luma=0.005, max_luma=251, max_cll=251, max_fall=261
[vo/gpu-next/wayland] target: min_luma=0.005, max_luma=389, max_cll=389, max_fall=220

by using this command:
mpv --vo=gpu-next --gpu-api=vulkan --target-colorspace-hint=yes --ytdl-format=701 <url>

so kwin clearly takes it, and mpv is updating the target as the scene changes. Firefox just isn't sending anything.

Can test a patched build if it helps.

Comment 8

[Runar]

(Disclaimer: AI was used to help translate/sentence check, english is not my native language, please ask if something is unclear in my comment)

Comment 9

[Martin Stránský [:stransky] (ni? me)]

Please submit the patch via. Phabricator:
https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html
Thanks.

Comment 10

[Louis-Philippe Manette [:louisphilippe]]

Attachment: Bug 2057751 - Hardware decoded videos HDR metadata aren’t passed to compositor causing lack of luminance/washed colors. r=stransky

Comment 11

[Louis-Philippe Manette [:louisphilippe]]

The patch is submitted to Phabricator

Comment 12

[Louis-Philippe Manette [:louisphilippe]]

(In reply to Runar from comment #7)

Getting this too, just giving a reply to support that the bug is there, on a completely different setup, so it's not just your card (and annoyance). 7900XTX, NobaraOS (Fedora based), Plasma Wayland with HDR on, Firefox 152 from RPM. Monitor is an MSI MAG 321UP QD-OLED, EDID says 456 peak / 254 max avg, KWin set to 450 peak and 110 SDR white.

Both the clips you linked wash out for me in Firefox (in addition to a lot of other test videos). Same streams in mpv on the same monitor look fine, looks vibrant and nice there. Spent a while chasing this before I found your bug, so, stuff that made no difference: fullscreen vs windowed, scale 1.0 vs my usual 1.55, Flatpak vs RPM build. All identical.

The useful one though — with gfx.wayland.hdr=false and gfx.color_management.hdr=true the video plays as correct SDR. Looks right, no washing. So decoding and Firefox's own tone mapping seems fine? it might look like just the handoff to the compositor is the problem?

ffprobe on LXb3EKWsInQ (format 701, av01.0.13M.10), metadata is definitely there in the stream:

side_data_type=Mastering display metadata
min_luminance=1/10000
max_luminance=10000000/10000
side_data_type=Content light level metadata
max_content=1100
max_average=180

and mpv's wayland log on the same clip:

[vo/gpu-next/wayland] Compositor supports setting mastering display primaries.
[vo/gpu-next/wayland] target: min_luma=0.005, max_luma=251, max_cll=251, max_fall=261
[vo/gpu-next/wayland] target: min_luma=0.005, max_luma=389, max_cll=389, max_fall=220

by using this command:
mpv --vo=gpu-next --gpu-api=vulkan --target-colorspace-hint=yes --ytdl-format=701 <url>

so kwin clearly takes it, and mpv is updating the target as the scene changes. Firefox just isn't sending anything.

Can test a patched build if it helps.

Thanks a lot for the detailed report and the logs!
It's great to have feedback from a different hardware setup.

I have actually submitted a patch to pass the HDR metadata, which is currently under review here: https://phabricator.services.mozilla.com/D317125

@Stransky, since Runar is offering to test, is there a way I could provide a test build for Nobara?

Comment 13

[Martin Stránský [:stransky] (ni? me)]

Will land in nightly when you rebase it to latest trunk - fails to land now.
Thanks.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/fe77f218df48
https://hg.mozilla.org/integration/autoland/rev/a0679b199d29
Hardware decoded videos HDR metadata aren’t passed to compositor causing lack of luminance/washed colors. r=stransky,gtk-reviewers

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/733b7a68e16c
https://hg.mozilla.org/integration/autoland/rev/8ef452ee9910
Revert "Bug 2057751 - Hardware decoded videos HDR metadata aren’t passed to compositor causing lack of luminance/washed colors. r=stransky,gtk-reviewers" for causing bustage @ TestDMABufSurface.cpp

Comment 16

[Norisz Fay [:noriszfay]]

Backed out or causing bustage @ TestDMABufSurface.cpp

Backout link

Push with failures

Failure log

Comment 17

[Louis-Philippe Manette [:louisphilippe]]

Sorry again for the Lando hiccups earlier, and thanks for landing this!

I'm leaving this here for future reference.
My understanding of the reason why hdr video without setting CLLI or MDCV looked washed is KWin's fallback luminance value for PQ
https://wayland.app/protocols/color-management-v1#wp_color_manager_v1:enum:transfer_function

"This TF implies these default luminances

primary color volume minimum: 0.005 cd/m²
primary color volume maximum: 10000 cd/m²
reference white: 203 cd/m²"

For screens that peak about 1000 or less (basically all of them) Kwin would crush the overall luminosity of the video to fit this dynamic range.
This implies that the small amount of PQ videos on YouTube that lack HDR metadata could still look washed out.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/569d6495ebfd
https://hg.mozilla.org/integration/autoland/rev/01296da0646f
Hardware decoded videos HDR metadata aren’t passed to compositor causing lack of luminance/washed colors. r=stransky,gtk-reviewers

Comment 19

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/01296da0646f

Comment 20

[Louis-Philippe Manette [:louisphilippe]]

(In reply to Runar from comment #7)

Getting this too, just giving a reply to support that the bug is there, on a completely different setup, so it's not just your card (and annoyance). 7900XTX, NobaraOS (Fedora based), Plasma Wayland with HDR on, Firefox 152 from RPM. Monitor is an MSI MAG 321UP QD-OLED, EDID says 456 peak / 254 max avg, KWin set to 450 peak and 110 SDR white.

Both the clips you linked wash out for me in Firefox (in addition to a lot of other test videos). Same streams in mpv on the same monitor look fine, looks vibrant and nice there. Spent a while chasing this before I found your bug, so, stuff that made no difference: fullscreen vs windowed, scale 1.0 vs my usual 1.55, Flatpak vs RPM build. All identical.

The useful one though — with gfx.wayland.hdr=false and gfx.color_management.hdr=true the video plays as correct SDR. Looks right, no washing. So decoding and Firefox's own tone mapping seems fine? it might look like just the handoff to the compositor is the problem?

ffprobe on LXb3EKWsInQ (format 701, av01.0.13M.10), metadata is definitely there in the stream:

side_data_type=Mastering display metadata
min_luminance=1/10000
max_luminance=10000000/10000
side_data_type=Content light level metadata
max_content=1100
max_average=180

and mpv's wayland log on the same clip:

[vo/gpu-next/wayland] Compositor supports setting mastering display primaries.
[vo/gpu-next/wayland] target: min_luma=0.005, max_luma=251, max_cll=251, max_fall=261
[vo/gpu-next/wayland] target: min_luma=0.005, max_luma=389, max_cll=389, max_fall=220

by using this command:
mpv --vo=gpu-next --gpu-api=vulkan --target-colorspace-hint=yes --ytdl-format=701 <url>

so kwin clearly takes it, and mpv is updating the target as the scene changes. Firefox just isn't sending anything.

Can test a patched build if it helps.

The fix has just landed in the latest nightly builds. Could you please test it and confirm if it solves the issue for you?

You can download it here:
https://www.firefox.com/en-US/channel/desktop/#nightly