2083037 - Wayland: Firefox submits unadvertised NV12/P010 buffers with HDR enabled

Status: ASSIGNED

(Core :: Widget: Gtk, defect)

Description

[Oleksandr Kulkov]

Attachment: Synthetic SDR/HDR gray-pattern clips, playback pages, and format-negotiation log excerpts

Steps to reproduce

Firefox 158.0 (Arch Linux 158.0-1), Build ID 20261013210526, native Wayland, fresh profiles.

1. Run Hyprland PR #16570 at commit 23298506ad216f82cc9800c8651b33dc07be1156, which removes YUV formats from DMA-BUF feedback: https://github.com/hyprwm/Hyprland/pull/16570
2. Set gfx.color_management.hdr=true in a fresh Firefox profile and restart Firefox.
3. Play the SDR and HDR gray-pattern test clips.

Test setup: nested Hyprland with a virtual 10-bit HDR output, AMD GPU. Nightly and a native DRM/KMS session running the PR have not been tested.

Actual results

The compositor advertises neither NV12 nor P010, but Firefox adds both back:

DMABufFormats::EnsureBasicFormat(): 30313050 is missing, adding.
DMABufFormats::EnsureBasicFormat(): 3231564e is missing, adding.

Wayland traces then show create_immed and surface attachment of NV12 buffers for SDR video and P010 buffers for HDR video. The gray patches render red. With gfx.color_management.hdr=false, the HDR clip instead uses RGB buffers and renders neutral gray.

Possible cause: SetModifiersToGfxVars() calls EnsureBasicFormats() on the compositor's format list, adding P010/NV12 even when absent:
https://github.com/mozilla-firefox/firefox/blob/3c71aeacdb296402e41fb2084b13e797d1770642/widget/gtk/DMABufFormats.cpp#L360-L385
https://github.com/mozilla-firefox/firefox/blob/3c71aeacdb296402e41fb2084b13e797d1770642/widget/gtk/DMABufFormats.cpp#L444-L475

Expected results

Firefox should use only compositor-advertised format/modifier pairs for Wayland buffer submission and fall back when NV12/P010 are unavailable. Formats needed internally for decoding should not imply compositor support.

Related: bug 1967366. This report concerns format negotiation; Hyprland's YUV rendering issue is tracked separately in https://github.com/hyprwm/Hyprland/discussions/16420.

Comment 1

[Oleksandr Kulkov]

Attachment: Candidate fix: respect Wayland DMA-BUF formats and render RGB video fallbacks

Attached a candidate patch against Firefox 0f89b57864c7fecffd873697a6bda30b162070df, developed and tested with Codex. The original issue also reproduces on Nightly 160.0a1 (Build ID 20261016094257).

The patch makes two changes:

• Keep optional DMA-BUF format fallbacks off on Wayland, preserving the X11 behavior.
• Pass external_surface.external_image_id.is_some() when WebRender resolves compositor-surface images, so RGB fallback gets real GL textures.

The format change alone stops unadvertised YUV submissions but produces solid green video because the fallback texture creation is skipped. Unmodified Nightly also shows that green failure with image-rendering: pixelated.

The combined patch builds successfully and passes mach format. Local playback tests in nested Hyprland on AMD show RGB fallback with VA-API still active, successful SDR/HDR-clip playback through X11, and P010 with BT.2020/PQ metadata when the compositor advertises it.

With YUV unavailable, this restores RGB fallback without an HDR surface assignment; it does not establish HDR output on that path. Physical HDR accuracy, native DRM/KMS, other GPUs, and software decoding remain untested.

I do not have Try access. Could a GTK/WebRender maintainer give feedback on this approach and run it through Try CI?

Comment 2

[Martin Stránský [:stransky] (ni? me)]

We just need to disable the compositor completely if the GBM_FORMAT_P010/NV12 is missing - we can't show any video in such case.
We can convert YUV to RGBA on Firefox side of course (that's https://phabricator.services.mozilla.com/D322446 for) but I'm afraid you'll get the same issue (https://github.com/hyprwm/Hyprland/issues/16097).

Anyway, we surely need to check the formats are supported before we enable the compositor. I think the best location for it is around this place:
https://searchfox.org/firefox-main/rev/8014960797439064c025f6bbc6a236558bd4f35f/widget/gtk/GfxInfo.cpp#1364

Feel free to submit a patch via phabricator:
https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html

btw. I see hyprland folks complains about AI usage. Mozilla doesn't have such policy - nobody cares how you create the patch, it's just your responsibility to make sure it's correct when you submit it regardless of the actual tool you use.

Comment 3

[Martin Stránský [:stransky] (ni? me)]

btw. YUV->RGB conversion needs compositor support for GBM_FORMAT_ABGR2101010 composition (or float frame buffers but I don't expect hyprland may support that when it doesn't work even with YUV).
It's possible to blend HDR to 8-bit RGBA but it may be a bit crazy.

Comment 4

[Martin Stránský [:stransky] (ni? me)]

Hm, I wonder what happened to the original check at https://phabricator.services.mozilla.com/D253312 and why it doesn't work?

Comment 5

[Oleksandr Kulkov]

Attachment: Bug 2083037 - Respect advertised Wayland video formats and render RGB fallbacks. r?stransky

Only add optional fallback formats on X11. Synthesizing NV12/P010 in Wayland feedback currently makes unsupported video formats appear available.

Require both advertised NV12 and P010 before enabling HDR and the native compositor in GfxInfo. Unsupported compositors keep RGB playback with HDR disabled.

The previous whole-compositor check from D253312 was removed by D258227 (bug 1978540).

Comment 6

[Zahar]

Attachment: Bug 2083037 - [Wayland][HDR] Convert PQ P010 video to RGB if the compositor doesn't advertise P010. r?stransky

Generalize DMABufSurfaceYUV::ConvertHLGToPQ() to ConvertToRGBPQ(): for PQ
input the shader only applies the BT.2020 YUV->RGB matrix and keeps the PQ
signal, so HDR is preserved.

Remember whether the compositor itself advertised P010 and ABGR2101010 before
EnsureBasicFormats() adds them, and convert PQ P010 surfaces to ABGR2101010 when
P010 isn't advertised but ABGR2101010 is.

widget.wayland.hdr.convert-yuv-to-rgb forces the conversion for compositors
that advertise P010 but sample it incorrectly (Hyprland before YUV formats were
removed from its dmabuf feedback).

Comment 7

[Zahar]

I see the same red HDR video on Hyprland, on a different setup: NVIDIA RTX 4070 Ti SUPER, proprietary driver 615.78.08, Hyprland 0.56.2 (still advertises P010/NV12), hardware decode via Vulkan Video. Details of the NVIDIA side are in bug 2083177 (SW decode can't upload to DMABuf) and bug 2083182 (Vulkan Video is the only HW decode path inside the RDD sandbox on NVIDIA).

Disabling the native compositor when P010 isn't advertised (comment 2) fixes the colors but loses HDR entirely. Since bug 2066397 we already have a P010 -> ABGR2101010 shader for HLG, so I generalized it to keep HDR on such compositors:

• DMABufSurfaceYUV::ConvertHLGToPQ() becomes ConvertToRGBPQ(): for PQ input the shader only applies the BT.2020 YUV->RGB matrix and keeps the PQ signal; HLG is unchanged.
• GlobalDMABufFormats remembers whether the compositor itself advertised P010 and ABGR2101010, before EnsureBasicFormats() adds them.
• NativeLayerWaylandExternal::AttachExternalImage() converts PQ P010 surfaces when P010 isn't advertised but ABGR2101010 is (per comment 3, ABGR2101010 is required for this).
• widget.wayland.hdr.convert-yuv-to-rgb (default false) forces the conversion for compositors that advertise P010 but sample it incorrectly, like Hyprland releases before PR #16570.

The case where neither P010 nor ABGR2101010 is advertised and SDR NV12 are not covered by this patch; it's meant to complement D334764.

The two patches currently overlap on one point: D334764 blocks VIDEO_HDR and the native compositor unless NV12 and P010 are advertised, and stops synthesizing P010 on Wayland, so on a compositor without YUV the automatic path in D334769 would never be reached (HDR off, and SupportsDirectComposition(P010) false). If keeping HDR there is wanted, I can rebase D334769 on top of D334764 so that ABGR2101010 is accepted as an alternative to P010 for VIDEO_HDR, and P010 surfaces stay promotable when they will be converted. If disabling HDR on such compositors is the preferred direction, D334769 can be reduced to the PQ conversion plus the pref.

Tested on a mozilla-central build (e77c5131fd) with the patch:

• forced mode: HDR YouTube (640x360 up to 2160p, VP9/AV1) switches the output to HDR with correct colors, fullscreen and windowed.
• format detection logs compositor P010 1 ABGR2101010 1 on Hyprland 0.56.2, so the automatic path correctly stays off there. I couldn't test the automatic conversion itself, since I don't have a Hyprland build with PR #16570.

Also observed on mozilla-central with Vulkan decode: SDR video is promoted as an NV12 external layer with the linear modifier and shows up black on Hyprland 0.56.2 + NVIDIA (the patch doesn't touch NV12). 157 composited the same SDR video itself, so this looks like the NV12 side of this bug.

Patch: https://phabricator.services.mozilla.com/D334769

Comment 8

[Martin Stránský [:stransky] (ni? me)]

Please run on terminal with MOZ_LOG="Dmabuf:5" env variable in HDR mode and attach the log here. I'd like to know which formats are advertised on Hyprland.
Thanks.

Comment 9

[Oleksandr Kulkov]

Attachment: HDR Dmabuf logs: Hyprland PR #16570 and its parent, Firefox 158.0

Attached the Dmabuf:5 logs from the original reproduction with Firefox 158.0, gfx.color_management.hdr=true, and fresh profiles in nested Hyprland on AMD with a virtual 10-bit HDR output.

• With Hyprland PR #16570 (23298506ad21), the feedback has no NV12/P010. Firefox then logs EnsureBasicFormat(): ... is missing, adding for both formats.
• Its parent (19fb395d4531) advertises both NV12 and P010.
• Both advertise ABGR2101010 (0x30334241), including the linear modifier.

The attachment includes the advertised format/modifier lists and the remaining Dmabuf records from both runs; unrelated logging modules are omitted.

Comment 10

[Martin Stránský [:stransky] (ni? me)]

Thanks, will look at it.

Comment 11

[Martin Stránský [:stransky] (ni? me)]

NV12 can be converted to RGBA too. But I think we should use default gecko rendering path for it as it's already supported.

Comment 12

[Martin Stránský [:stransky] (ni? me)]

Bug 2084900 has a solution for it. It uses new flag in compositor host to use standard pipeline with broken YUV direct composition. Will add more patches for the specific Hyprland setup.

Comment 13

[Tymur Boiko]

Regarding the [Wayland][NVIDIA] Firefox crashes with "zwp_linux_buffer_params_v1 error 7". See my comment at Bug 2083037 Comment 8