1941671 - [Linux/Wayland] Implement HDR video playback for SW video decoding / rendering

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement)

Description

[Martin Stránský [:stransky] (ni? me)]

Right now the HDR rendering uses VA-API only. We need to implement it for SW decode / SW rendering too.

Comment 1

[Martin Stránský [:stransky] (ni? me)]

HDR video is decoded to AV_PIX_FMT_YUV420P10LE by SW decoder and we need to convert it to AV_PIX_FMT_P010LE / dmabuf which can be directly composited by Wayland compositor.

Comment 2

[Martin Stránský [:stransky] (ni? me)]

Attachment: wip-hdr.patch

Comment 3

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1941671 Add SurfaceFormat::YUV420P10 to support FFmpeg HDR SW decoded formats r?lsalzman

YUV420P10 is produced by FFmpeg when we decode HDR video formats by SW decoder.

Comment 4

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1941671 Allow to create PlanarYCbCrData from VideoData::YCbCrBuffer r?lsalzman

Comment 5

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1941671 [Linux] Update BlitHelper to convert YUV420P10 to P010 r?lsalzman

Comment 6

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1941671 [Linux] Remove widget/gtk/gbm.h and use third_party/gbm/gbm.h r?emilio

Comment 7

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1941671 [Linux] Implement DMABufSurfaceYUV update and convert from layers::PlanarYCbCrData r?emilio

• Use new third_party/gbm/gbm.h instead of outdated widget/gtk/gbm.h.
• Remove mBufferModifiers array for RGBA formats as it uses ony one modifier for all planes.
• Remove unused DMABufSurfaceYUV::CreateYUVSurface() for pixel data as we use layers::PlanarYCbCrData now.
• Implement DMABufSurfaceYUV::UpdateYUVData() for layers::PlanarYCbCrData and use conversion from YUV420P10 to P010 (HDR).
• Create YUV dmabuf planes with liner modifier by default

Comment 8

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1941671 [Linux] Decode HDR video produced by SW FFmpeg decoder to dmabuf surfaces r?alwu

We wan't to pass HDR produced frames to dmabuf surfaces (YUV420P10 format), convert them on GPU to P010 and pass directly to
Wayland compositor as wl_buffer to enable HDR playback on Linux/Wayland.

Comment 9

[Martin Stránský [:stransky] (ni? me)]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6110ef71c5e62a1cd6a0fbcc39d2771e59f665f3

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e4469e382b83
Add SurfaceFormat::YUV420P10 to support FFmpeg HDR SW decoded formats r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/71e2250a00e8
Allow to create PlanarYCbCrData from VideoData::YCbCrBuffer r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/28fd322cd3e3
[Linux] Update BlitHelper to convert YUV420P10 to P010 r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/30ed10ab92dc
[Linux] Remove widget/gtk/gbm.h and use third_party/gbm/gbm.h r=emilio
https://hg.mozilla.org/integration/autoland/rev/967ff0a140e1
[Linux] Implement DMABufSurfaceYUV update and convert from layers::PlanarYCbCrData r=emilio
https://hg.mozilla.org/integration/autoland/rev/8cb760c9f946
[Linux] Decode HDR video produced by SW FFmpeg decoder to dmabuf surfaces r=alwu

Comment 11

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3c4d0d9b6f88
Backed out 6 changesets for causing bp-nu bustages @GLBlitHelper.cpp. CLOSED TREE

Comment 12

[Martin Stránský [:stransky] (ni? me)]

https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=IYNoMsuBSZe8IUVkiGw_bA.0

/builds/worker/checkouts/gecko/gfx/gl/GLBlitHelper.cpp:174:19: error: unused variable 'kFragSample_TwoPlaneUVP010' [-Werror,-Wunused-const-variable]
/builds/worker/checkouts/gecko/gfx/gl/GLBlitHelper.cpp:266:19: error: unused variable 'kFragConvertYUVP010' [-Werror,-Wunused-const-variable]

Comment 13

[agoloman]

Backed out for causing bp-nu bustages @GLBlitHelper.cpp.

• Backout link
• Push with failures
• Failure Log

Comment 14

[Martin Stránský [:stransky] (ni? me)]

Updated, Thanks.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/018867282c94
Add SurfaceFormat::YUV420P10 to support FFmpeg HDR SW decoded formats r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/8450e297275d
Allow to create PlanarYCbCrData from VideoData::YCbCrBuffer r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/9a62416c10da
[Linux] Update BlitHelper to convert YUV420P10 to P010 r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/9bcfd19641e1
[Linux] Remove widget/gtk/gbm.h and use third_party/gbm/gbm.h r=emilio
https://hg.mozilla.org/integration/autoland/rev/d231f7e99554
[Linux] Implement DMABufSurfaceYUV update and convert from layers::PlanarYCbCrData r=emilio
https://hg.mozilla.org/integration/autoland/rev/3ca00c33b586
[Linux] Decode HDR video produced by SW FFmpeg decoder to dmabuf surfaces r=alwu

Comment 16

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/018867282c94
https://hg.mozilla.org/mozilla-central/rev/8450e297275d
https://hg.mozilla.org/mozilla-central/rev/9a62416c10da
https://hg.mozilla.org/mozilla-central/rev/9bcfd19641e1
https://hg.mozilla.org/mozilla-central/rev/d231f7e99554
https://hg.mozilla.org/mozilla-central/rev/3ca00c33b586

Comment 17

[Robert Mader [:rmader]]

Just as a heads up: I started implementing YUV420P10 and similar formats in Mesa now in order to make conversions to P010 unnecessary: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34303

Closely related to that I'm working on a implementation in Gstreamer that allows to directly share buffer between SW-decoders and GPUs by allocating buffer with udmabuf (which since recently is available by default in many distros) which we'll probably want to adopt in FF: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/8540

Together that allows to improve playback performance for SW-decoded video quite substantially by avoiding unnecessary copies. Upstreaming the formats and the Mesa support will still take some time though.