1783005 - (egl-linux-v4l) [meta][Linux/EGL] Implement ffmpeg/V4L video playback

Status: NEW

(Core :: Audio/Video: Playback, enhancement)

Description

[Robert Mader [:rmader]]

VAAPI will likely never be supported by a wide range of hardware - most importantly ARM devices - due to the API being designed for Intel hardware.

Many ARM devices support V4L2 - however there's at least two APIs here: stateful and stateless.

Another option could be Vulkan, once video decoding extensions become more widely available, see e.g. 1.

1.: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13940/

Comment 1

[Leonard Lausen]

https://source.chromium.org/chromiumos/chromiumos/codesearch/+/main:src/platform/drm-tests/v4l2_stateful_decoder.c is a good example of v4l2 stateful decoder use in Chromium. It's part of the testsuite but thus easier to understand than piles of chrome abstraction layers

Comment 2

[Robert Mader [:rmader]]

Some more random notes from talking to ffmpeg and kernel devs regarding V4L decoding:

• https://mpv.io/ - player with very good VAAPI support, also making use of the post processor API (to reduce GPU work). Apparently working on a V4L backend
• https://github.com/jernejsk/FFmpeg/commits/v4l2-request-hwaccel-4.4 - WIP branch for V4L stateless decoder in ffmpeg
• "And here some hacks by Benjamin to make it work with HEVC: https://gitlab.collabora.com/benjamin.gaignard/FFmpeg/-/commits/hevc-no-slice-decoding"
• "Here an example command using hwaccel with FFmpeg": ffmpeg -hwaccel drm -i resources/JCT-VC-HEVC_V1/AMP_A_Samsung_7/AMP_A_Samsung_7.bin -vf format=pix_fmts=yuv420p -f rawvideo results/JCT-VC-HEVC_V1/AMP_A_Samsung_7.out
• Apparently it shouldn't be too much work to support ffmpeg-v4l backends once they work fully on DRM (i.e. just let give DMABUFs), however it's not clear if that's already the case with upstream ffmpeg atm. Lots of downstream patches flowing around.

Comment 3

[Leonard Lausen]

Besides the WIP branch for V4L stateless decoder in ffmpeg you mention, there is already support for the V4L stateful decoders and encoders in ffmpeg (these are postfixed by m2m):

% ffmpeg -decoders | grep m2m
V..... h263_v4l2m2m V4L2 mem2mem H.263 decoder wrapper (codec h263)
V..... h264_v4l2m2m V4L2 mem2mem H.264 decoder wrapper (codec h264)
V..... hevc_v4l2m2m V4L2 mem2mem HEVC decoder wrapper (codec hevc)
V..... mpeg1_v4l2m2m V4L2 mem2mem MPEG1 decoder wrapper (codec mpeg1video)
V..... mpeg2_v4l2m2m V4L2 mem2mem MPEG2 decoder wrapper (codec mpeg2video)
V..... mpeg4_v4l2m2m V4L2 mem2mem MPEG4 decoder wrapper (codec mpeg4)
V..... vc1_v4l2m2m V4L2 mem2mem VC1 decoder wrapper (codec vc1)
V..... vp8_v4l2m2m V4L2 mem2mem VP8 decoder wrapper (codec vp8)
V..... vp9_v4l2m2m V4L2 mem2mem VP9 decoder wrapper (codec vp9)

% ffmpeg -encoders | grep m2m
V..... h263_v4l2m2m V4L2 mem2mem H.263 encoder wrapper (codec h263)
V..... h264_v4l2m2m V4L2 mem2mem H.264 encoder wrapper (codec h264)
V..... hevc_v4l2m2m V4L2 mem2mem HEVC encoder wrapper (codec hevc)
V..... mpeg4_v4l2m2m V4L2 mem2mem MPEG4 encoder wrapper (codec mpeg4)
V..... vp8_v4l2m2m V4L2 mem2mem VP8 encoder wrapper (codec vp8)

This is with ffmpeg version 5.1-2+b1 Copyright (c) 2000-2022 the FFmpeg developers from Debian Bookworm. On Snapdragon 7c device, ffmpeg uses driver 'qcom-venus' on card 'Qualcomm Venus video encoder' and 'Qualcomm Venus video decoder' respectively.

Comment 4

[wiagn233]

Agree. Having no v4l2 decoding support also blocks raspberry pi's hw accel support and incoming apple m1's hw accel support. rk3588 is also a promising arm Linux device.

Comment 5

[Martin Stránský [:stransky] (ni? me)]

Please create a new tracking bug for that, not VA-API related.

Comment 6

[Robert Mader [:rmader]]

Alright, lets turn this into a tracker bug for what has to be done to support the V4L backends in ffmpeg.

Comment 8

[Robert Mader [:rmader]]

For everyone interested: there's no one actively working on this ATM. I personally find it interesting, but don't know if and when I'll find time to dive into it. Also I don't think Martin Stránský or the Mozilla GFX team will do so in the foreseeable future. So it would greatly help if anyone would like to volunteer or even just play around a bit. If there's some PoC patches that make it work, I'm sure Martin, me or others can help bring things into shape :)

Comment 9

[David Turner]

I have just started working on this, initially targetting H.264 on the Raspberry Pi (and doing zero-copy decoding with Wayland) but it shouldn't be difficult to extend my approach to other codecs and V4L2 accelerators so long as they have working FFmpeg wrapper codecs and support DRM-PRIME exporting. I didn't find this issue in my brief search so I made my own tracking issue at 1833354.

Comment 10

[wiagn233]

Notice that v4l2 has two accelerator modes, stateful(m2m) and stateless(request). This bug cannot be closed with only m2m supported.