1745285 - Enable AV1 decoding via bundled ffvpx

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement)

Description

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

As we use DAV1DDecoder we need to claim claim AV1 by ffmpeg only when it's really supported by hardware.

Comment 1

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

Hm, it looks like ffmpeg uses DAV1DDecoder by default for SW decode:
https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/dc0806dd25882f41f6085c8356712f95fded56c7

Maybe we can remove standalone DAV1DDecoder decoder and use ffmpeg which falls back to DAV1DDecoder. In such solution we can use direct decoding to shm (Bug 1713276) or upload frames to dmabuf on Linux.

Alwu, what do you think?

Comment 2

[Alastor Wu [:alwu]]

So does that mean we can rely on ffmpeg decoder for AV1 on both hw & sw decoding?

If so, this sounds good to me, especially considering most AV1 decoding for now are software decoding. But on Windows and MacOS, we don't have ffmpeg decoder enabled, we need to use DAV1Decoder still, right?

In addition, as I don't know much about DAV1Decoder, I would like to hear if Jon has any thought about this (he touched some DAV1Decoder codes before)

Thanks!

Comment 3

[Alastor Wu [:alwu]]

Oh, wait, if bug 1652958 aims to update ffvpx in order to support hw av1 decoding, will it be also possible to support sw av decoding?

Comment 4

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

(In reply to Alastor Wu [:alwu] from comment #2)

So does that mean we can rely on ffmpeg decoder for AV1 on both hw & sw decoding?

Yes.

If so, this sounds good to me, especially considering most AV1 decoding for now are software decoding. But on Windows and MacOS, we don't have ffmpeg decoder enabled, we need to use DAV1Decoder still, right?

I'm sorry, I meant ffvpx (ffvpx is just a subset of ffmpeg). Bug 1652958 is primary about AV1 VA-API decoding but I can test SW AV1 decoding too and update the code path for it so we can use ffvpx to decode AV1 on all platforms (and use your shm patch there for AV1 too).

ffvpx/ffmpeg uses DAV1Decoder internally so we will ship DAV1Decoder as a part of bundled ffvpx anyway.

Comment 5

[Jon Bauman [:jbauman:]]

(In reply to Alastor Wu [:alwu] from comment #2)

In addition, as I don't know much about DAV1Decoder, I would like to hear if Jon has any thought about this (he touched some DAV1Decoder codes before)

My experience with dav1d has been in the context of AVIF, but both our AV1 and AVIF decoding share at least some of the logic in mozilla::DAV1DDecoder. If we can replace that with an interface that both can share and have it use ffvpx, that seems reasonable.

That said, I think the hardware acceleration story is pretty different for video versus still images. It's unlikely to benefit us to use hardware for AVIF and could even result in counterproductive contention with AV1 video, so maybe it's best to leave them separate. At the very least, I think we'd want to make sure we have a way to control whether the caller should be considered for hardware acceleration or not.

Comment 6

[Darkspirit]

• bug 1745228: Can libdav1d be used instead of svt-av1? (And possibly Mozilla's still alive librav1e for AV1 sw encoding in the future?)
• Is removal of vpx possible?
  • bug 1257123 comment 1: On ARM, vpx has fast NEON assembly while ffvpx (Firefox' vp8+vp9 ffmpeg fork) had not back then. Does ffmpeg now have neon assembly for ARM?
  • Is ffvpx now able to handle alpha? So far, vpx is used instead: https://searchfox.org/mozilla-central/search?q=forces+use+of+VPXDecoder+when+alpha+is+present.&path=&case=false&regexp=false

Comment 7

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

Attachment: Bug 1745285 Claim correct libdav1d version r?alwu

Comment 8

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

Attachment: Bug 1745285 Build AV1 decoding support in ffvpx r?alwu

Enable DAV1D decoder in bundled ffvpx and build ffvpx with bundled DAV1D decoder.

Depends on D135033

Comment 9

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

Attachment: Bug 1745285 Implement and use FindAVCodec with explicit hardware decoding param r?alwu

We want to explicitly state hardware decoding setup for decoder to select correct decoder
and avoid failure later.

Depends on D135034

Comment 10

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

Attachment: Bug 1745285 Claim av1 decode support of ffvpx r?alwu

Depends on D135035

Comment 11

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

With the patches here AV1 video decoding will be done by bundled ffvpx (both SW and HW decoding) and can use all features provided by ffvpx (direct decode to shm, decode to dmabuf, VA-API acceleration).

Comment 12

[Robert Mader [:rmader]]

Alastor, just for you to know (as it's not directly obvious and I saw some comment referring to Wayland explicitly): #ifdef MOZ_WAYLAND in fact often doesn't mean we're running Wayland - it's rather says we build with Wayland and thus DRM support. As for hardware video decoding, we require DRM for DMABUF support, however we also use it on X11 with EGL (shipped in 95 for the majority of Mesa users, instead of GLX). Untangling DMABUF from Wayland support is tracked in bug 1677855. Sorry for the noise, just wanted to say that this work affects most Linux (and possibly also *BSD) users, not just those who are already on Wayland :)

Comment 13

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

Looks like there's test failures of ./mach reftest dom/media/test/reftest/color_quads/

Comment 14

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

That fails with:
[RDD 145981: MediaPDecoder #2]: D/PlatformDecoderModule FFVPX: Got one frame output with pts=0 dts=0 duration=40000 opaque=-9223372036854775808
[RDD 145981: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: avcodec_send_packet error: -541478725

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c1c5a7d9a805
Claim correct libdav1d version r=alwu
https://hg.mozilla.org/integration/autoland/rev/817eb128f1ff
Build AV1 decoding support in ffvpx r=alwu
https://hg.mozilla.org/integration/autoland/rev/ff4b2e9ee068
Implement and use FindAVCodec with explicit hardware decoding param r=alwu,media-playback-reviewers
https://hg.mozilla.org/integration/autoland/rev/77ab7af01642
Claim av1 decode support of ffvpx r=alwu,media-playback-reviewers

Comment 16

[Cristian Tuns]

Backed out for causing build bustages.

• Backout link
• Push with failures
• Failure Log
• Failure line: gmake[4]: *** [/builds/worker/checkouts/gecko/config/rules.mk:531: libmozavcodec.so] Error 1

Comment 17

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

Will look at it, Thanks.

I checked at ffvpx <-> dav1d integration and it looks like dav1d ffvpx module does not use get_buffer method needed for Bug 1713276 (direct decode to shm). So let's restrict ffvpx/dav1d decode path to Linux only where we can use VA-API and keep default decoder on other platforms.

Comment 18

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f8de49a397ac860e17daedab5f648b7531a7c5be

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b723d23e9056
Claim correct libdav1d version r=alwu
https://hg.mozilla.org/integration/autoland/rev/9ccc8b7b3b73
Build AV1 decoding support in ffvpx r=alwu
https://hg.mozilla.org/integration/autoland/rev/96f726e72941
Implement and use FindAVCodec with explicit hardware decoding param r=alwu,media-playback-reviewers
https://hg.mozilla.org/integration/autoland/rev/7656625bf5bf
Claim av1 decode support of ffvpx r=alwu,media-playback-reviewers

Comment 20

[Norisz Fay [:noriszfay]]

Backed out for causing av1 reftest failures and build bustage

Backout link: https://hg.mozilla.org/integration/autoland/rev/cbccd71db313485fe2b6a0483636a60f23f07ac0

Push with failures: reftest failures // bustage failure

Failure log(s): reftest log // bustage log

Failure line(s): REFTEST TEST-UNEXPECTED-FAIL | dom/media/test/reftest/color_quads/720p.png.bt709.bt709.tv.yuv420p.av1.webm == dom/media/test/reftest/color_quads/720p.png | image comparison, max difference: 255, number of differing pixels: 273095 // gmake[4]: *** [/builds/worker/checkouts/gecko/config/rules.mk:531: libmozavutil.dylib] Error 1

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/51b1b14a6b1d
Claim correct libdav1d version r=alwu
https://hg.mozilla.org/integration/autoland/rev/b77cd64bad8b
Build AV1 decoding support in ffvpx r=alwu,media-playback-reviewers
https://hg.mozilla.org/integration/autoland/rev/6d4ab14dce8e
Implement and use FindAVCodec with explicit hardware decoding param r=alwu,media-playback-reviewers
https://hg.mozilla.org/integration/autoland/rev/3023ec77ac9a
Claim av1 decode support of ffvpx r=alwu,media-playback-reviewers

Comment 22

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/51b1b14a6b1d
https://hg.mozilla.org/mozilla-central/rev/b77cd64bad8b
https://hg.mozilla.org/mozilla-central/rev/6d4ab14dce8e
https://hg.mozilla.org/mozilla-central/rev/3023ec77ac9a

Comment 23

[mac198442]

Either this needs to be backed out or the patch in bug 1749852 needs to land.

Comment 24

[Martin Stránský [:stransky] (ni? me) (PTO, Back Jun 24)]

bug 1749852 is in Lando now.