1652958 - Update bundled ffvpx library to 4.4.1 to support AV1 (HW) decode.

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Darkspirit]

+++ This bug was initially created as a clone of Bug #1652945 +++

https://github.com/intel/media-driver/commit/9491998f40d496fc458d282f213c0e9e945b8062

"[Decode] This enables HW AV1 decode acceleration on Gen12"

Intel has AV1 fixed function hardware decoding in Gen12 integrated GPU shipping in Tiger Lake later this year and Rocket Lake whenever that is released, Mozilla should add AV1 hardware decoding support when possible.

https://bugzilla.mozilla.org/show_bug.cgi?id=1292374

Previous bug showing how VP9 fixed function hardware decoding support was added for reference.

iHD (intel-media-driver) depends on bug 1619585, which depends on bug 1595994. Currently, only i965 (intel-vaapi-driver) works without sandbox violation.

Comment 1

[GMA]

https://github.com/intel/media-driver/releases/tag/intel-media-20.3.0

"Enabled AV1 4:2:0 8bit and 10bit decoding for both driver and MSDK, up to 8K video decoding and 16K still picture decoding."

https://github.com/Intel-Media-SDK/MediaSDK/releases/tag/intel-mediasdk-20.3.0

"Added AV1 decode support"

Comment 2

[GMA]

https://github.com/intel/media-driver/releases
https://github.com/Intel-Media-SDK/MediaSDK/releases

Intel released new 2021Q1 updates for media driver & media sdk for AV1 hardware decoding.

Comment 3

[Manuel Grießmayr]

There is now VA-API AV1 decode support in Mesa 21.3 Gallium3D:
https://www.phoronix.com/scan.php?page=news_item&px=VA-API-AV1-In-Gallium3D

Comment 4

[Aidan Harris]

I tried to enable this but failed because I don't really know what I'm doing:

diff --git a/dom/media/platforms/agnostic/AgnosticDecoderModule.cpp b/dom/media/platforms/agnostic/AgnosticDecoderModule.cpp
--- a/dom/media/platforms/agnostic/AgnosticDecoderModule.cpp
+++ b/dom/media/platforms/agnostic/AgnosticDecoderModule.cpp
@@ -54,7 +54,9 @@ static bool IsAvailableInRdd(DecoderType
   switch (type) {
 #ifdef MOZ_AV1
     case DecoderType::AV1:
-      return StaticPrefs::media_av1_enabled();
+      // TODO: there should be a media_rdd_av1_enabled preference?
+      //return StaticPrefs::media_av1_enabled();
+      return StaticPrefs::media_rdd_vpx_enabled();
 #endif
     case DecoderType::Opus:
       return StaticPrefs::media_rdd_opus_enabled();
diff --git a/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp b/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp
--- a/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp
+++ b/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp
@@ -11,6 +11,7 @@
 #include "MP4Decoder.h"
 #include "MediaInfo.h"
 #include "VideoUtils.h"
+#include "AOMDecoder.h"
 #include "VPXDecoder.h"
 #include "mozilla/layers/KnowsCompositor.h"
 #ifdef MOZ_WAYLAND_USE_VAAPI
@@ -807,6 +808,11 @@ AVCodecID FFmpegVideoDecoder<LIBAV_VER>:
     return AV_CODEC_ID_VP9;
   }
 #endif
+#if LIBAVCODEC_VERSION_MAJOR >= 58
+  if (AOMDecoder::IsAV1(aMimeType)) {
+    return AV_CODEC_ID_AV1;
+  }
+#endif
 
   return AV_CODEC_ID_NONE;
 }

This attempt failed:

[Child 11209: MediaPDecoder #8]: D/PlatformDecoderModule Initialising VA-API FFmpeg decoder
[Child 11209: MediaPDecoder #8]: D/PlatformDecoderModule Decoder does not support VAAPI device type
[Child 11209: MediaPDecoder #8]: D/PlatformDecoderModule Couldn't find ffmpeg VA-API decoder
[Child 11209: MediaPDecoder #8]: D/PlatformDecoderModule Initialising FFmpeg decoder.

Although I couldn't get vaapi working this did still increase performance because it allowed me to use DMABuf. With DMABuf being used playing back 8K video (https://www.youtube.com/watch?v=zCLOJ9j1k2Y) is possible with a negligible amount of frames dropped (bear in mind my system has a 5950X, I suspect slower systems could still end up dropping a ton of frames). Without DMABuf and using firefox's builtin DAV1DDecoder instead of the systems ffmpeg playback is stutter-fest with hundreds of frames dropped.

I also found a bug in ffmpeg where this test av1 file (https://raw.githubusercontent.com/SPBTV/video_av1_samples/master/spbtv_sample_bipbop_av1_960x540_25fps.mp4) fails to playback correctly in mpv (and potentially firefox if VAAPI can be made to work?), it's fixed in ffmpeg59 (git master) but this version of ffmpeg has multiple things deprecated so it doesn't work properly with firefox.

Comment 5

[Darkspirit]

https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/vaapi_av1.c

Comment 6

[Martin Stránský [:stransky] (ni? me)]

We'd need to update our bundled ffmpeg to 59 (and the media decoder).

Comment 7

[Martin Stránský [:stransky] (ni? me)]

Well, maybe just adding the file may work.

Comment 8

[Martin Stránský [:stransky] (ni? me)]

For AV1 VA-API we need Mesa 21.3 which is not available on Fedora 35 yet. We'd also need to update the libavcodec for that.

Comment 9

[Robert Mader [:rmader]]

(In reply to Martin Stránský [:stransky] (ni? me) from comment #8)

For AV1 VA-API we need Mesa 21.3 which is not available on Fedora 35 yet. We'd also need to update the libavcodec for that.

Mesa 21.3 just made it into updates-testing.

Comment 10

[Martin Stránský [:stransky] (ni? me)]

We need to update whole bundled library to 4.4.1 to support AV1 (HW) decode.

Comment 11

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1652958 Update bundled libavutil library to 4.4.1 r?alwu

Depends on D133420

Comment 12

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1652958 Update bundled libavcodec library to 4.4.1 r?alwu

Depends on D133421

Comment 13

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1652958 Update ffvpx build config to build AV1 VA-API playback when MOZ_WAYLAND is defined r?alwu

Depends on D133422

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d24922f60614
Update bundled libavutil library to 4.4.1 r=alwu
https://hg.mozilla.org/integration/autoland/rev/851e98f06010
Update bundled libavcodec library to 4.4.1 r=alwu
https://hg.mozilla.org/integration/autoland/rev/9a5c5dba4869
Update ffvpx build config to build AV1 VA-API playback when MOZ_WAYLAND is defined r=alwu

Comment 15

[Iulian Moraru]

Backed out for causing multiple build bustages.

Push with failures

Failure log for Windows 2012 AArch64 opt -the same for debug-
Failure log for OS X AArch64 Cross Compiled Shippable opt
Failure log for Android 4.1 ARMv7 opt -the same for debug-
Failure log for Android 5.0 ARMv7 opt -the same for debug-
Failure log for B-gcp Android 5.0 ARMv7 opt -the same for debug-

Backout link

[task 2021-12-22T11:57:56.254Z] 11:57:56     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec'
[task 2021-12-22T11:57:56.256Z] 11:57:56     INFO -  /builds/worker/fetches/clang/bin/lld-link -NOLOGO -DLL -OUT:mozavcodec.dll -PDB:mozavcodec.pdb -SUBSYSTEM:WINDOWS,6.02 -MACHINE:ARM64 @/builds/worker/workspace/obj-build/media/ffvpx/libavcodec/mozavcodec_dll.list mozavcodec.dll.res  -LARGEADDRESSAWARE -DEBUG -PDBALTPATH:%_PDB% -OPT:REF,ICF -guard:cf,nolongjmp   ../libavutil/mozavutil.lib -DEF:mozavcodec.dll.def  user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib
[task 2021-12-22T11:57:56.257Z] 11:57:56     INFO -  lld-link: warning: mpegaudiodecheader.obj: locally defined symbol imported: avpriv_mpa_freq_tab (defined in mpegaudiodata.obj) [LNK4217]
[task 2021-12-22T11:57:56.257Z] 11:57:56     INFO -  lld-link: warning: mpegaudiodecheader.obj: locally defined symbol imported: avpriv_mpa_bitrate_tab (defined in mpegaudiodata.obj) [LNK4217]
[task 2021-12-22T11:57:56.258Z] 11:57:56     INFO -  lld-link: error: undefined symbol: ff_vp9_copy128_aarch64
[task 2021-12-22T11:57:56.259Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:125
[task 2021-12-22T11:57:56.259Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_10bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.260Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:125
[task 2021-12-22T11:57:56.260Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_10bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.261Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:125
[task 2021-12-22T11:57:56.262Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_12bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.262Z] 11:57:56     INFO -  >>> referenced 1 more times
[task 2021-12-22T11:57:56.263Z] 11:57:56     INFO -  lld-link: error: undefined symbol: ff_vp9_copy64_aarch64
[task 2021-12-22T11:57:56.264Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:126
[task 2021-12-22T11:57:56.264Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_10bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.265Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:126
[task 2021-12-22T11:57:56.266Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_10bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.266Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:126
[task 2021-12-22T11:57:56.267Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_12bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.268Z] 11:57:56     INFO -  >>> referenced 3 more times
[task 2021-12-22T11:57:56.268Z] 11:57:56     INFO -  lld-link: error: undefined symbol: ff_vp9_copy32_aarch64
[task 2021-12-22T11:57:56.269Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:127
[task 2021-12-22T11:57:56.270Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_10bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.270Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:127
[task 2021-12-22T11:57:56.271Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_10bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.272Z] 11:57:56     INFO -  >>> referenced by /builds/worker/checkouts/gecko/media/ffvpx/libavcodec/aarch64/vp9dsp_init_16bpp_aarch64_template.c:127
[task 2021-12-22T11:57:56.272Z] 11:57:56     INFO -  >>>               aarch64/vp9dsp_init_12bpp_aarch64.obj:(vp9dsp_mc_init_aarch64)
[task 2021-12-22T11:57:56.273Z] 11:57:56     INFO -  >>> referenced 3 more times
[task 2021-12-22T11:57:56.273Z] 11:57:56    ERROR -  gmake[4]: *** [/builds/worker/checkouts/gecko/config/rules.mk:531: mozavcodec.dll] Error 1
[task 2021-12-22T11:57:56.274Z] 11:57:56     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec'
[task 2021-12-22T11:57:56.275Z] 11:57:56    ERROR -  gmake[3]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:72: media/ffvpx/libavcodec/target] Error 2
[task 2021-12-22T11:57:56.275Z] 11:57:56     INFO -  gmake[3]: *** Waiting for unfinished jobs....
[task 2021-12-22T11:57:56.276Z] 11:57:56     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/media/gmp-clearkey/0.1'

Comment 16

[Martin Stránský [:stransky] (ni? me)]

Will look at it, thanks.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/008fd283bb70
Update bundled libavutil library to 4.4.1 r=alwu
https://hg.mozilla.org/integration/autoland/rev/0539b77bcb0b
Update bundled libavcodec library to 4.4.1 r=alwu
https://hg.mozilla.org/integration/autoland/rev/0128019f5e7d
Update ffvpx build config to build AV1 VA-API playback when MOZ_WAYLAND is defined r=alwu

Comment 18

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c5352079ce27
Backed out 4 changesets (bug 1652958, bug 1745284) for causing osx libavutil related build bustages. CLOSED TREE

Comment 19

[Cosmin Sabou [:CosminS]]

Backed out for causing osx libavutil related build bustages.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Cretry&searchStr=os%2Cx%2Caarch64%2Ccross%2Ccompiled%2Cshippable%2Copt%2Cprofile-guided%2Coptimization%2Cbuilds%2Cbuild-macosx64-aarch64-shippable%2Fopt%2Cb&fromchange=af8465c1e69b13b20b6a9fe17b2babffc64d094d&tochange=5b0b786c19a2bdb92e878918c9ec2dde5688b801&group_state=expanded&selectedTaskRun=QVkbCIKkTheUE3GzT_wgeQ.1

Failure log: https://treeherder.mozilla.org/logviewer?job_id=362179635&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/c5352079ce279fe683e673992ad8a59198157bee

Comment 20

[Martin Stránský [:stransky] (ni? me)]

Thanks.

I didn't see the build-macosx64-aarch64-shippable in ./mach try fuzzy so I missed this one.
But I found ./mach try chooser --full where it's present so I can debug it now.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6865485dae38
Update bundled libavutil library to 4.4.1 r=alwu
https://hg.mozilla.org/integration/autoland/rev/d299ca3cdd7c
Update bundled libavcodec library to 4.4.1 r=alwu
https://hg.mozilla.org/integration/autoland/rev/b7ff43bbee96
Update ffvpx build config to build AV1 VA-API playback when MOZ_WAYLAND is defined r=alwu

Comment 22

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/6865485dae38
https://hg.mozilla.org/mozilla-central/rev/d299ca3cdd7c
https://hg.mozilla.org/mozilla-central/rev/b7ff43bbee96

Comment 23

[Jim]

Is there something specific needed to enable this? I am using 97 and AV1 is not being accelerated. vainfo shows VAProfileAV1Profile0 being supported.

Comment 24

[Martin Stránský [:stransky] (ni? me)]

Not yet finished, tracked by Bug 1745225.