1758948 - AV1 VA-API playback low framerate

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect, P3)

Description

[serfreeman1337]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0

Steps to reproduce:

1. Set media.ffmpeg.vaapi.enabled to true.
2. Start firefox with the MOZ_DISABLE_RDD_SANDBOX=1 environment variable.
3. Play AV1 encoded video.

Actual results:

Video seems to have low frameware, although youtube reports no frame dropped. It happens in both youtube or if video file was opened directly.

Expected results:

Video should be played with smooth framerate.

Comment 1

[serfreeman1337]

Attachment: MOZ_LOG="PlatformDecoderModule:5"

Additional info:
VLC plays AV1 videos using VA-API just fine.
VA-API VP9 playback also works fine.

vainfo for AV1:
 ```
vainfo: VA-API version: 1.13 (libva 2.13.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 22.2.2 ()
vainfo: Supported config attributes per profile/entrypoint pair
VAProfileAV1Profile0/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV420_10
                                             VA_RT_FORMAT_YUV420_10BPP
    VAConfigAttribDecSliceMode             : VA_DEC_SLICE_MODE_NORMAL
                                             VA_DEC_SLICE_MODE_BASE
    VAConfigAttribDecProcessing            : VA_DEC_PROCESSING_NONE
    VAConfigAttribMaxPictureWidth          : 16384
    VAConfigAttribMaxPictureHeight         : 16384
    VAConfigAttribProcessingRate           : VA_PROCESSING_RATE_DECODE
```

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[serfreeman1337]

My bad, VLC actually fallbacks to software decoding and not using vaapi at all for AV1.
I did another test with mpv and this time AV1 was played smoothly using VAAPI.

intel_gpu_top shows that with mpv actual GPU frequency was 100Mhz:

intel-gpu-top: Intel Tigerlake (Gen12) @ /dev/dri/card0 -   18/ 100 MHz
    46% RC6;      369 irqs/s

Firefox plays AV1:

intel-gpu-top: Intel Tigerlake (Gen12) @ /dev/dri/card0 -   14/  22 MHz
    80% RC6;      374 irqs/s

Comment 4

[Martin Stránský [:stransky] (ni? me)]

Which clip do you test? Can this be a variant of Bug 1750388 ?

Comment 5

[serfreeman1337]

It happens with every youtube's av1 encoded video. Tested from 144p to 1080p.

Example stats for nerds:

Video ID / sCPN: -q7JDqZUxhM / E949 0NE3 JNX9
Viewport / Frames: 688x387 / 0 dropped of 935
Current / Optimal Res: 640x360@30 / 640x360@30
Codecs: av01.0.01M.08 (396) / opus (251)
Color: bt709 / bt709

And its media profile: https://share.firefox.dev/38XIl9h

VP9 vaapi works fine for me.

Comment 6

[Martin Stránský [:stransky] (ni? me)]

Can you test mpv player with va-api enabled?

Comment 7

[serfreeman1337]

Yes, no problem playing same video with mpv.

mpv -v --hwdec=vaapi --ytdl-format="396+251" "https://www.youtube.com/watch?v=-q7JDqZUxhM"
verbose output:

...
[cplayer]  (+) Video --vid=1 (*) (av1 640x360 29.970fps)
[cplayer]  (+) Audio --aid=1 --alang=eng (*) (opus 2ch 48000Hz)
...
[vd] Trying hardware decoding via av1-vaapi.
[vd] Using underlying hw-decoder 'av1'
[vd] Selected codec: libdav1d (dav1d AV1 decoder by VideoLAN)
[vf] User filter list:
[vf]   (empty)
...
[mkv] select track 0
[vd] Pixel formats supported by decoder: cuda vaapi yuv420p
[vd] Codec profile: Main (0x0)
[vd] Requesting pixfmt 'vaapi' from decoder.
[vd] Using hardware decoding (vaapi).
[vd] Decoder format: 640x360 vaapi[nv12] bt.709/bt.709/bt.1886/limited/auto CL=unknown
[vf] [in] 640x360 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] 640x360 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] (disabled)
[vf] [autorotate] 640x360 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [autorotate] (disabled)
[vf] [convert] 640x360 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [convert] (disabled)
[vf] [out] 640x360 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
...

And on intel_gpu_top I see 1.25% to 1.35% usage on Video, while playing same video with Firefox I only see 0.25% to 0.30%.

Comment 8

[Martin Stránský [:stransky] (ni? me)]

Thanks, it's dupe of Bug 1750388 then.

Comment 9

[hujq]

Attachment: Firefox 100.0b9, MOZ_LOG=MediaDecoder:5,PlatformDecoderModule:5

Is this really bug 1750388? The conclusion there seems to be it's only for 8K 60fps, possibly related to hardware capabilities. But this one is affecting video with much lower resolutions and frame rate. I would say it's even more noticeable for videos with fps<30.

The example in comment #7 (can be downloaded with "yt-dlp https://www.youtube.com/watch?v=-q7JDqZUxhM --format=396") is just 640x360 30fps, and certainly nowhere near hardware limitations. Yet with VA-API in Firefox it plays like around 15fps and there is very visible stuttering. Software decoding with Firefox or VA-API with mpv both work as expected. VA-API VP9/H264 is also fine in Firefox, so this is strictly for AV1 too.

Comment 10

[Martin Stránský [:stransky] (ni? me)]

Well, let's reopen it if you feel more comfortable with it. I think it's a duplicate due to va-api threading/timing issue but it does not matter.

Comment 11

[Schlaefer]

I notice the same issue on gen12 AV1 VA-API playback. Just browsing through youtube every video with 25 fps or 30 fps even at or below 1080p (av01.0.05M.08, av01.0.08M.08) shows a very visible stuttering/low framerate. Every video with 60 fps (same as display) plays butter smooth no matter the resolution even at 8k (av01.0.17M.08).

Comment 12

[Martin Stránský [:stransky] (ni? me)]

Thanks. When Bug 1768191 lands please run Firefox on terminal with MOZ_LOG="PlatformDecoderModule:5" env variable and play the slow AV1 decode and attach the log here. It contains playback speed statistics.
Thanks.

Comment 13

[Schlaefer]

Attachment: 102.0a1 (2022-05-09) MOZ_LOG=MediaDecoder:5,PlatformDecoderModule:5.txt

Log with the latest/additional statistics.

File played: https://www.youtube.com/watch?v=m1jY2VLCRmY at 1080p 24fps.

A side by side screen recording showing the stutter I'm experiencing on the right: https://files.catbox.moe/k09y21.mkv

Comment 14

[serfreeman1337]

Attachment: 1080p24 MOZ_LOG=MediaDecoder:5,PlatformDecoderModule:5.txt

My log for playing same video from comment #13.

Codec reported by youtube: av01.0.08M.08 (399)

Comment 15

[serfreeman1337]

Attachment: 1080p60 MOZ_LOG=MediaDecoder:5,PlatformDecoderModule:5.txt

Also can confirm comment #11 that there is no stuttering playing 60fps video.
Log file is attached.

File played: https://www.youtube.com/watch?v=R3AKlscrjmQ at 1080p 60fps.
Codec reported by youtube: av01.0.09M.08 (399)

Comment 16

[Martin Stránský [:stransky] (ni? me)]

Average decode time looks ok, let's log decode times for each frame to check if there's any local delay.

Comment 17

[hujq]

Attachment: decode_time.log

Attaching log of Nightly 2022-05-11 playing downloaded 360p 24fps version of https://www.youtube.com/watch?v=m1jY2VLCRmY

Comment 18

[Martin Stránský [:stransky] (ni? me)]

That definitely doesn't look like decoding issue - frame duration is 41.708 ms but decode times are under 1 ms.

Just out of curiosity, does it help if you flip media.ruin-av-sync.enabled in about:config and restart browser?
Thanks.

Comment 19

[Schlaefer]

Attachment: 1080@24-102.0a1 (2022-05-09) MOZ_LOG=MediaDecoder:5,PlatformDecoderModule:5.txt

Decode times for each frame playing 1080p 24fps https://www.youtube.com/watch?v=m1jY2VLCRmY

media.ruin-av-sync.enabled: No perceivable difference/improvement turning it on (was off).

Comment 20

[serfreeman1337]

Can confirm previous comment. No difference with setting media.ruin-av-sync.enabled to true.
FFVPX: decode time 0.08 ms averange decode time 0.73 ms decoded frames 261

Comment 21

[Martin Stránský [:stransky] (ni? me)]

Please attach content of about:support.

Also do you have set gfx.webrender.compositor / gfx.webrender.compositor.force-enabled in about:config? (that may cause the regression).

Can you try to capture performance of the playback by gecko profiler and share it here?
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems#Firefox_performance_issues

Thanks.

Comment 22

[serfreeman1337]

Attachment: about_support_20220511.txt

about:suppor is attached.

I've only flipped to true following prefs: gfx.webrender.all, gfx.x11-egl.force-enabled, layers.acceleration.force-enabled, media.ffmpeg.vaapi.enabled.

File played: https://www.youtube.com/watch?v=R3AKlscrjmQ

Gecko profiles with "Media" capturing preset:

• 480p at 30fps: https://share.firefox.dev/39eDSz3
• 1080p at 60fps: https://share.firefox.dev/39gkp0X

Comment 23

[Martin Stránský [:stransky] (ni? me)]

Can you try ffplay utility from ffmpeg package with va-api? I wonder if that's a bug in ffmpeg itself.
Thanks.

Comment 24

[Martin Stránský [:stransky] (ni? me)]

btw. I don't see any performance issue there.

Comment 25

[serfreeman1337]

I was only able to use hw av1 by piping it from ffmpeg like this:
ffmpeg -hwaccel vaapi -c:v av1 -i "Japan in 8K-m1jY2VLCRmY.mkv" -vcodec rawvideo -acodec copy -f matroska - | ffplay -i -
No stuttering playing hw accelerated (confirmed by looking at intel_gpu_top) via ffmpeg too.

Comment 26

[hujq]

Attachment: media.ruin-av-sync.enabled=true

media.ruin-av-sync.enabled doesn't seem to make a difference (log attached).

Comment 27

[hujq]

Attachment: about:support.log

about:support attached. Both gfx.webrender.compositor and gfx.webrender.compositor.force-enabled are false.

Gecko profiler when playing https://www.youtube.com/watch?v=m1jY2VLCRmY (360p):
Nightly preset: https://share.firefox.dev/3yB52ep
Media preset: https://share.firefox.dev/37Qh5cO

Comment 28

[hujq]

Attachment: ffmpeg.log

`ffmpeg -hwaccel vaapi -i m1jY2VLCRmY_360p24.mp4 -vcodec rawvideo -f matroska - | ffplay -i -` also has no problem here. GPU usage of ffmpeg shows up in `nvidia-smi`, and only when `-hwaccel vaapi` is used.

Comment 29

[Martin Stránský [:stransky] (ni? me)]

Alastor, any idea how to debug it? Doesn't look like decoding performance issue.
Thanks.

Comment 30

[Alastor Wu [:alwu]]

If we didn't see any drop frames and the decoding performance is good, then I would guess the issue maybe be related with image composition. I know we had some video frame jitter case before which caused a/v unsync/unstable.

Could you help me to capture the profiled result with graphic preset this time? Also, just want to make sure I fully understand this issue, does this issue only happen on AV1, and only happen on low fps (<30)? Is it only happening when VAAPI is on?

Thank you.

Comment 31

[serfreeman1337]

I'm no longer able to play 30fps av1 videos using vaapi in latest nightly version. Video is either stuck or shows infinity load. 60fps works fine though.

Gecko profiles with "Graphics" capturing preset did with version from 2022-05-09:

• 360p at 30fps: https://share.firefox.dev/3MzkdsF
• 1080p at 60fps: https://share.firefox.dev/39EInDB

Yes, it only happens with vaapi and only with <30fps AV1 videos.

Comment 32

[Alastor Wu [:alwu]]

FYI, I also noticed another bug (bug 1770943) where people also reported bad performance on AV1. I will work on that one and hope it would help.

Comment 34

[Martin Stránský [:stransky] (ni? me)]

I can reproduce that after update to Fedora 36.

Comment 35

[Martin Stránský [:stransky] (ni? me)]

Seems to be caused by ffmpeg 5.0 and change of frame timestamps format there. GetFramePts() gives us wrong values.

Comment 36

[Martin Stránský [:stransky] (ni? me)]

It's bug in bundled ffvpx 4.4.1 VA-API AV1 decoder. Setting media.ffvpx.enabled to false and use system ffmpeg 5.0 to decode AV1 fixes that.

Comment 37

[Gurenko Alex]

(In reply to Martin Stránský [:stransky] (ni? me) from comment #36)

It's bug in bundled ffvpx 4.4.1 VA-API AV1 decoder. Setting media.ffvpx.enabled to false and use system ffmpeg 5.0 to decode AV1 fixes that.

I can confirm that on my setup, disabling media.ffvpx setting got AV1 playback performance back.

Comment 38

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1758948 [FFmpeg] Use AVFrame::pts instead of AVFrame::pkt_pts on ffmpeg 4.x r?alwu

AVFrame::pkt_pts has been deprecated and gives us wrong values for AV1 VA-API. Let's use AVFrame::pts instead on ffmpeg 4.x
as well as we use on ffmpeg 5.0 where AVFrame::pkt_pts is removed.

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/89f793811320
[FFmpeg] Use AVFrame::pts instead of AVFrame::pkt_pts on ffmpeg 4.x r=alwu

Comment 40

[bszekely]

https://hg.mozilla.org/mozilla-central/rev/89f793811320

Comment 41

[Monica Chiorean]

I could not reproduce the issue on Ubuntu 20.4 using build 102.0a1(20220511214930) and steps from description. Are there any other steps that I can use in order to reproduce it?
If not can you please confirm issue is not reproducing on latest Beta (https://archive.mozilla.org/pub/firefox/candidates/103.0b7-candidates/) and Nightly(https://archive.mozilla.org/pub/firefox/nightly/2022/07/)? Than you so much.

Comment 42

[serfreeman1337]

No problem with both Beta and Nightly. HW acceleration works great.
Tested with new profile and gfx.webrender.all: 1 (also gfx.x11-egl.force-enabled: 1, media.ffmpeg.vaapi.enabled: 1 for Beta).

Comment 43

[Monica Chiorean]

Mark as verified based on comment#42.

Comment 45

[Jan Klos [:drujd]]

Is this supposed to be fixed in 103.0.2? With my AMD 6800 XT on X11, I am getting ~0.5 sec freezes and many dropped frames on YouTube AV1 4K videos.