1909646 - Performance issues with animated transparent AVIF

Status: UNCONFIRMED

(Core :: Graphics: ImageLib, defect)

Description

[Jake Archibald [:jakea]]

Steps to reproduce:

https://static-misc-3.glitch.me/alpha-video-test/avif-img.html

This is a 60fps animated AVIF with alpha transparency.

Actual results:

Firefox struggles to render at 60fps, particularly on Android.

Expected results:

60fps playback.

An equivalent VP9 plays smoothly:

https://static-misc-3.glitch.me/alpha-video-test/vp9-video.html

Comment 1

[Mayank Bansal]

Nightly (Desktop):
Hw-wr AVIF: https://share.firefox.dev/4bYTdif
Sw-wr AVIF: https://share.firefox.dev/3Yg4Q0I
Hw-wr VP9: https://share.firefox.dev/3Lyo62a
sw-wr VP9: https://share.firefox.dev/4cZDdxO
General dav1d decoding and maybe QCMS is slow?

Nightly (Android)
AVIF: https://share.firefox.dev/4cQIwj1
Vp9: https://share.firefox.dev/3WxKdMi
We perform better than Chrome on the AVIF image on Desktop.
On Android, visually we are faster than Chrome.

Comment 2

[Mayank Bansal]

Attachment: about:support

Comment 3

[Jake Archibald [:jakea]]

Yes, performance is better than Chrome (https://issues.chromium.org/issues/349566435), but it's still way worse than an equivalent AV1 video.

Comment 4

[Jake Archibald [:jakea]]

https://static-misc-3.glitch.me/alpha-video-wc/icon.html - here's a demo where I use an AV1 video and handle the alpha channel myself. It performs much better. It shouldn't be quicker to hand-roll this stuff.

Comment 5

[Mayank Bansal]

(In reply to Jake Archibald from comment #4)

https://static-misc-3.glitch.me/alpha-video-wc/icon.html - here's a demo where I use an AV1 video and handle the alpha channel myself. It performs much better. It shouldn't be quicker to hand-roll this stuff.

This appears to use canvas: https://share.firefox.dev/3WkO12p

Comment 6

[Jake Archibald [:jakea]]

(In reply to Mayank Bansal from comment #5)

(In reply to Jake Archibald from comment #4)

https://static-misc-3.glitch.me/alpha-video-wc/icon.html - here's a demo where I use an AV1 video and handle the alpha channel myself. It performs much better. It shouldn't be quicker to hand-roll this stuff.

This appears to use canvas: https://share.firefox.dev/3WkO12p

Yes. See the source. It takes this video https://cdn.glitch.global/2f4b4e08-395c-470d-aad5-9ba01f7c4597/out-icon.mp4?v=1721828206586 and applies the bottom half as a mask to the top half using webgl.

Doing it in this manually/hacky way shouldn't be faster than browser internals doing a similar thing with a format built to support this use-case.

Comment 7

[Teodor Tanasoaia [:teoxoy]]

Timothy, is there a relevant meta bug this should block?

Comment 8

[Jeff Muizelaar [:jrmuizel]]

It looks like we're spending time doing an sRGB to sRGB conversion on Android.

Comment 9

[Timothy Nikkel (:tnikkel)]

Attachment: ae2-420-s2-qa97.avif

The original avif file in this bug is 1/1/1 (color primaries, transfer functions, matrix coeffs). These are all Bt.709. Weirdly, the macOS preview app reports this avif file as
Color Model: RGB
Profile Name: sRGB IEC61966-2.1

I converted this file to sRGB, specifically to 1/13/6 (bt.709 which is what sRGB uses for primaries, sRGB transfer functions, BT601 coeffs although this doesn't necessarily matter if the goal is sRGB). This is the default of avifenc and what jpeg uses: https://github.com/AOMediaCodec/libavif/wiki/CICP macOS preview reports this new file the same as above.

To do the conversion I first extracted the frames:

for i in {0..478}; do
frame_number=$(printf "%03d" $i)
avifdec --index $i 2.avif frame_$frame_number.png
done

Then encoded:
avifenc --fps 60 -y 420 --qalpha 97 --cicp 1/13/6 -s 2 frame_*.png ae2-420-s2-qa97.avif

I couldn't figure out how to make ffmpeg work for either one (the animation or transparency tripped me up) although there's probably a way to make it work, I couldn't find it before I got avifenc/dec working.

The size of the original file is very impressive! I'm not sure how to recreate that. The smallest that avifenc will produce with this method is around 1mb with -s 2 -q 1 -qa 1.

Comment 10

[Timothy Nikkel (:tnikkel)]

Attachment: 2.avif

The original file in case it goes away.

Comment 11

[Jake Archibald [:jakea]]

There's another demo at https://jakearchibald.com/2024/video-with-transparency/#avif-aint-it in case that's useful

Comment 12

[Timothy Nikkel (:tnikkel)]

(In reply to Jake Archibald from comment #11)

There's another demo at https://jakearchibald.com/2024/video-with-transparency/#avif-aint-it in case that's useful

Thanks for the link. With the command line there I was able to produce much smaller avifs. I wonder what the difference is. I thought maybe avifenc was in intra only mode but that's not true: both avif files have approx the same number of I and P frames (ffprobe -select_streams v -show_frames -show_entries frame=pict_type -of csv output.avif).

With a command line like this:

ffmpeg -i frame_%03d.png -strict -1 -pix_fmt yuva444p -f yuv4mpegpipe - | avifenc --stdin out.avif

the output is as small as I would expect from the ffmpeg only command, so it's not a limitation of avifenc. If I then split up that command line:

ffmpeg -i frame_%03d.png -strict -1 -pix_fmt yuva444p out.y4m && avifenc out.y4m y4mout.avif

I'm back to the large files, which is a head scratcher.

Comment 13

[Jake Archibald [:jakea]]

This issue is about playback performance, not encode size

Comment 14

[Timothy Nikkel (:tnikkel)]

In order to investigate being able to encode my own avifs is important, hence why I am posting that.

Comment 15

[Jake Archibald [:jakea]]

I believe avifenc creates YUV444 via the above methods, whereas the ffmpeg command in the blog post results in YUV420. I wouldn't expect it to be massively different, but I'm not sure what size differences you're referring to.