1686338 - (avifs) AVIF (AV1 Image File Format): image sequence and animation support

Status: RESOLVED WORKSFORME

(Core :: Graphics: ImageLib, enhancement, P3)

Description

[Jon Bauman [:jbauman:]]

The current ImageLib-based AVIF implementation in gecko lacks support for image sequences and animations, which are part of the AVIF spec

Comment 1

[jaikk]

Hi Jon, we're seeing some demand for animated support in Chrome, as a potential alternative to mp4 video support in the img tag. If its going to be a while before there's full spec support for AVIF decode in FF, is there an option to prioritize animation support?

Comment 2

[Jon Bauman [:jbauman:]]

Thanks for the feedback! Animation/sequence support is definitely high on the list

Comment 3

[benjamin.j.grant]

Any new info on this?

Comment 4

[sahil.ms]

If firefox doesn't support AVIF image sequences / animation, shouldn't it not load the image sequences / animation instead of displaying a still image? This also doesn't allow the <picture> Tag to load an alternative image (webp or gif). Is there a workaround for this issue? If not, the severity and priority should be S2 i my opinion.

Comment 5

[Jon Bauman [:jbauman:]]

AVIF images can supply both an image sequence (in a moov box) as well as a still image (indicated as the "primary item" via the pitm box). In the case there's only an image sequence, Firefox indeed displays an error. It's only in the case where a still "primary item" is also provided that this is rendered.

Comment 6

[Jon Bauman [:jbauman:]]

Attachment: sequence-without-pitm.avif

An AVIF with an image sequence, but no primary item (no pitm box), which Firefox will not display

Comment 7

[Jon Bauman [:jbauman:]]

Attachment: sequence-with-pitm.avif

An image sequence with a still image primary item, which Firefox will currently display

Comment 8

[Jon Bauman [:jbauman:]]

Another thing to consider is that an AVIF with both a still primary item and a sequence may be intended to be viewed as either of the two. The way to signal which is the intended is by using the major_brand field of the ftyp box:

• avif: display the pitm still image (something like the iOS "Live Image" feature)
• avis: display the image sequence, or if not possible (user agent prefers no motion, for example), display pitm still image

I do think it would be a positive change to make <picture> select a fallback in the case of an AVIF image sequence where the major brand is avis, indicating that the intended use case is animation. It's a question of priorities and available resources, and right now AVIF image sequences aren't very common. The priority to support them would become higher if that changes.

Comment 9

[sahil.ms]

(In reply to Jon Bauman [:jbauman:] from comment #5)

AVIF images can supply both an image sequence (in a moov box) as well as a still image (indicated as the "primary item" via the pitm box). In the case there's only an image sequence, Firefox indeed displays an error. It's only in the case where a still "primary item" is also provided that this is rendered.

Thanks for the reply.
Which one of them is part of the standard? According to your comment here https://github.com/AOMediaCodec/libavif/issues/566#issuecomment-812773655, it seems having a primary item is indeed part of the standard (Please correct me if I'm wrong I'm really new to this).

Comment 10

[Jon Bauman [:jbauman:]]

They're both part of the standard (or rather the group of standards which collectively define AVIF, which includes ISOBMFF, HEIF, MIAF, etc.). Probably the most relevant piece here is the definitions of the avif and avis brands: https://aomediacodec.github.io/av1-avif/#brands-overview.

To summarize the relevant bits for this discussion:

• If a file includes avif in its compatible_brands list (or just in the major_brand field), it must have a pitm box, indicating a still image
• If a file includes avis in its compatible_brands list (or just in the major_brand field), it must have a moov box, indicating an image sequence
• If a file includes both avif and avis brands, then the intended rendering is indicated by which one is the major_brand; there can be only one.

Does that answer your question?

Comment 11

[chris]

(In reply to Jon Bauman [:jbauman:] from comment #8)

It's a question of priorities and available resources, and right now AVIF image sequences aren't very common. The priority to support them would become higher if that changes.

I worry that's the wrong way to think about it. As a developer, I am waiting on better support for AVIF in Firefox and Chrome (HDR10 is over-saturated currently) before I can start using AVIF sequences on websites.

Comment 12

[Jon Bauman [:jbauman:]]

To be clear, lack of current usage isn't the entirety of the reasoning, it's merely one factor. Currently, the dominant use case for short animations on the web is covered by video. This can be done transparently, like the way most (perhaps all?) "GIFs" on twitter, are really mp4 videos. As such, the case for AVIF animation support doesn't necessarily provide that much over what is supported today using AV1 in an mp4 container. Since we have lots more work to do than we have resources to do it, we have to prioritize the things according to how much additional functionality they provide.

So, the fact that this use case is basically already covered via video and there hasn't yet been broad use of animated AVIF in the wild contribute to determining the priority of doing this work. If things change, we will reevaluate.

Comment 13

[sahil.ms]

(In reply to Jon Bauman [:jbauman:] from comment #8)

It's a question of priorities and available resources, and right now AVIF image sequences aren't very common. The priority to support them would become higher if that changes.

There's another way to look at it. As a developer, if I want to use AVIF image sequences, I simply can't in my web app because 1 out of 2 major browsers renders them the way not intended, because the commonly known library libavif capable of producing image sequences produces them that way. There's no workaround for this issue either, since putting sequence-without-pitm.avif inside <picture> tag prevents any source or fallback image from loading. So if developers intend to ship their app with AVIF image sequence, they can't (due to the above mentioned reason), leading to AVIF image sequences being uncommon.

As such, the case for AVIF animation support doesn't necessarily provide that much over what is supported today using AV1 in an mp4 container.

The only major difference is AV1 doesn't support alpha channel. If I want to use an image sequence with transparent background, I can't follow this approach.

Since we have lots more work to do than we have resources to do it, we have to prioritize the things according to how much additional functionality they provide.

I totally understand this, that's why if image sequences support cannot be prioritized, it'd be great if 'not supporting' it properly can be priortized. There hasn't been broad use of animated AVIF, and there won't be (on the web) if this issue isn't fixed. On Safari / Edge, we can simply use the <picture> tag, and It'll render atleast one of the given sources. On Firefox this is not the case, since either a still AVIF image is loaded or no image at all. This eventually prevents many web developers from shipping web apps with AVIF animations just because FIrefox can't handle them correctly, creating a situation of limited use of animated AVIF on web.
This is what I've found in my testing, please let me know if I'm doing it the wrong way, or the the team behind libAVIF so I can raise the issue with them.

Thanks

Comment 14

[Jon Bauman [:jbauman:]]

(In reply to sahil.ms from comment #13)

There's another way to look at it. As a developer, if I want to use AVIF image sequences, I simply can't in my web app because 1 out of 2 major browsers renders them the way not intended, because the commonly known library libavif capable of producing image sequences produces them that way. There's no workaround for this issue either, since putting sequence-without-pitm.avif inside <picture> tag prevents any source or fallback image from loading. So if developers intend to ship their app with AVIF image sequence, they can't (due to the above mentioned reason), leading to AVIF image sequences being uncommon.

It's also the case that Safari (which is the only rendering engine on iOS still) has no AVIF support, so I think there's still a need to handle at least 3 different rendering engines currently. Which I agree is not ideal, but it's not the case that adding animated AVIF support in Firefox will fix that situation.

The only major difference is AV1 doesn't support alpha channel. If I want to use an image sequence with transparent background, I can't follow this approach.

I think this approach is workable today. The tooling isn't as convenient as one would like yet, but I believe the support exists for making a video with transparency with the AV1 codec. You are right that AV1 itself doesn't support an alpha channel, but the ISOBMFF container (aka MP4) does. It's just a question of combining the two AV1 tracks. This issue may be of interest: https://github.com/AOMediaCodec/libavif/issues/777#issuecomment-955022277

…if image sequences support cannot be prioritized, it'd be great if 'not supporting' it properly can be priortized. There hasn't been broad use of animated AVIF, and there won't be (on the web) if this issue isn't fixed. On Safari / Edge, we can simply use the <picture> tag, and It'll render atleast one of the given sources. On Firefox this is not the case, since either a still AVIF image is loaded or no image at all.

You make a good point. I think there's a reasonable argument in favor of treating an AVIF with avis (indicating animation/sequence) as the major brand as "unsupported". The question is what to do with an AVIF that has both an image sequence and a primary item, indicated by an avis major brand in addition to an avif compatible brand. That's a bit of a gray area, but if allowing <picture> fallback on an avis-only AVIF would suffice for your use case, I think we might be able to make that happen. Let me know what you think, and we can create a separate issue to track that work.

Comment 15

[sahil.ms]

(In reply to Jon Bauman [:jbauman:] from comment #14)

You make a good point. I think there's a reasonable argument in favor of treating an AVIF with avis (indicating animation/sequence) as the major brand as "unsupported". The question is what to do with an AVIF that has both an image sequence and a primary item, indicated by an avis major brand in addition to an avif compatible brand. That's a bit of a gray area, but if allowing <picture> fallback on an avis-only AVIF would suffice for your use case, I think we might be able to make that happen. Let me know what you think, and we can create a separate issue to track that work.

Hi Jon, regarding the case of AVIF that has both an image sequence and a primary item, we can assume the intended case is image sequence, since it's highly unlikely for a developer to ship a higher size just to render a single image. Still considering there are people out there who might want to do it, it can be supported inside the <img> (1). I think allowing <picture> fallback on an avis AVIF (that has both an image sequence and a primary item) would really benefit me and other developers (currently Chrome treats AVIF files that has both an image sequence and a primary item as image sequences, and libavif produces the same from what I have tested) (2). This way both mentioned cases will be covered. If that's not possible, even avis-only is sufficient to start rolling out AVIF image sequences. Thanks for mentioning iOS Safari, it's a major browser which I totally ignored. Still, Firefox enabling some workaround for the problem means a lot and is step in the right direction.

Thanks for your consideration.

Comment 16

[Jon Bauman [:jbauman:]]

…regarding the case of AVIF that has both an image sequence and a primary item, we can assume the intended case is image sequence, since it's highly unlikely for a developer to ship a higher size just to render a single image.

I disagree. Consider something like the "live photo" feature on iOS (or image bursts). The primary intent is a single still image, but there's a sequence of images captured around that which allow for viewing the surrounding context or making later revisions of the "key" image. In AVIF, this could be implemented as an avif major brand, but with avis as a compatible brand. When an image has both a primary (still) image and a sequence, we really can't assume what the intent is, but since we have the major brand, we don't have to. This is something I've discussed with the libavif developer and it's already been implemented: https://github.com/AOMediaCodec/libavif/pull/765. So, this is the way chrome behaves as well. I'll upload examples shortly.

Still considering there are people out there who might want to do it, it can be supported inside the <img> (1). I think allowing <picture> fallback on an avis AVIF (that has both an image sequence and a primary item) would really benefit me and other developers

Ok, I'll file an issue and see if we can get that prioritized.

Comment 17

[Jon Bauman [:jbauman:]]

Attachment: sequence-with-pitm-avif-major.avif

currently Chrome treats AVIF files that has both an image sequence and a primary item as image sequences

It depends on the major brand. You can see in comment 7 what happens when the major brand is avis: Chrome renders the animation, Firefox renders the primary item. However, take that same file and change the major brand to avif (leaving avif in the compatible brands) and both browsers render the primary (still) item.

Comment 18

[sahil.ms]

(In reply to Jon Bauman [:jbauman:] from comment #17)

Created attachment 9249114 [details]
sequence-with-pitm-avif-major.avif

currently Chrome treats AVIF files that has both an image sequence and a primary item as image sequences

It depends on the major brand. You can see in comment 7 what happens when the major brand is avis: Chrome renders the animation, Firefox renders the primary item. However, take that same file and change the major brand to avif (leaving avif in the compatible brands) and both browsers render the primary (still) item.

Thanks for the clarification. I got confused between have no primary item and changing the major brand.

Comment 19

[sahil.ms]

(In reply to Jon Bauman [:jbauman:] from comment #17)

It depends on the major brand. You can see in comment 7 what happens when the major brand is avis: Chrome renders the animation, Firefox renders the primary item. However, take that same file and change the major brand to avif (leaving avif in the compatible brands) and both browsers render the primary (still) item.

One more question, is there a tool to read avif and modify metadata (like major item here)?

Comment 20

[Jon Bauman [:jbauman:]]

One more question, is there a tool to read avif and modify metadata (like major item here)?

The MP4Box tool which is part of https://github.com/gpac/gpac is really good for this. To change the major brand you can just:

$ MP4Box sequence-with-pitm.avif -out sequence-with-pitm-avif-major.avif -brand avif

Comment 21

[Caspy7]

FYI Kickstarter has now switched over to using only AVIF for animated images over a certain size (uncertain what, somewhere under 1MB though).

https://twitter.com/Enichan/status/1500176971141111815

All Firefox users are now excluded from viewing these images (and Kickstarter directly indicated they have no intention of fixing this).

Comment 22

[Jon Bauman [:jbauman:]]

Kickstarter directly indicated they have no intention of fixing this

What's your source for this information? I've spoken to a contact at KSR directly, and according to them this change wasn't initiated by them, but rather by their CDN. KSR has already overridden the change and is serving GIFs to all user agents currently. I'm pursuing discussion with the relevant parties to get this sorted and avoid similar impacts at other sites which use the same CDN.

Comment 23

[Szmozsánszky István [:flaki]]

Not sure if Hashnode might be using the same CDN but the large (4 megapixel) animated GIF here is being served back as AVIF and that breaks the animation in Firefox.

The CDN URL requested is https://cdn.hashnode.com/res/hashnode/image/upload/v1653572083885/SQk5a39bj.gif?auto=format,compress&gif-q=60&format=webm.

Comment 24

[Ksenia Berezina [:ksenia]]

Setting webcompat priority as as P3 for now as the only site breakage report we got was with kickstarter and it is now fixed by the site.

Comment 25

[sadff]

The funding site tumblbug, similar to Kickstarter, also uses avif animated images, and Firefox users can only see still images.

I hope Firefox won't be the successor to Internet Explorer.

Comment 26

[Kestrel]

I can get tumblbug to fall back to webp by setting image.avif.enabled = false and refreshing without the cache (Ctrl+F5) but can confirm the default configuration shows static or broken images.

Comment 27

[chris]

FFmpeg can now create avif animations directly and will increase their popularity as well.

Comment 28

[Mathias Bynens]

Since animated AVIF files now use the same extension and MIME type as non-animated AVIF images, this bug constitutes an interoperability / web compatibility issue. Consider the following HTML, where the developer properly makes use of <picture> to let the browser pick the most appropriate image format to display an animation:

<picture>
  <source srcset="animation.avif" type="image/avif">
  <source srcset="animation.webp" type="image/webp">
  <source srcset="animation.apng" type="image/apng">
  <img src="animation.gif" width="420" height="420" alt="…">
</picture>

In browsers with animated AVIF support, and browsers without AVIF support, this renders an animation.
In Firefox, it renders a still image, since it supports AVIF but not animated AVIF.

Comment 29

[Mathew Hodson]

(In reply to Mathias Bynens from comment #28)

In browsers with animated AVIF support, and browsers without AVIF support, this renders an animation.
In Firefox, it renders a still image, since it supports AVIF but not animated AVIF.

That's bug 1739210.

Comment 30

[oliverwilliams345]

Safari 16 (released September 7) will support animated AVIF so "Setting webcompat priority as as P3 for now as the only site breakage report we got was with kickstarter and it is now fixed by the site" seems a bit short-sighted.

Comment 31

[Mathias Bynens]

(In reply to oliverwilliams345 from comment #30)

Safari 16 (released September 7) will support animated AVIF so "Setting webcompat priority as as P3 for now as the only site breakage report we got was with kickstarter and it is now fixed by the site" seems a bit short-sighted.

It’s true that back in July, the WebKit team stated their implementation supports animated AVIF: https://twitter.com/jensimmons/status/1548996915110592515

Yes, our implementation does include support for [AVIF] image sequences.

But now, the official WebKit announcement suggests that AVIF support is limited to still images:

Safari for iOS 16 includes support for still images compressed using the AVIF format.

I’m worried this causes the same web compat issue we’ve been discussing here, although at a much larger scale since Safari’s AVIF support is enabled by default. :(

Comment 32

[Kornel]

HEIF has image/heif and image/heif-sequence MIME types, so image/avif-sequence could theoretically still be used in the future to indicate support.

However, I think Safari gives a great opportunity to drop support for AVIF sequences from the Web platform.

AVIF "animations" are redundant and entirely unnecessary. It's the same MP4 container and the same AV1 codec. They don't bring anything valuable to the platform, only require different tooling and implementation busywork.

Browsers can still support MP4+AV1 in <img> by supporting these when they're tagged as a regular video file.

Comment 33

[sahil.ms]

https://developer.apple.com/documentation/safari-release-notes/safari-16_1-release-notes

New Features

• Added support for non-animated AVIF on macOS Ventura, and iPadOS 16.
• Added support for AVIF animated image sequences on macOS Ventura, iPadOS 16, and iOS 16.

Never though I'd see a day when Safari would surpass Firefox in terms of some standard adoption.

Comment 34

[Jeff Muizelaar [:jrmuizel]]

We're going to be prototyping this in bug 1788119

Comment 35

[[e-mail]]

Another sample which is not working yet:
https://joedrago.github.io/red_green_flash.avif

Comment 36

[Timothy Nikkel (:tnikkel)]

So with bug 1825580 landing and 113 being released for a while now we have animated avif support on release.

Is there any reason to keep this bug open to track anything? Or should we close it?

Comment 37

[Zaggy1024]

I think it should be fine to close this. IIRC, rotation/mirroring doesn't work for still images the same as animated ones, so that can be tracked generally for AVIF.

For selection of the color track, it looks like libavif still hasn't changed their methodology here, so I think the mp4parse implementation is fine. I was previously concerned that that may changed to be based on some ID specified by the pitm box or elsewhere in the format.

I need to have another look at Bug 1788866 since animated AVIF is often used with an alpha plane, moreso than still images, but that also affects still images, I believe.