Open Bug 1686338 Opened 11 months ago Updated 29 days ago

AVIF (AV1 Image File Format): image sequence and animation support

Categories

(Core :: ImageLib, task, P3)

task

Tracking

()

People

(Reporter: jbauman, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

The current ImageLib-based AVIF implementation in gecko lacks support for image sequences and animations, which are part of the AVIF spec

Severity: -- → S3

Hi Jon, we're seeing some demand for animated support in Chrome, as a potential alternative to mp4 video support in the img tag. If its going to be a while before there's full spec support for AVIF decode in FF, is there an option to prioritize animation support?

Thanks for the feedback! Animation/sequence support is definitely high on the list

Any new info on this?

If firefox doesn't support AVIF image sequences / animation, shouldn't it not load the image sequences / animation instead of displaying a still image? This also doesn't allow the <picture> Tag to load an alternative image (webp or gif). Is there a workaround for this issue? If not, the severity and priority should be S2 i my opinion.

Flags: needinfo?(aosmond)

AVIF images can supply both an image sequence (in a moov box) as well as a still image (indicated as the "primary item" via the pitm box). In the case there's only an image sequence, Firefox indeed displays an error. It's only in the case where a still "primary item" is also provided that this is rendered.

Flags: needinfo?(aosmond)

An AVIF with an image sequence, but no primary item (no pitm box), which Firefox will not display

Attached image sequence-with-pitm.avif

An image sequence with a still image primary item, which Firefox will currently display

Another thing to consider is that an AVIF with both a still primary item and a sequence may be intended to be viewed as either of the two. The way to signal which is the intended is by using the major_brand field of the ftyp box:

  • avif: display the pitm still image (something like the iOS "Live Image" feature)
  • avis: display the image sequence, or if not possible (user agent prefers no motion, for example), display pitm still image

I do think it would be a positive change to make <picture> select a fallback in the case of an AVIF image sequence where the major brand is avis, indicating that the intended use case is animation. It's a question of priorities and available resources, and right now AVIF image sequences aren't very common. The priority to support them would become higher if that changes.

(In reply to Jon Bauman [:jbauman:] from comment #5)

AVIF images can supply both an image sequence (in a moov box) as well as a still image (indicated as the "primary item" via the pitm box). In the case there's only an image sequence, Firefox indeed displays an error. It's only in the case where a still "primary item" is also provided that this is rendered.

Thanks for the reply.
Which one of them is part of the standard? According to your comment here https://github.com/AOMediaCodec/libavif/issues/566#issuecomment-812773655, it seems having a primary item is indeed part of the standard (Please correct me if I'm wrong I'm really new to this).

Flags: needinfo?(jbauman)

They're both part of the standard (or rather the group of standards which collectively define AVIF, which includes ISOBMFF, HEIF, MIAF, etc.). Probably the most relevant piece here is the definitions of the avif and avis brands: https://aomediacodec.github.io/av1-avif/#brands-overview.

To summarize the relevant bits for this discussion:

  • If a file includes avif in its compatible_brands list (or just in the major_brand field), it must have a pitm box, indicating a still image
  • If a file includes avis in its compatible_brands list (or just in the major_brand field), it must have a moov box, indicating an image sequence
  • If a file includes both avif and avis brands, then the intended rendering is indicated by which one is the major_brand; there can be only one.

Does that answer your question?

Flags: needinfo?(jbauman)

(In reply to Jon Bauman [:jbauman:] from comment #8)

It's a question of priorities and available resources, and right now AVIF image sequences aren't very common. The priority to support them would become higher if that changes.

I worry that's the wrong way to think about it. As a developer, I am waiting on better support for AVIF in Firefox and Chrome (HDR10 is over-saturated currently) before I can start using AVIF sequences on websites.

To be clear, lack of current usage isn't the entirety of the reasoning, it's merely one factor. Currently, the dominant use case for short animations on the web is covered by video. This can be done transparently, like the way most (perhaps all?) "GIFs" on twitter, are really mp4 videos. As such, the case for AVIF animation support doesn't necessarily provide that much over what is supported today using AV1 in an mp4 container. Since we have lots more work to do than we have resources to do it, we have to prioritize the things according to how much additional functionality they provide.

So, the fact that this use case is basically already covered via video and there hasn't yet been broad use of animated AVIF in the wild contribute to determining the priority of doing this work. If things change, we will reevaluate.

(In reply to Jon Bauman [:jbauman:] from comment #8)

It's a question of priorities and available resources, and right now AVIF image sequences aren't very common. The priority to support them would become higher if that changes.

There's another way to look at it. As a developer, if I want to use AVIF image sequences, I simply can't in my web app because 1 out of 2 major browsers renders them the way not intended, because the commonly known library libavif capable of producing image sequences produces them that way. There's no workaround for this issue either, since putting sequence-without-pitm.avif inside <picture> tag prevents any source or fallback image from loading. So if developers intend to ship their app with AVIF image sequence, they can't (due to the above mentioned reason), leading to AVIF image sequences being uncommon.

As such, the case for AVIF animation support doesn't necessarily provide that much over what is supported today using AV1 in an mp4 container.

The only major difference is AV1 doesn't support alpha channel. If I want to use an image sequence with transparent background, I can't follow this approach.

Since we have lots more work to do than we have resources to do it, we have to prioritize the things according to how much additional functionality they provide.

I totally understand this, that's why if image sequences support cannot be prioritized, it'd be great if 'not supporting' it properly can be priortized. There hasn't been broad use of animated AVIF, and there won't be (on the web) if this issue isn't fixed. On Safari / Edge, we can simply use the <picture> tag, and It'll render atleast one of the given sources. On Firefox this is not the case, since either a still AVIF image is loaded or no image at all. This eventually prevents many web developers from shipping web apps with AVIF animations just because FIrefox can't handle them correctly, creating a situation of limited use of animated AVIF on web.
This is what I've found in my testing, please let me know if I'm doing it the wrong way, or the the team behind libAVIF so I can raise the issue with them.

Thanks

(In reply to sahil.ms from comment #13)

There's another way to look at it. As a developer, if I want to use AVIF image sequences, I simply can't in my web app because 1 out of 2 major browsers renders them the way not intended, because the commonly known library libavif capable of producing image sequences produces them that way. There's no workaround for this issue either, since putting sequence-without-pitm.avif inside <picture> tag prevents any source or fallback image from loading. So if developers intend to ship their app with AVIF image sequence, they can't (due to the above mentioned reason), leading to AVIF image sequences being uncommon.

It's also the case that Safari (which is the only rendering engine on iOS still) has no AVIF support, so I think there's still a need to handle at least 3 different rendering engines currently. Which I agree is not ideal, but it's not the case that adding animated AVIF support in Firefox will fix that situation.

The only major difference is AV1 doesn't support alpha channel. If I want to use an image sequence with transparent background, I can't follow this approach.

I think this approach is workable today. The tooling isn't as convenient as one would like yet, but I believe the support exists for making a video with transparency with the AV1 codec. You are right that AV1 itself doesn't support an alpha channel, but the ISOBMFF container (aka MP4) does. It's just a question of combining the two AV1 tracks. This issue may be of interest: https://github.com/AOMediaCodec/libavif/issues/777#issuecomment-955022277

…if image sequences support cannot be prioritized, it'd be great if 'not supporting' it properly can be priortized. There hasn't been broad use of animated AVIF, and there won't be (on the web) if this issue isn't fixed. On Safari / Edge, we can simply use the <picture> tag, and It'll render atleast one of the given sources. On Firefox this is not the case, since either a still AVIF image is loaded or no image at all.

You make a good point. I think there's a reasonable argument in favor of treating an AVIF with avis (indicating animation/sequence) as the major brand as "unsupported". The question is what to do with an AVIF that has both an image sequence and a primary item, indicated by an avis major brand in addition to an avif compatible brand. That's a bit of a gray area, but if allowing <picture> fallback on an avis-only AVIF would suffice for your use case, I think we might be able to make that happen. Let me know what you think, and we can create a separate issue to track that work.

Flags: needinfo?(sahil.ms)

(In reply to Jon Bauman [:jbauman:] from comment #14)

You make a good point. I think there's a reasonable argument in favor of treating an AVIF with avis (indicating animation/sequence) as the major brand as "unsupported". The question is what to do with an AVIF that has both an image sequence and a primary item, indicated by an avis major brand in addition to an avif compatible brand. That's a bit of a gray area, but if allowing <picture> fallback on an avis-only AVIF would suffice for your use case, I think we might be able to make that happen. Let me know what you think, and we can create a separate issue to track that work.

Hi Jon, regarding the case of AVIF that has both an image sequence and a primary item, we can assume the intended case is image sequence, since it's highly unlikely for a developer to ship a higher size just to render a single image. Still considering there are people out there who might want to do it, it can be supported inside the <img> (1). I think allowing <picture> fallback on an avis AVIF (that has both an image sequence and a primary item) would really benefit me and other developers (currently Chrome treats AVIF files that has both an image sequence and a primary item as image sequences, and libavif produces the same from what I have tested) (2). This way both mentioned cases will be covered. If that's not possible, even avis-only is sufficient to start rolling out AVIF image sequences. Thanks for mentioning iOS Safari, it's a major browser which I totally ignored. Still, Firefox enabling some workaround for the problem means a lot and is step in the right direction.

Thanks for your consideration.

Flags: needinfo?(sahil.ms)

…regarding the case of AVIF that has both an image sequence and a primary item, we can assume the intended case is image sequence, since it's highly unlikely for a developer to ship a higher size just to render a single image.

I disagree. Consider something like the "live photo" feature on iOS (or image bursts). The primary intent is a single still image, but there's a sequence of images captured around that which allow for viewing the surrounding context or making later revisions of the "key" image. In AVIF, this could be implemented as an avif major brand, but with avis as a compatible brand. When an image has both a primary (still) image and a sequence, we really can't assume what the intent is, but since we have the major brand, we don't have to. This is something I've discussed with the libavif developer and it's already been implemented: https://github.com/AOMediaCodec/libavif/pull/765. So, this is the way chrome behaves as well. I'll upload examples shortly.

Still considering there are people out there who might want to do it, it can be supported inside the <img> (1). I think allowing <picture> fallback on an avis AVIF (that has both an image sequence and a primary item) would really benefit me and other developers

Ok, I'll file an issue and see if we can get that prioritized.

currently Chrome treats AVIF files that has both an image sequence and a primary item as image sequences

It depends on the major brand. You can see in comment 7 what happens when the major brand is avis: Chrome renders the animation, Firefox renders the primary item. However, take that same file and change the major brand to avif (leaving avif in the compatible brands) and both browsers render the primary (still) item.

See Also: → 1739210

(In reply to Jon Bauman [:jbauman:] from comment #17)

Created attachment 9249114 [details]
sequence-with-pitm-avif-major.avif

currently Chrome treats AVIF files that has both an image sequence and a primary item as image sequences

It depends on the major brand. You can see in comment 7 what happens when the major brand is avis: Chrome renders the animation, Firefox renders the primary item. However, take that same file and change the major brand to avif (leaving avif in the compatible brands) and both browsers render the primary (still) item.

Thanks for the clarification. I got confused between have no primary item and changing the major brand.

(In reply to Jon Bauman [:jbauman:] from comment #17)

It depends on the major brand. You can see in comment 7 what happens when the major brand is avis: Chrome renders the animation, Firefox renders the primary item. However, take that same file and change the major brand to avif (leaving avif in the compatible brands) and both browsers render the primary (still) item.

One more question, is there a tool to read avif and modify metadata (like major item here)?

One more question, is there a tool to read avif and modify metadata (like major item here)?

The MP4Box tool which is part of https://github.com/gpac/gpac is really good for this. To change the major brand you can just:

$ MP4Box sequence-with-pitm.avif -out sequence-with-pitm-avif-major.avif -brand avif
You need to log in before you can comment on or make changes to this bug.