1696090 - AVIF grid-based images are not supported

Status: NEW

(Core :: Graphics: ImageLib, defect)

Description

[Jon Bauman [:jbauman:]]

While most AVIF files use an av01 box as their primary item, it's also possible to construct an AVIF out of many av01 boxes arranged in a grid pattern. For example: https://github.com/AOMediaCodec/av1-avif/blob/master/testFiles/Microsoft/Summer_in_Tomsk_720p_5x4_grid.avif

Comment 1

[Jeff Muizelaar [:jrmuizel]]

FWIW, Chrome supports this file.

Comment 2

[Jeff Muizelaar [:jrmuizel]]

Do we know what authoring tools produce these files?

Comment 3

[Jon Bauman [:jbauman:]]

Looks like libavif's avifenc tool does since late 2020

Comment 4

[Jeff Muizelaar [:jrmuizel]]

But it looks like it's opt-in and won't happen by default?

Comment 5

[Jon Bauman [:jbauman:]]

Correct. Somewhat related, the max image size for avifenc is currently 16,384 x 16,384, but there's an open issue to make it configurable. See also this discussion regarding maximum image sizes for AVIF https://bugzilla.mozilla.org/show_bug.cgi?id=1443863#c34.

Comment 6

[Yuan Tong]

There is a tool called Paint.NET that always produces grid avif image if condition is satisfied (roughly when the image width or height >512 and is a multiple of an even number).

By the way, grid images can be rendered grid by grid, and we may want to do it.

Comment 7

[Jon Bauman [:jbauman:]]

There is a tool called Paint.NET that always produces grid avif image if condition is satisfied (roughly when the image width or height >512 and is a multiple of an even number).

Huh, that seems like a weird behavior. Any idea why it does that for relatively small images?

By the way, grid images can be rendered grid by grid, and we may want to do it.

I'm not sure I understand what you mean by this. Can you elaborate?

Comment 8

[Yuan Tong]

Any idea why it does that for relatively small images?
It's an GUI application so it wants more responsiveness. libaom encoding can not be cancelled, and on big images it's slow and memory hungry. The author wants to reduce memory usage, allow cancellation during encoding and provide progress info.

Can you elaborate?
Grid is only a container level thing, each grid is an AV1 stream that can be decoded independently. The idea is: when we received data of some grid, we can decode this grid and render the pixels in this grid first, instead of waiting until all grids are received and decoded before we render the image.

Comment 10

[Timothy Nikkel (:tnikkel)]

FWIW, Safari (Version 16.5 (17615.2.9.11.6, 17615), the latest on my system) does not display the grid image from comment 0.

Comment 11

[Claus Andersen]

I would like to argue that this issue is much worse than it seems at first look. And it can really hurt Firefox reputation or avif image adoption in general.

1. It does not only happen for relatively small images (you can always argue relativism :-)). It also happens in sizes 4K and above which in web context currently is considered big.
2. The authors of the imaging software claims that: "It is Firefox that is not following the spec, by their AVIF decoder not supporting tiled/grid images"
3. It fails silently in the most common usecase!!!

ad 1: I have attached an extremely simple sample image which fails. It is a 1 KB 1920x908 blank white canvas generated by Paint.NET

ad 2: You can see the claim here: https://github.com/0xC0000054/pdn-avif/issues/24#issuecomment-1928207603
If that is the case then this issue should be added as a dependency for 1701426 (The meta issue for avif compliance).

ad 3: This was really painful as I ended up spending a lot of time debugging. When you know what is wrong it is easy to test but Firefox is not as helpful as could be. I am unsure if this is best described as an avif issue or a more general problem with source sets. The behaviour is the same for the HTML tag img srcset and when doing CSS background-image: image-set.

These images opens and re-opens perfectly in the editor (Paint.NET) and XnView. It is easy to discover when you know if you open it directly (right-click open in) in Firefox as it clearly displays an error. As avif is new and shiny I would however claim that the image stack below is the more common approach:

<img src="img/radio1920x908.png"
     alt=""
     srcset="img/radio5604x2650.avif 5604w,
             img/radio5120x2421.avif 5120w,
             img/radio4096x1937.avif 4096w,
             img/radio3840x1816.avif 3840w,
             img/radio2560x1211.avif 2560w,
             img/radio2048x968.avif  2048w,
             img/radio1920x908.avif  1920w,
             img/radio1920x908.png   1920w,
             img/radio1280x605.avif  1280w,
             img/radio1280x605.png   1280w,
             img/radio640x303.avif    640w,
             img/radio640x303.png     640w,
             img/radio320x151.avif    320w,
             img/radio320x151.png     320w"
     sizes="100dvw">

This works as expected in a certain Blink based browser. An avif image is fetched and displayed for all resolutions when changing the viewport. Have not tested Webkit. With Goanna it does pick up the srcset but tries the avif which they do not support. The png fallback does not work as hoped.

It is the same image resized to different resolutions and then saved using the same settings using medium profile to get the optimal filesize.

In Firefox you will see the image for 640w, 1280w, 2560w, 4096w easily on a 4K monitor. And by using mobile responsive view you can verify 320w and 5120w.

But for 1920w, 2048w, 3840w and 5604w it shows nothing and blanks the image!

As I have done image stacks before (especially for webp) I spent a lot of time looking into this. I had a very clear expectation that any failures would be logged to the console or somehow be prominently displayed in developer tools. Maybe I need to manage my expectations but I consider this a bug in how errorhandling is done with srcset. We can spin it and call it "sub-optimal". But cursing was involved :-D.

In Network Tools we see GET with imageset as the Initiator. Bytes are transferred as expected when we resize. If we have been at a resolution before we see "Cached" as expected. If we are too fast we see "$bytes (raced)". All with a status of "200 OK". If we are much too fast then we get "NS-BINDING_ABORTED" with a failed status. All very good and as expected.

No errors anywhere. Nothing in the console. Only headscratching.

With this new after the fact knowledge I found another route. It is a smarter way but I think it is buried too deep (said the village idiot :-)).

Go to the Firefox Profiler. You will not see it in the default view for "Call Tree". But if you select "All frames" or "Native" for the "Call Tree" you will now see an additional entry after "imgLoader::LoadImage". A little gray "mozilla::image::RasterImage::DoError". DoError Does not Do what (I think) it should do.

If you venture further into the profiler we can look at the "Marker Table". You will see offending images with a "Load" and think things are OK. But you need to look for what you cannot see. There are no "Image Paint" for the offending images.

Expected behaviour to me:
A) Prominent display of error in the developer tools. Preferably the console log.
B) A display of the broken image icon when rendering fails.
C) This is debateable: But I expected a fallback to the next when multiple formats are supplied for the same width in the srcset.

Comment 12

[Claus Andersen]

Attachment: 1920testblank.avif

This is a 1 KB small avif test image generated by Paint.NET. The canvas size is 1920x908 and is just plain white when it renders.
It works with Paint.Net, XnView and Chrome. But not in Firefox. This is expected to be related to this tile issue.

Comment 13

[Claus Andersen]

Jon Bauman [:jbauman:] does not seem to be around anymore and this bug is still open and unassigned. As I think the srcset errorhandling is a problem in itself I have created 1878852 (srcset errors not reported in console) for the DevTools/console people.

Comment 14

[Emilio Cobos Álvarez (:emilio)]

You could argue this is a defect in our avif decoder, in a way.

Comment 15

[Claus Andersen]

I would claim that there is no argument. It is a straight up defect. Firefox does not adhere to the spec.

There seems to be a misunderstanding that this only concerns "grid-like" images. Especially the "Summer in Tomsk" is an unfortunate example as it contains a "grid" of images which is easy to understand for the viewer. And the implementer might consider this an uncommon scenario.

The problem here is that avif may be encoded in tiles. These tiles can be part of a regular contiguous image which might not be a specific grid like structure.

But Yuan Tong clearly stated it previously : There are encoders in the wild which will encode images with tiles. Common image sizes. Regular pictures.

Firefox does does not decode these images. Hence Firefox only has partial avif support. And it is an important part which is missing - not just an edge case.

Comment 16

[Timothy Nikkel (:tnikkel)]

(In reply to Timothy Nikkel (:tnikkel) from comment #10)

FWIW, Safari (Version 16.5 (17615.2.9.11.6, 17615), the latest on my system) does not display the grid image from comment 0.

An update on this: it seems that they've added support for this in macOS 13 (macOS 12 still does not support it).

Comment 17

[Yuan Tong]

I'd like to clarify that under AVIF context "grid" and "tile" are usually used to refer different things. "Tile" is an AV1 feature (in one AV1 bitstream, encode each cell independently), while "Grid" is an AVIF feature (encode each cell in it's own AV1 bitstream). Firefox decodes AVIF images with tiling (that is, encoded with --tilerowslog2 and --tilecolslog2 using avifenc for example) just fine, but failed to decode Grid AVIF images (encoded with --grid).

As for the ubiquity of grid AVIF images, I'd like to mention that nowadays HEIF images taken by phones are almost always grid images. Once mobile SoCs starting to support hardware AV1 encoding, there will likely be some manufactures deciding to support taking AVIF images, therefore an significant increase of popularity of grid AVIF images.

Comment 18

[petr.minar]

Just to give you my opinion on AVIF files with "Grid". Our program generates them in this way, so the images are not usable in Firefox (of course that is the reason I am looking at this bug report now). The main reason behind this design choice was that iPhone stores HEIC files this way (limitation of H265 HW? Lower memory requirements? Who knows exactly...) and AVIF is just HEIC with different codec. So the behavior remained the same in AVIF. Additionally AV1 has maximum resolition limit and after that usage of "Grid" images is unavoidable, right?

Also with "Grid" images some thing are easier. For example when using 420 subsampling you cannot encode image with odd resolution (like 1023x759). To do so you have add extra row / col, so subsampling is possible. And you have to introduce additional transformative property clap to crop the extra lines away when the image is loaded. "Grid" images have cropping "built-in".