2072679 - Support progressive rendering of AVIF (a1lx)

Status: ASSIGNED

(Core :: Graphics: ImageLib, enhancement)

Description

[Jake Archibald [:jakea]]

https://aomediacodec.github.io/av1-avif/#layered-image-indexing-property

Comment 1

[Jake Archibald [:jakea]]

Attachment: Bug 2072679 - Support progressive rendering of AVIF images.

Here's an AI summary, which I've read & edited, but I'm fuzzy on some details:

A layered AVIF already contains up to four successively better renderings of
the same picture, and Firefox already decodes such files correctly -- it just
waits for the last byte before showing anything. This makes the decoder render
those layers as they arrive.

This is a latency feature, not a correctness fix: the finished image is
byte-identical either way, and only what is on screen mid-download changes.
Gated on image.avif.progressive.enabled, which defaults on in Nightly only.

Bug 1712813 asks for both a1lx and a1op. This implements a1lx, which is
what progressive decoding needs; a1op (OperatingPointSelectorProperty) stays
unsupported and still forbids the item under non-permissive strictness.

A progressive AVIF is a layered image item: one item whose payload holds up to
four AV1 frames distinguished by their spatial_id. Three item properties
describe it:

• a1lx (AV1LayeredImageIndexingProperty) gives the byte sizes of the first
  three layers within the payload; the last layer is the remainder. It is the
  only signal saying where one layer ends and the next begins, and it is
  non-essential, because a decoder that ignores it still renders the final
  image.
• lsel (LayerSelectorProperty) means "render exactly layer N". An item is
  progressively renderable only if it has no lsel, or an lsel whose
  layer_id is 0xFFFF ("no selection").
• a1op (OperatingPointSelectorProperty) picks an AV1 operating point. Still
  unsupported, and out of scope here.

An alpha auxiliary item can be layered too, with its own a1lx, in which case
colour layer N has to be paired with alpha layer N.

See https://aomediacodec.github.io/av1-avif/#layered-image-item.

a1lx used to be recorded as an unsupported-but-non-fatal feature and skipped
without reading its payload. It is now parsed (read_a1lx), and
ItemProperty::LayeredImageIndexing carries the layer sizes.
Feature::A1lx becomes supported().

AvifItem also retains its iloc extents and construction method, exposed
through primary_item_extents / alpha_item_extents and the
*_is_file_construction predicates, plus a new ItemExtent repr(C) type. The
decoder needs to know where an item's bytes live so it can slice the payload
into layers itself, rather than relying on mp4parse's own item-data copy -- which
for an item spread across several extents would have captured the whole thing.

A malformed a1lx (truncated, overlong, or non-zero reserved bits) is
deliberately not fatal: read_ipco records the property as present with no
layer sizes, which callers read as "not a layered image". Before this patch such
a file decoded fine, and failing it would be a regression.

nsAVIFDecoder::DoDecodeInternal's loop used to be while (!mReadCursor), and
mReadCursor was only set on SourceBufferIterator::COMPLETE. It now tries to
make progress on every chunk, via TryDecode.

• Parsing a partial file. mp4parse fails the parse unless every iloc
  extent is present, so TryDecodeInternal zero-pads mBufferedData out to the
  length of the whole file before parsing. That length comes from
  PeekTotalLength, a walk of the top-level boxes only -- no item, property or
  ipma logic -- which returns the end of the mdat. Capped at
  kMaxProgressivePrealloc; larger files are simply not decoded progressively.
  mp4parse never inspects item data, so the zeros are invisible to it.

  Growing mBufferedData reallocates, and mp4parse holds pointers into it (see
  AVIFDecoderStream::GetContiguousAccess), so WriteBufferedData drops the
  parser whenever it has to grow. mReceivedLength tracks how much of the
  buffer is real.

  This is worth doing for metadata decodes too: nothing can be rendered until
  the metadata decode has posted the image size. An early parse that turns out
  not to buy anything -- not layered, or an image with no ispe, whose size can
  only come from the bitstream -- is thrown away by WaitForWholeSource, which
  also hands back the padding so a file that overstates its mdat size does not
  leave us sitting on the declared length for the length of the download.

• Choosing the path. SetupProgressiveDecode decides once, after the parse
  succeeds. BuildLayers cuts the item's extents into per-layer file ranges --
  an extent boundary and a layer boundary need not coincide -- and records for
  each layer the highest file offset that must have arrived for it, and every
  layer before it, to be decodable.

• Feeding the decoder. dav1d is opened with all_layers = 1 for the layered
  path, because decoding layer by layer needs each layer's picture rather than
  just the top one; the filtering moves to the caller. The AOM path stays
  single-shot. HighestAvailableLayer and ReadLayer then feed layers in
  order, since layers generally use inter-layer prediction, and only the last
  one decoded is rendered.

• The surface. The surface pipe is retained across layers and rewound with
  ResetToFirstRow, so each layer refines the same surface instead of
  allocating a second frame. Creation is deferred until there are pixels to
  write, so a progressive image never exposes an empty surface. Animated images
  keep getting a fresh pipe per frame, since each carries its own
  AnimationParams. The RGB conversion buffer is retained for the same reason.

• Spatial scalability. A layer may be coded smaller than the image and
  scaled up to it, signalled through the frame header's render size. Such a
  layer is converted at its own size into a scratch buffer and upscaled with
  libyuv::ARGBScale. Filtering has to happen on premultiplied data: on
  non-premultiplied data it weights every sample's colour equally however
  transparent it is, so the colour hiding under fully transparent pixels haloes
  the opaque side of an alpha edge. The scale is bracketed by ARGBAttenuate /
  ARGBUnattenuate when the conversion did not already produce premultiplied
  data.

• Falling back. The per-layer path has failure modes the single-shot path
  does not -- most obviously a layer whose alpha plane does not match its colour
  plane, fatal here (bug 1682318) but impossible once the layers are
  concatenated. Since the layers are only ever an early preview, any failure
  while decoding them abandons the layered path and retries the whole image in
  one go. The pipe is deliberately not reset, so no second frame is
  allocated.

  Note that this makes a broken layered path invisible to tests that only check
  the final pixels, which is why the RendersEveryLayer assertions matter.

• Truncated downloads. If the source ends -- a stalled transfer, a truncated
  file -- after a layer has reached the surface, that layer is kept and the image
  is called done, rather than replacing a good preview with alt text. Every
  layer is a whole frame, so there is nothing half-drawn to hide. This is what
  the JXL decoder does with its partial frame.

  Jake: I've tested this, and it's correct, but an errored stream will still
  cause a fallback to an error state, so it's consistent with how we handle
  other progressive image formats. Tested with:
  https://random-stuff.jakearchibald.com/apps/partial-img-decode/

• avif.progressive (labeled counter, absent / active / unusable):
  whether a still AVIF was layered and, if so, whether we rendered it layer by
  layer. Only recorded while the image is still downloading, since one that has
  fully arrived is decoded in one shot regardless.

• avif.progressive_layers_rendered: how many layers reached the surface,
  including the final one.

• avif.a1lx now reads the parsed property rather than the
  unsupported-features bitfield. Feature::A1lx becoming supported clears that
  bit permanently, which would otherwise have made this existing never-expiring
  metric report absent for every AVIF forever.

Four gtest fixtures, all encoded --lossless so identity matrix coefficients
make the colours survive the YUV round trip exactly and the tests need no fuzz.
Generation commands are in progressive-avif-plan.md.

fixture	covers
progressive.avif	3 layers, blue then red then green at one resolution; layer 0 is a keyframe and the rest inter-predicted, so it exercises feeding layers in order
progressive-alpha.avif	layered alpha aux item; pairing colour layer N with alpha layer N
progressive-scaled.avif	spatial scalability: 25x25 layer 0, 100x100 layer 1
progressive-scaled-alpha-edge.avif	scaled layer 0 with a hard alpha edge, opaque green on one side and fully transparent red on the other

Notable assertions beyond the usual single-chunk / multi-chunk / incremental
matrix:

• AVIFProgressiveRendersEveryLayer: fed a byte at a time all three layers
  reach the surface; fed in one chunk none of them do, because there is nothing
  to gain from decoding layer by layer once all the data is in hand.
• AVIFProgressiveScaledRendersEveryLayer and its alpha counterpart: the 25x25
  layer 0 really was scaled up rather than rejected as a size mismatch.
• AVIFProgressiveScaledAlphaLayerHasNoColorFringe: decodes only the scaled
  layer 0 and bounds the red channel over pixels that are at least half opaque,
  the ones a halo would be visible on. Premultiplying before the upscale leaves
  at most 7 there; filtering the non-premultiplied data straight out of the
  conversion leaves 58.
• AVIFProgressiveTruncatedKeepsLastLayer and
  AVIFProgressiveTruncatedBeforeFirstLayerFails: a stalled download keeps
  whichever layer arrived, but an image with nothing decodable at all is still
  a failure.
• AVIFProgressiveDownscaleDuringDecode: progressive decoding does not change
  what a downscale-during-decode produces.

image.avif.progressive.enabled is forced on in AutoInitializeImageLib
alongside the JXL pref, so the gtests do not depend on the channel default.

test_avif_progressive.html covers the pixels actually reaching the screen:
sendprogressiveavif.sjs streams the file one layer at a time and the test
checks the image goes blue, then red, then green, in that order, and that a
truncated response keeps its first layer. Skipped on http3, where the response
is not released incrementally so no intermediate layer ever shows.

Local results: ./mach gtest 'Image*' 517/517, ./mach mochitest image/test/mochitest 550/550, the latter also verified under
--use-http2-server.

• The zero-padding. PeekTotalLength only returns a length once every box
  before the mdat has arrived, so the metadata really is present by then, but a
  file can still declare an mdat far larger than what it sends. The padding is
  released as soon as we decide the image is not worth parsing early, and
  kMaxProgressivePrealloc bounds it, but the value of that cap is a judgement
  call worth a second opinion -- progressive AVIFs that benefit here are small,
  so a lower cap would cost little.
• Buffer and parser lifetime. mp4parse reads item data in place, so
  mBufferedData must not reallocate under a live parser. Everything that
  resizes it drops mParser first; that invariant is the one to check.
  Relatedly, Vector::shrinkStorageToFit is not usable on this buffer: it
  leaves a capacity that is not one of the sizes Vector's growth computes, and
  the next single-byte append then asserts.
• The retained pipe, and the deliberate decision not to reset it when
  falling back to a single-shot decode.
• mp4parse leniency for a malformed a1lx, which trades strict validation
  for not regressing files that render today.

1. Upstream the mp4parse change. third_party/rust/mp4parse{,_capi} is
   vendored from github.com/mozilla/mp4parse-rust and was edited in place, with
   .cargo-checksum.json refreshed, so the C++ could be written and tested. It
   has to land upstream and be re-vendored with ./mach vendor rust first. The
   change is additive: the a1lx payload and read_a1lx, Feature::A1lx
   becoming supported, ItemExtent, and the new Mp4parseAvifInfo fields.
2. Get a data review for the two new metrics. They now cite bug 1712813,
   but data_reviews: points at the bug rather than at a review, because there
   is not one yet. It has to be updated to the #cN anchor of the
   data-review comment before this ships.
3. Decide the shape of avif.progressive_layers_rendered. It is a
   custom_distribution for a value that is always 1 to 4; a labeled counter
   would be cheaper to collect and easier to read.

Not in scope: row-wise incremental decode, the orthogonal mechanism Chromium
uses for non-progressive stills (decoder_->allowIncremental), which renders a
single AV1 frame top-down as its tiles arrive. That needs grid and tile support
and is separate work.

Comment 2

[Jake Archibald [:jakea]]

Attachment: WIP: Bug 2072679 - Support progressive rendering of AVIF images.