2056028 - WebCodecs VideoEncoder emits a hardcoded decoderConfig.colorSpace and signals no color into the AV1 bitstream

Status: NEW

(Core :: Audio/Video: Web Codecs, defect)

Description

[C.M.Chang[:chunmin]]

Summary

Firefox's WebCodecs VideoEncoder produces color metadata that is disconnected from the content it encodes, via two related defects:

1. The emitted EncodedVideoChunkMetadata.decoderConfig.colorSpace is a hardcoded constant — always { primaries: "bt709", transfer: "bt709", matrix: "bt709", fullRange: false } — regardless of the input VideoFrame's color space, the VideoEncoderConfig, or the actual encoded bitstream. This is codec-agnostic (it is in the shared WebCodecs encoder code, not a per-codec path), and is confirmed empirically for AV1, VP9, VP8, and H.264 (both avc and annexb): all emit the identical constant for the same non-BT.709 input — see the reproduction below.

2. For AV1 (prefer-software, ffvpx/libaom) the encoder writes no color description into the bitstream. The color-setup code is disabled, so color_primaries / matrix_coefficients / transfer_characteristics are left UNSPECIFIED (CICP 2) and the range signals limited — even though the pixels were converted to BT.601 limited range by the RGB→YUV path.

The net effect is that an application encoding a frame with a known color space (e.g. BT.2020/PQ HDR, or BT.601 SD content) gets back a decoderConfig claiming BT.709 and an AV1 bitstream that carries no color information. Any consumer that trusts decoderConfig.colorSpace, or that recovers color from the bitstream, ends up with the wrong color space. Chromium does not have this problem: its emitted decoderConfig.colorSpace and its AV1 bitstream both consistently signal the real color space.

All source links below are pinned to revision c0dff85fa67e.

Reproduction

Defect 1 — hardcoded decoderConfig.colorSpace (assertion-based; codec-agnostic)

This testharness test encodes a frame tagged with a known non-BT.709 color space and asserts, component by component, that the emitted decoderConfig.colorSpace reflects it. It FAILs today (the encoder returns the BT.709/limited constant) and should PASS once the encoder derives the emitted config from the content. Run with ./mach wpt <path> --headless; it is also runnable as a standalone HTTPS/localhost page (WebCodecs needs a secure context) by wrapping the body in async and replacing the asserts with logging.

// META: global=window,dedicatedworker
const inColor = { primaries: "bt2020", transfer: "pq", matrix: "bt2020-ncl", fullRange: true };
promise_test(async t => {
  const w = 320, h = 200, y = w * h, ch = (w >> 1) * (h >> 1);
  const buf = new Uint8Array(y + 2 * ch); buf.fill(128); buf.subarray(0, y).fill(120);
  const cfg = { codec: "av01.0.04M.08", width: w, height: h, bitrate: 1000000, framerate: 30, hardwareAcceleration: "prefer-software" };
  assert_implements_optional((await VideoEncoder.isConfigSupported(cfg)).supported, "AV1 software encode");
  const frame = new VideoFrame(buf, { format: "I420", codedWidth: w, codedHeight: h, timestamp: 0, colorSpace: inColor });
  let emitted;
  const enc = new VideoEncoder({ output: (chunk, md) => { if (md.decoderConfig && !emitted) emitted = md.decoderConfig.colorSpace; }, error: t.unreached_func("encoder error") });
  t.add_cleanup(() => { try { enc.close(); } catch (e) {} });
  enc.configure(cfg); enc.encode(frame, { keyFrame: true }); frame.close();
  await enc.flush();
  assert_not_equals(emitted, undefined, "decoderConfig emitted");
  // The emitted config must reflect the (already-YUV I420) input, not a fixed constant.
  assert_equals(emitted.primaries, inColor.primaries, "primaries");
  assert_equals(emitted.transfer, inColor.transfer, "transfer");
  assert_equals(emitted.matrix, inColor.matrix, "matrix");
  assert_equals(emitted.fullRange, inColor.fullRange, "fullRange");
}, "VideoEncoder decoderConfig.colorSpace reflects the input frame colorSpace");

Current result (Firefox, this build, confirmed via ./mach wpt): FAIL — emitted decoderConfig.colorSpace = { primaries: "bt709", transfer: "bt709", matrix: "bt709", fullRange: false } for the bt2020/pq/full input. This is codec-agnostic: swapping only the codec (and avc format) reproduces the identical constant for every codec tested (each row is a real observed run on this build):

codec	input colorSpace	emitted decoderConfig.colorSpace
av01.0.04M.08 (AV1)	bt2020 / pq / bt2020-ncl / full	bt709 / bt709 / bt709 / limited
vp09.00.10.08 (VP9)	(same)	bt709 / bt709 / bt709 / limited
vp8 (VP8)	(same)	bt709 / bt709 / bt709 / limited
avc1.42001E + avc (H.264)	(same)	bt709 / bt709 / bt709 / limited
avc1.42001E + annexb (H.264)	(same)	bt709 / bt709 / bt709 / limited

Expected after fix: the emitted decoderConfig.colorSpace reflects the encoded content; for an already-YUV I420 frame carrying an explicit color space, that is the input's color space. The exact value is a fix-design choice (an encoder is spec-permitted to re-label when a color conversion is unavoidable), so a component-wise assertion is preferred over exact JSON equality — but the current behavior (one identical BT.709/limited constant for every input and every codec) is unambiguously wrong.

Defect 2 — AV1 bitstream carries no color (covered by full-cycle-test)

The Defect-1 test above only checks the emitted config; it does not prove the bitstream carries correct color. The existing WPT full-cycle-test.https.any.html?av1 is the gate for that: its stripDecoderConfigColorSpace subtest blanks the decoder config and forces color to be recovered from the bitstream, then asserts it matches the encoder's declaration. On a build whose decoder surfaces the real bitstream color (instead of guessing BT.709), it fails with colorSpace.primaries expected "bt709" but got "bt470bg" — the UNSPECIFIED bitstream (Defect 2) decoded to a BT.601-family default that disagrees with the hardcoded-BT.709 config (Defect 1). Chromium passes this test (it signals a real, consistent color space on both sides). No separate standalone repro is provided for Defect 2 — full-cycle-test?av1 covers it.

Root cause (code trace)

Defect 1 — decoderConfig.colorSpace is a hardcoded literal

When the encoder emits a new decoder config (gated by mOutputNewDecoderConfig — i.e. the first chunk and on any config change), EncoderTemplate builds it by calling EncoderConfigToDecoderConfig at EncoderTemplate.cpp#363-366. VideoEncoder::EncoderConfigToDecoderConfig then sets the color space to BT.709/BT.709/BT.709 + fullRange=false as a literal, without consulting the input frame, the active config, or the encoder output — VideoEncoder.cpp#604-610. The only color-related field ever attached to the platform EncoderConfig is a range (limited, and only for Realtime usage) at VideoEncoder.cpp#279-284; no primaries/matrix/transfer are ever derived. So the BT.709 is a constant, not derived from anything.

Defect 2 — the AV1 encoder signals no color into the bitstream

Prefer-software AV1 encodes through ffvpx libaom-av1 via FFmpegVideoEncoder. That wrapper sets pix_fmt = YUV420P but leaves the codec context's color fields unset — the block that would assign color_primaries/colorspace/color_trc is commented out with the note "Setting this like that crashes encoders" — FFmpegVideoEncoder.cpp#401-416. The per-frame AVFrame sets only format/size/pict_type — none of color_primaries/colorspace/color_trc/color_range are ever set on it — FFmpegVideoEncoder.cpp#651-655. The RGB→YUV conversion of the input is a fixed BT.601 limited-range matrix via libyuv ARGBToI420/ABGRToI420 — ImageConversion.cpp#187-205 — so the pixels are BT.601 while no metadata is written. ffvpx forwards the (unset) AVCodecContext color fields to libaom at libaomenc.c#991-999; their defaults are UNSPECIFIED (2) per options_table.h#274, and AVCOL_RANGE_UNSPECIFIED maps to studio/limited at libaomenc.c#574-587. libaom copies these straight into the sequence-header color config — av1_cx_iface.c#1427-1429. So the bitstream color description is UNSPECIFIED.

The disconnect

These are two independent code paths that never consult each other: the emitted metadata is a BT.709 constant, and the encoder is handed UNSPECIFIED because the color-setup is disabled — and neither reflects the true BT.601 pixels. Nothing copies the frame/config color into the encoder, nor the encoder's real color config back into decoderConfig.

How it manifests end-to-end (and why it is usually hidden)

On decode, the color space is normally guessed, which masks the inconsistency. For the WebCodecs prefer-software AV1 decode path (ffvpx FFmpegVideoDecoder), an UNSPECIFIED stream leaves the decoded image's mColorPrimaries unset (UNKNOWN) and its matrix defaulted to DefaultColorSpace(size) = BT.601 for sub-720 heights — FFmpegVideoDecoder.cpp#1151-1165, VideoUtils.h#546-549. VideoDecoder::GuessColorSpace then derives primaries by guessing from the matrix (VideoDecoder.cpp#459-488): BT.601 → Smpte170m → bt470bg.

Today full-cycle-test.https.any.html?av1 passes because a companion decoder path guesses BT.709 (matching the hardcoded config) in the case it exercises. When a decoder change surfaces the real (BT.601-derived) color on the GPU-texture delivery path, that test fails its stripDecoderConfigColorSpace subtest with colorSpace.primaries expected "bt709" but got "bt470bg" — i.e. the hardcoded-BT.709 config disagrees with the content. That failure is a symptom of this encoder bug (config and bitstream both wrong/disconnected), not a decoder bug: the bt470bg is Firefox's reasonable BT.601-for-SD default, and the hardcoded BT.709 config is the incorrect side.

Cross-browser

Running the equivalent encode→(strip config)→decode in Chromium (WebCodecs, av01.0.04M.08, sRGB canvas source), both the emitted decoderConfig.colorSpace and the value recovered from the bitstream are smpte170m / smpte170m / smpte170m / false — self-consistent. Chromium derives and signals a real color space on both sides; Firefox emits a BT.709 constant and signals UNSPECIFIED in the bitstream.

Impact

• decoderConfig.colorSpace from Firefox's WebCodecs VideoEncoder is meaningless (always BT.709/limited); applications relying on it to configure a decoder or to persist color metadata get wrong values, for all codecs.
• AV1 bitstreams produced by Firefox WebCodecs carry no color description, so downstream tools/players must guess; HDR (BT.2020/PQ-HLG) and BT.601 SD content are silently mislabeled.
• Encode→decode round-trips do not preserve color-space fidelity.

Suggested investigation / fix direction (not verified)

• Derive EncoderConfigToDecoderConfig's color space from the input frame's color space and/or the real encode conversion, instead of a constant.
• Implement AV1 (and general) encoder color signaling — the ffvpx color-setup block is disabled with a "crashes encoders" note, so this needs a correct approach (likely setting AVCodecContext/AVFrame color fields, or the aom AV1E_SET_* controls, in a way that does not crash).
• Ensure the RGB→YUV conversion color space (currently a fixed BT.601 in libyuv) is consistent with the signaled metadata, or make the conversion matrix follow the intended color space.

Related bugs and references

• Bug 2055639 — WebCodecs VideoDecoder drops the H.264 full-range flag (decoder-side color-range propagation). Sibling of this bug; its decoder-side fix (propagating real color metadata) is what surfaced this encoder inconsistency. This encoder bug is separate.
• Bug 1869825 — GPU/texture-path color metadata is incomplete for RDD software decode; referenced inline at VideoDecoder.cpp#467-470.
• Bug 2048686 — VideoFrame colorSpace unification / null colorSpace from HTMLVideoElement; the broader color-metadata cleanup this fits into.
• Bug 1749047 — the 2023 landing that first plumbed decoded-frame color metadata over IPC.
• WPT — full-cycle-test.https.any.js (self-consistency check that Chromium passes and that exposes this once the decoder stops masking it).
• Spec — WebCodecs VideoDecoderConfig.colorSpace / VideoColorSpace.