2048686 - VideoFrame from HTMLVideoElement has a null colorSpace

Status: NEW

(Core :: Audio/Video: Web Codecs, defect, P3)

Description

[C.M.Chang[:chunmin]]

Attachment: standalone-htmlvideo-colorspace.html

A VideoFrame constructed from an HTMLVideoElement reports an empty colorSpace in Firefox: videoFrame.colorSpace.primaries, .transfer, .matrix, and .fullRange are all null. Every other CanvasImageSource accepted by the VideoFrame constructor (canvas, OffscreenCanvas, ImageBitmap, HTMLImageElement, SVGImageElement, another VideoFrame) returns a populated colorSpace, and Chrome returns a populated colorSpace for the <video> source too. This is a minor spec/interop correctness gap with no security relevance; it is a good-first-bug candidate.

Steps to reproduce

Open the attached standalone repro standalone-htmlvideo-colorspace.html over a secure context (file://, https, or localhost) in Firefox. The page paints a canvas, feeds it to an HTMLVideoElement via canvas.captureStream(), waits until the element has a decodable current frame, then runs new VideoFrame(video, { timestamp: 0 }) and reads frame.colorSpace; for contrast it also builds a VideoFrame from the source canvas to isolate the failure to the <video> source.

Observe the on-page result: Firefox reports FAIL — the <video> frame's colorSpace is { primaries: null, transfer: null, matrix: null, fullRange: null } while the canvas frame's is populated. Chrome reports PASS (both populated). The same comparison across every CanvasImageSource is captured by the broader probe attachment #9600616 [details] in bug 2044874.

Expected vs Actual

Expected: a non-null colorSpace. The <video> frame here is BGRA (RGB-backed), so the expected value is specifically the sRGB fallback { "primaries": "bt709", "transfer": "iec61966-2-1", "matrix": "rgb", "fullRange": true } — exactly what Chrome returns for the htmlVideo case and what Firefox already returns for its own canvas/image sources. (YUV-backed sources instead use { "primaries": "bt709", "transfer": "bt709", "matrix": "bt709", "fullRange": false }.)

Actual (Firefox own build, htmlVideo case): { "primaries": null, "transfer": null, "matrix": null, "fullRange": null }.

The full cross-source results captured below (Firefox and Chrome agree on every source except htmlVideo):

Source	Frame format	colorSpace — Firefox	colorSpace — Chrome
RGBA buffer	RGBA	bt709 / iec61966-2-1 / rgb / true	same
I420 buffer	I420	bt709 / bt709 / bt709 / false	same
NV12 buffer	NV12	bt709 / bt709 / bt709 / false	same
HTMLCanvasElement	BGRA	bt709 / iec61966-2-1 / rgb / true	same
OffscreenCanvas	BGRA	bt709 / iec61966-2-1 / rgb / true	same
ImageBitmap	BGRA	bt709 / iec61966-2-1 / rgb / true	same
HTMLImageElement	BGRA	bt709 / iec61966-2-1 / rgb / true	same
SVGImageElement	BGRA	bt709 / iec61966-2-1 / rgb / true	same
VideoFrame (frame-from-frame)	RGBA	bt709 / iec61966-2-1 / rgb / true	same
HTMLVideoElement	BGRA	null / null / null / null	bt709 / iec61966-2-1 / rgb / true

So Firefox's htmlVideo frame is BGRA (RGB-backed) just like its canvas/image sources, yet is the only source returning a null colorSpace — and the only point of divergence from Chrome.

Root cause

The VideoFrame(HTMLVideoElement, ...) constructor builds a VideoFrameData with an empty (default-constructed) VideoColorSpaceInternal as its last argument, instead of deriving or inferring one, and a // TODO: Retrive/infer the duration, and colorspace. comment marks this as a known gap at https://searchfox.org/firefox-main/rev/e869b0f71ca72591c57293364571b97a18f666be/dom/media/webcodecs/VideoFrame.cpp#1650-1655 (the trailing {} on line 1655 is the empty colorSpace).

A default-constructed VideoColorSpaceInternal leaves all four members as Nothing() — see the = default constructor at https://searchfox.org/firefox-main/rev/e869b0f71ca72591c57293364571b97a18f666be/dom/media/webcodecs/WebCodecsUtils.h#222 — and a Nothing() member serializes to JS null, which is exactly the observed output.

By contrast, the path used for canvas/image/bitmap/SVG sources, InitializeFrameWithResourceAndSize, never leaves the colorSpace empty: it picks a fallback based on whether the surface is YUV, calling FallbackColorSpaceForVideoContent() (BT709 limited-range) or FallbackColorSpaceForWebContent() (sRGB / BT709 full-range) at https://searchfox.org/firefox-main/rev/e869b0f71ca72591c57293364571b97a18f666be/dom/media/webcodecs/VideoFrame.cpp#1177-1182, whose definitions are at https://searchfox.org/firefox-main/rev/e869b0f71ca72591c57293364571b97a18f666be/dom/media/webcodecs/WebCodecsUtils.cpp#560 and https://searchfox.org/firefox-main/rev/e869b0f71ca72591c57293364571b97a18f666be/dom/media/webcodecs/WebCodecsUtils.cpp#570. The <video> constructor instead routes through InitializeFrameFromOtherFrame, which faithfully copies whatever colorSpace it is handed in VideoFrameData (https://searchfox.org/firefox-main/rev/e869b0f71ca72591c57293364571b97a18f666be/dom/media/webcodecs/VideoFrame.cpp#1191) — so the empty colorSpace passed by the caller is propagated unchanged to the resulting frame.

A correct fix should read the colorSpace from the decoded image rather than guess it from the surface format. A decoded <video> frame carries its real color metadata, and Firefox already has the code to read it.

• The decoded image carries color metadata. PlanarYCbCrData holds mYUVColorSpace / mColorPrimaries / mTransferFunction / mColorRange / mColorDepth at gfx/layers/ImageContainer.h#770-776, populated by the software decoders (ffmpeg, dav1d, AOM, VPX, AppleVT). GPU-backed images (GPUVideoImage, D3D11ShareHandleImage, MacIOSurface) expose the same metadata through accessors, and it is carried into the content process with the frame via RemoteImageHolder / the SurfaceDescriptor, so reading it needs no GPU-process pixel access.
• A ready-made reader exists: GuessColorSpace(layers::Image*) type-dispatches across PlanarYCbCrImage / NVImage / GPUVideoImage / MacIOSurfaceImage / DMABUFSurfaceImage, reads each one's color fields, and converts the gfx color enums into a VideoColorSpaceInternal via the To* helpers in WebCodecsUtils. The WebCodecs decode path uses it at VideoDecoder.cpp#929. The <video> constructor should derive its colorSpace from image through this reader instead of passing an empty {}, then complete any components the reader leaves unspecified from the format fallback (see Phase 1 below).

FallbackColorSpaceForVideoContent() / FallbackColorSpaceForWebContent() are faithful to the WebCodecs pick color space spec defaults (step 2: sRGB for RGB formats; step 3: BT709 limited-range for YUV), and they are the right answer when a source genuinely carries no color information — a canvas, image, or SVG surface. They are the wrong answer for a <video> frame, which does carry that information: the BT709-limited guess mislabels BT.601 SD content, drops HDR (BT2020 primaries / PQ-HLG transfer / 10-bit depth), and misreports full-range YUV. The format-based heuristic in InitializeFrameWithResourceAndSize also cannot see a video frame's true format — it inspects image->GetAsSourceSurface()->GetFormat(), which has already flattened the YUV/NV12/GPU image to a surface — and the <video> constructor does not take that path regardless.

The fallback is therefore best used only to fill in what the image leaves unspecified, not as the primary source. Two implementation caveats: (1) GuessColorSpace can still leave individual components Nothing() (→ null) when the decoder reports an unspecified value (e.g. ToPrimaries(gfx::ColorSpace2::UNKNOWN)), and the existing decode path uses its result raw — so the robust approach is GuessColorSpace(image) with the spec fallback filling only the components that remain unspecified. (2) The hardware/remote-decode path initializes color metadata to hardcoded defaults (YUVColorSpace::Default, ColorSpace2::UNKNOWN, TransferFunction::BT709, ColorRange::LIMITED) at dom/media/ipc/RemoteVideoDecoder.cpp#259 and only the software path extracts the real values at #295, so for some hardware-decoded frames even the real-metadata route is currently incomplete — a separate, deeper gap.

The fix for this bug is Phase 1 of a broader cleanup, noted in videoframe-colorspace-unification-design.md (attachment #9598872 [details]): make the <video> constructor read the image metadata and complete any still-unspecified components from the format fallback, add the same no-null completion on the decode path, and relocate the shared metadata reader into WebCodecsUtils so VideoFrame and VideoDecoder use one copy (all strictly additive, so the other sources' values do not change). Done this way the fix is forward-compatible with the full design rather than throwaway: the shared reader becomes the metadata layer of the unified ResolveColorSpace resolver and the no-null completion becomes its format-default layer, so the later phases generalize this code instead of rewriting it. The full cross-source unification — the single layered resolver, a uniform Image::GetColorData() accessor, the regression guardrails, and the deferred hardware-metadata fix — is written up in videoframe-colorspace-unification-design.md.

Comment 1

[C.M.Chang[:chunmin]]

Attachment: videoframe-colorspace-unification-design.md