2060629 - GMP/OpenH264 H.264 encoder omits input color information from SPS VUI

Status: NEW

(Core :: Audio/Video, defect, P3)

Description

[C.M.Chang[:chunmin]]

Problem

When Firefox encodes an explicitly color-tagged I420 frame as H.264 using the software GMP/OpenH264 encoder, the resulting SPS does not signal the input color information in its VUI.

The encoded stream consequently omits:

• Video full-range indication
• Color primaries
• Transfer characteristics
• Matrix coefficients

Consumers of the standalone H.264 stream must guess these properties, potentially interpreting the encoded pixels incorrectly.

Steps to reproduce

1. Create an I420 frame carrying BT.709 primaries, BT.709 transfer characteristics, BT.709 matrix coefficients, and full range.
2. Encode the frame using the software GMP H.264 encoder.
3. Extract the SPS from the first key frame.
4. Parse the SPS VUI.

A standalone regression test named MediaDataEncoderTest.GMPH264SignalsInputColor is available for this bug.

Test dashboard:

https://tests.firefox.dev/test.html?test=dom/media/gtest/TestMediaDataEncoder.cpp

Actual result

The SPS parses successfully, but video_signal_type_present_flag is false. Therefore video_full_range_flag, colour_primaries, transfer_characteristics, and matrix_coefficients are not signaled.

Expected result

For the described input, the SPS should contain:

• video_signal_type_present_flag = 1
• video_full_range_flag = 1
• colour_description_present_flag = 1
• colour_primaries = 1 (BT.709)
• transfer_characteristics = 1 (BT.709)
• matrix_coefficients = 1 (BT.709)

The bitstream should remain self-describing when it is stored, muxed, transmitted, or decoded without the originating WebCodecs configuration.

Upstream source evidence

At Firefox revision 7f451c7918ffe36840b0ef31dfc2272ded671f81:

• GMPVideoEncoder::InitComplete initializes GMPVideoCodec version 36 and supplies an empty initialization codec-specific payload.
• GMPVideoEncoder::Encode converts the input to I420 and submits an empty per-frame codec-specific payload.
• GMPVideoCodec contains dimensions, rate-control, profile, level, and temporal-layer fields, but no color description.
• GMPCodecSpecificInfoH264 contains only mSimulcastIdx.

At Cisco OpenH264 revision 4a2615fac570c6ca1ed4f157b9fdab9466edfd80:

• The GMP adapter initialization maps codec settings into SEncParamExt but does not populate VUI color settings.
• The adapter accepts but does not consume the per-frame codec-specific input.
• SSpatialLayerConfig provides bVideoSignalTypePresent, bFullRange, bColorDescriptionPresent, uiColorPrimaries, uiTransferCharacteristics, and uiColorMatrix.
• The SPS VUI writer emits these fields when enabled, and SPS initialization copies them from the spatial-layer configuration.

Thus, OpenH264 can represent the required H.264 VUI information, but the Firefox-to-GMP-to-OpenH264 path does not provide it.

Impact

Missing signaling can cause full-range video to be treated as limited range and can lead to incorrect matrix, primaries, or transfer-function interpretation.

Related: bug 2055639.

Comment 1

[C.M.Chang[:chunmin]]

Attachment: gmp-h264-color-signaling-test.patch

Comment 2

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[media-triaging-bot]

Confirmed the Gecko-side gap.

Analysis

The colour information is available in Gecko but has no way across the GMP boundary. EncoderConfig::SampleFormat carries a VideoColorSpace, but GMPVideoEncoder::InitComplete populates a GMPVideoCodec that has no colour-description fields at all -- the newest additions at kGMPVersion36 are temporal-layer options -- and passes an empty codec-specific init payload. With nothing signalled, the Cisco adapter leaves bVideoSignalTypePresent / bColorDescriptionPresent false, so OpenH264's SPS writer skips the VUI colour syntax (ITU-T H.264 Annex E). Decoders then infer colour_primaries / transfer_characteristics / matrix_coefficients as 2 (Unspecified) and video_full_range_flag as 0, which is where visibly wrong colour comes from: BT.709 SD content picks up a BT.601 matrix guess, and full-range content is rendered as limited.

Two notes on scope:

• The GMP encoder is only registered as a platform encoder when media.gmp.encoder.enabled is true (PEMFactory.cpp), and that pref defaults to false, so a default profile does not reach this path via WebCodecs VideoEncoder or MediaRecorder today. It does become blocking for turning that pref on.
• The WebRTC GMP encoder (WebrtcGmpVideoCodec) goes through the same colour-less GMP API, so it is likely affected by the same gap -- worth confirming, and possibly splitting into its own bug.

Suggested direction

Add colour-description fields to GMPVideoCodec behind a new GMP API version -- bVideoSignalTypePresent, bFullRange, bColorDescriptionPresent, plus CICP primaries / transfer / matrix -- mirroring SSpatialLayerConfig so the adapter mapping stays mechanical, then populate them in GMPVideoEncoder::InitComplete from the config's VideoColorSpace, version-gated on mGMPApiVersion so older plugin builds keep working. Smuggling the values through the opaque codecSpecific payload instead would still need the same upstream adapter change, so it buys nothing.

This cannot land Gecko-only. Sequencing: tag the new header in mozilla/gmp-api, land the module/gmp-openh264.cpp change upstream at Cisco, bump the revision in taskcluster/kinds/fetch/openh264.yml, then land the Gecko patch and enable the attached gtest.

One thing worth resolving first: bug 2056028 reports the shared WebCodecs encoder layer emitting a hardcoded BT.709 decoderConfig.colorSpace. If the input VideoColorSpace also arrives unpopulated at the platform encoder, fixing this bug on its own would write a constant into the VUI rather than the real colour space -- so the producer side should be confirmed before the consumer side.