2025302 - HEVC/H.265 in Matroska silently fails when CodecPrivate contains malformed HEVCDecoderConfigurationRecord (hvcC)

Status: NEW

(Core :: Audio/Video, defect, P2)

Description

[Charles]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:148.0) Gecko/20100101 Firefox/148.0

Steps to reproduce:

Saved https://www.libde265.org/hevc-bitstreams/tos-1720x720-cfg01.mkv locally and tried to play it back using Firefox on Ubuntu 26.04 (pre-release)

Actual results:

Firefox fails to play HEVC video in Matroska (.mkv) containers when the CodecPrivate element contains a malformed HEVCDecoderConfigurationRecord. Playback appears to start but immediately ends, the video element transitions to the ended state within milliseconds. No error is reported to the user or to the web console.

Chromium-based browsers play the same file without issue by falling back to inline parameter sets (VPS/SPS/PPS embedded in IDR frames).

tos-1720x720-cfg01.mkv (Tears of Steel HEVC test encode, encoded 2014-03-18 with libebml v1.3.0 + libmatroska v1.4.1)

Video element immediately transitions to ended state. With MOZ_LOG="MediaDecoder:4" the state machine shows:

state=DECODING_FIRSTFRAME change state to: DECODING
state=DECODING change state to: COMPLETED ← immediate
VideoSink [Start]
OnMediaSinkVideoComplete ← no frames rendered
PlaybackEnded

The file's CodecPrivate is not a valid HEVCDecoderConfigurationRecord. It contains what appears to be a NAL unit of type 0 (invalid in extradata) with no Annex B start codes, maybe a bug in the 2014 encoder. This can be confirmed:

ffprobe -v trace tos-1720x720-cfg01.mkv
...

→ [extract_extradata] Failed to parse header of NALU (type 0): "Invalid data found"

→ [extract_extradata] No start code is found.

...

I think Firefox tries to read the hvcC to determine the NAL unit length size, then uses it to convert each packet from length-prefixed format to Annex B before passing it to the decoder. With a malformed hvcC, this conversion produces garbage, the decoder signals EOS after the first attempt.

The HEVC stream itself is fully decodable: ffmpeg -v error -i tos-1720x720-cfg01.mkv -f null - produces no errors, because libavcodec falls back to reading inline VPS/SPS/PPS from IDR frames. Remuxing through MPEG-TS (which uses Annex B and regenerates hvcC from the inline parameter sets) produces a file that Firefox plays correctly.

Expected results:

Video plays for its full duration (12:14)

When hvcC parsing fails (or produces an obviously invalid result such as NAL type 0), Firefox should fall back to extracting codec parameters from inline parameter sets in the first IDR frame, as libavcodec and Chromium do. The fallback path already exists in the underlying libavcodec, it seems like Firefox's preprocessing layer that prevents it from being reached

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Jim Mathies [:jimm]]

Attachment: bug-2025302-triage.md

I went with P2, as we are working on improving hevc support currently.

Comment 3

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 2025302 - Part 1: Add mochitest for MKV HEVC with malformed CodecPrivate.

Tests that an MKV file with configurationVersion=0xFF in its HEVC CodecPrivate
plays to completion instead of silently failing at demux time.

Comment 4

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 2025302 - Part 2: Extract real hvcC from first IDR when CodecPrivate is malformed in MKV.

When a Matroska file contains HEVC video whose CodecPrivate element is not a
valid HEVCDecoderConfigurationRecord, use fake extradata as a bootstrap for NAL
length parsing, then scan the first IDR packet's inline VPS/SPS/PPS to build a
real hvcC and back-patch it into the track info and all queued samples before
they reach the decoder.