Open Bug 2091293 Opened 10 days ago Updated 3 hours ago

HEVC playback can’t start decoding at a CRA keyframe: no video for streams that start on a CRA, and every seek fails in Matroska

Categories

(Core :: Audio/Video: Playback, defect, P2)

Firefox 159
Unspecified
macOS
defect

Tracking

()

UNCONFIRMED

People

(Reporter: hello, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

9.27 MB, video/mp4
Details
9.27 MB, application/octet-stream
Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0

Steps to reproduce

Generated synthetic HEVC test files with ffmpeg + libx265, using open GOP (CRA keyframes) vs closed GOP (IDR keyframes):

SRC="-f lavfi -i testsrc2=size=1280x720:rate=50 -f lavfi -i sine=frequency=440:sample_rate=48000 -t 60"
X265=preset=fast:keyint=100:min-keyint=100:scenecut=0:bframes=0:repeat-headers=1
ffmpeg $SRC -c:v libx265 -crf 28 -x265-params "$X265:open-gop=1" -c:a aac cra.mkv
ffmpeg $SRC -c:v libx265 -crf 28 -x265-params "$X265:open-gop=0" -c:a aac idr.mkv
ffmpeg -i cra.mkv -c copy -tag:v hvc1 -movflags +faststart cra.mp4
ffmpeg -i idr.mkv -c copy -tag:v hvc1 -movflags +faststart idr.mp4
ffmpeg -ss 20 -i cra.mkv -t 20 -c copy cra-start.mkv
ffmpeg -i cra-start.mkv -c copy -tag:v hvc1 -movflags +faststart cra-start.mp4

Stream structure (verified with -bsf:v trace_headers):

File Keyframes
cra.mkv 1× IDR_N_LP (frame 0), then 29× CRA_NUT, no RASL/RADL pictures
idr.mkv 30× IDR_N_LP
cra-start.* 10× CRA_NUT, first picture is a CRA

VPS/SPS/PPS are repeated before every keyframe, and Matroska keyframe flags are set on the CRAs.

I opened each file directly in a Firefox tab, played it for a few seconds, then seeked forward.

Environment: Firefox 159, macOS, VideoToolbox hardware HEVC decoding, media.mkv.enabled=true.

Originally found via Plex Web Direct Play of HandBrake 1.6.1 HEVC MKV files, which have the same structure (IDR first, CRA after).

Actual results

File On start After seek
cra.mkv video + audio no video, audio continues
cra.mp4 video + audio video + audio
cra-start.mkv no video, audio no video, audio
cra-start.mp4 no video, audio no video, no audio, jumps to end
idr.mkv video + audio video + audio
idr.mp4 video + audio video + audio

Decoding never starts on a CRA picture. MP4 seeks to CRAs only work if decoding already started on an IDR; it looks like the decoder isn't re-initialised on MP4 seeks. In Matroska every seek fails, which suggests the decoder is re-initialised there and can't restart on a CRA.

Observed on the real-world MKV after a seek:

  • totalVideoFrames stops increasing, droppedVideoFrames stays 0, there is no MediaError, and readyState is 4
  • HTMLMediaElement.buffered immediately jumps to [seekTime, duration], which isn't plausible over the network
  • Loading with #t=<offset> (e.g. resuming playback) shows the first frame, then no video

The same files play and seek correctly in QuickTime Player.

Expected results

Video should decode and play after seeking to, or starting at, a CRA keyframe, in both MKV and MP4.

A CRA picture with no leading pictures is a valid random access point in HEVC (it is treated as NoRaslOutputFlag=1 when it is the first picture), and QuickTime decodes these files correctly.

Attached video cra-start.mp4 —
Attached file cra-start.mkv —
Blocks: matroska
OS: Unspecified → macOS
See Also: → 2025302

This is the file playback side of bug 2065822, which covers the MSE path on macOS. The cause is in the current tree. The attached samples match the report: HEVC with a CRA_NUT at each sync sample, and no IDR, RASL or RADL picture.

Analysis

On macOS, both demuxers mark an HEVC sample as a keyframe only when it has an IDR picture. H265::IsKeyFrame() uses H265NALU::IsIframe(), which accepts only IDR_W_RADL and IDR_N_LP (H265.h). Thus a CRA picture is never a keyframe on macOS.

  • Matroska: MatroskaDemuxer::CheckKeyFrameByExamineByteStream() uses H265::IsKeyFrame() under MOZ_APPLEMEDIA, and H265::IsRandomAccessPoint() on other platforms (MatroskaDemuxer.cpp). The Matroska keyframe flags on the CRAs are not used.
  • MP4: MP4TrackDemuxer::GetNextSample() replaces the stss sync flag with the result of H265::IsKeyFrame() (MP4Demuxer.cpp).

After a seek, WebMTrackDemuxer::GetSamples() skips every sample until it finds a keyframe (WebMDemuxer.cpp). With no IDR after the seek point, it reads to the end of the file, and the video track ends. This explains the cra.mkv seek: no video, audio continues, no MediaError, and totalVideoFrames stops. It probably also explains the buffered range that jumps to the end of the file. cra-start.mkv has no IDR, so no video plays at start or after a seek.

MP4TrackDemuxer::Seek() tries again from the first stss sync sample when it finds no keyframe (MP4Demuxer.cpp). In cra.mp4, that sample is the IDR at 0, so the seek works but decodes from the start. cra-start.mp4 has no IDR. At start, MediaChangeMonitor drops every sample that is not a keyframe, with no error. The seek fails with end of stream, so the audio seek does not start and playback goes to the end.

Regression status

The MP4 part probably comes from bug 2049615 (Firefox 154), which is the regressor of bug 2065822. The Matroska part is not a recent regression. Bug 2065822 comment 9 reports the same MKV failure in Firefox 158 and in a Nightly from May.

Fix direction

The patch D338450 on bug 2065822 (comment 7, by Marcus) keeps the container keyframe flag in MP4Demuxer. It also skips the RASL pictures that VideoToolbox cannot decode, in AppleVTDecoder. It does not change MatroskaDemuxer. Bug 2065822 comment 9 reports that H265::IsRandomAccessPoint() in MatroskaDemuxer on macOS too makes MKV seeks work (0/10 to 10/10). This bug keeps the Matroska part open, and it blocks bug 1422891 (Matroska support).


If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.

Severity: -- → S3
Priority: -- → P2
See Also: → 2065822, 2006226, 2049615
See Also: → 2095073
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: