HEVC playback can’t start decoding at a CRA keyframe: no video for streams that start on a CRA, and every seek fails in Matroska
Categories
(Core :: Audio/Video: Playback, defect, P2)
Tracking
()
People
(Reporter: hello, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(2 files)
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0
Steps to reproduce
Generated synthetic HEVC test files with ffmpeg + libx265, using open GOP (CRA keyframes) vs closed GOP (IDR keyframes):
SRC="-f lavfi -i testsrc2=size=1280x720:rate=50 -f lavfi -i sine=frequency=440:sample_rate=48000 -t 60"
X265=preset=fast:keyint=100:min-keyint=100:scenecut=0:bframes=0:repeat-headers=1
ffmpeg $SRC -c:v libx265 -crf 28 -x265-params "$X265:open-gop=1" -c:a aac cra.mkv
ffmpeg $SRC -c:v libx265 -crf 28 -x265-params "$X265:open-gop=0" -c:a aac idr.mkv
ffmpeg -i cra.mkv -c copy -tag:v hvc1 -movflags +faststart cra.mp4
ffmpeg -i idr.mkv -c copy -tag:v hvc1 -movflags +faststart idr.mp4
ffmpeg -ss 20 -i cra.mkv -t 20 -c copy cra-start.mkv
ffmpeg -i cra-start.mkv -c copy -tag:v hvc1 -movflags +faststart cra-start.mp4
Stream structure (verified with -bsf:v trace_headers):
| File | Keyframes |
|---|---|
cra.mkv |
1× IDR_N_LP (frame 0), then 29× CRA_NUT, no RASL/RADL pictures |
idr.mkv |
30× IDR_N_LP |
cra-start.* |
10× CRA_NUT, first picture is a CRA |
VPS/SPS/PPS are repeated before every keyframe, and Matroska keyframe flags are set on the CRAs.
I opened each file directly in a Firefox tab, played it for a few seconds, then seeked forward.
Environment: Firefox 159, macOS, VideoToolbox hardware HEVC decoding, media.mkv.enabled=true.
Originally found via Plex Web Direct Play of HandBrake 1.6.1 HEVC MKV files, which have the same structure (IDR first, CRA after).
Actual results
| File | On start | After seek |
|---|---|---|
cra.mkv |
video + audio | no video, audio continues |
cra.mp4 |
video + audio | video + audio |
cra-start.mkv |
no video, audio | no video, audio |
cra-start.mp4 |
no video, audio | no video, no audio, jumps to end |
idr.mkv |
video + audio | video + audio |
idr.mp4 |
video + audio | video + audio |
Decoding never starts on a CRA picture. MP4 seeks to CRAs only work if decoding already started on an IDR; it looks like the decoder isn't re-initialised on MP4 seeks. In Matroska every seek fails, which suggests the decoder is re-initialised there and can't restart on a CRA.
Observed on the real-world MKV after a seek:
totalVideoFramesstops increasing,droppedVideoFramesstays 0, there is noMediaError, andreadyStateis 4HTMLMediaElement.bufferedimmediately jumps to[seekTime, duration], which isn't plausible over the network- Loading with
#t=<offset>(e.g. resuming playback) shows the first frame, then no video
The same files play and seek correctly in QuickTime Player.
Expected results
Video should decode and play after seeking to, or starting at, a CRA keyframe, in both MKV and MP4.
A CRA picture with no leading pictures is a valid random access point in HEVC (it is treated as NoRaslOutputFlag=1 when it is the first picture), and QuickTime decodes these files correctly.
Comment 3•10 days ago
|
||
| gecko-triage | ||
This is the file playback side of bug 2065822, which covers the MSE path on macOS. The cause is in the current tree. The attached samples match the report: HEVC with a CRA_NUT at each sync sample, and no IDR, RASL or RADL picture.
Analysis
On macOS, both demuxers mark an HEVC sample as a keyframe only when it has an IDR picture. H265::IsKeyFrame() uses H265NALU::IsIframe(), which accepts only IDR_W_RADL and IDR_N_LP (H265.h). Thus a CRA picture is never a keyframe on macOS.
- Matroska:
MatroskaDemuxer::CheckKeyFrameByExamineByteStream()usesH265::IsKeyFrame()underMOZ_APPLEMEDIA, andH265::IsRandomAccessPoint()on other platforms (MatroskaDemuxer.cpp). The Matroska keyframe flags on the CRAs are not used. - MP4:
MP4TrackDemuxer::GetNextSample()replaces the stss sync flag with the result ofH265::IsKeyFrame()(MP4Demuxer.cpp).
After a seek, WebMTrackDemuxer::GetSamples() skips every sample until it finds a keyframe (WebMDemuxer.cpp). With no IDR after the seek point, it reads to the end of the file, and the video track ends. This explains the cra.mkv seek: no video, audio continues, no MediaError, and totalVideoFrames stops. It probably also explains the buffered range that jumps to the end of the file. cra-start.mkv has no IDR, so no video plays at start or after a seek.
MP4TrackDemuxer::Seek() tries again from the first stss sync sample when it finds no keyframe (MP4Demuxer.cpp). In cra.mp4, that sample is the IDR at 0, so the seek works but decodes from the start. cra-start.mp4 has no IDR. At start, MediaChangeMonitor drops every sample that is not a keyframe, with no error. The seek fails with end of stream, so the audio seek does not start and playback goes to the end.
Regression status
The MP4 part probably comes from bug 2049615 (Firefox 154), which is the regressor of bug 2065822. The Matroska part is not a recent regression. Bug 2065822 comment 9 reports the same MKV failure in Firefox 158 and in a Nightly from May.
Fix direction
The patch D338450 on bug 2065822 (comment 7, by Marcus) keeps the container keyframe flag in MP4Demuxer. It also skips the RASL pictures that VideoToolbox cannot decode, in AppleVTDecoder. It does not change MatroskaDemuxer. Bug 2065822 comment 9 reports that H265::IsRandomAccessPoint() in MatroskaDemuxer on macOS too makes MKV seeks work (0/10 to 10/10). This bug keeps the Matroska part open, and it blocks bug 1422891 (Matroska support).
If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.
Description
•