Open Bug 2065822 Opened 3 months ago Updated 2 days ago

MSE silently drops an HEVC coded frame group that starts on a CRA picture (macOS, Gecko >=154)

Categories

(Core :: Audio/Video: Playback, defect, P2)

defect

Tracking

()

ASSIGNED
Tracking Status
firefox-esr115 --- unaffected
firefox-esr153 --- unaffected
firefox157 --- wontfix
firefox158 --- wontfix

People

(Reporter: leonard, Assigned: git)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file, 3 external files)

Follow-up to bug 2049615.

Reproducer: one self-contained 112 KB HTML file, repro-standalone.html. It carries the three test fragments inside it and needs no server and no sibling files. Open it and it prints the table below.

Summary

On macOS (Tahoe 26.6.2), Gecko 154 does not treat an HEVC CRA picture as a random access point. On the file-playback path the demuxer falls back to the preceding IDR. On the MSE path there is nothing to fall back to, because the page supplies the fragments. The coded frame group is never opened: the CRA is dropped, need random access point stays true, and every following sample is dropped with it.

The failure is silent. Appends succeed, updateend fires, readyState reaches HAVE_METADATA, SourceBuffer.buffered stays empty. No error event, no MediaError, nothing in the console.

In the open-GOP HEVC streams this affects, the only IDR is at t=0 and every later keyframe is a CRA. Such a stream plays from the start and cannot be seeked or resumed.

Environment

Affected Gecko 154 on macOS 26.6.2, Apple silicon (UA rv:154.0)
Not affected Gecko 153 on macOS; Chrome 151 on macOS
Content HEVC Main 10, open-GOP (x265 --open-gop)

Bug 2049615 is RESOLVED FIXED, target_milestone 154 Branch; cf_status_firefox152 and cf_status_firefox153 are wontfix.

Steps to reproduce

Open repro-standalone.html in the browser under test. Nothing else is needed; it runs from file:// as well as over HTTP.

file starts on leading pictures
A-idr-start.mp4 IDR_N_LP at t=0 none
B-cra-clean.mp4 CRA_NUT at t=6 none (bframes=0)
C-cra-with-rasl.mp4 CRA_NUT at t=6 4 RASL pictures

The page appends each file's init segment and first media segment to a fresh SourceBuffer, waits for updateend, and reports buffered plus getVideoPlaybackQuality().totalVideoFrames. It also rewrites the first slice NAL's type in place, to establish which types the engine opens a coded frame group on.

B-cra-clean.mp4 is the isolated case: a CRA with no leading pictures, so the fragment references nothing outside itself.

Measured

Same page, same files, same machine. frames is getVideoPlaybackQuality().totalVideoFrames after 1.5 s of playback.

case first slice NAL Gecko 153 Gecko 154 Chrome 151
A, IDR entry (control) IDR_N_LP 0.00-4.00 / 36 / no error 0.00-4.00 / 37 / no error 0.00-4.00 / 40 / no error
B, clean CRA entry CRA_NUT 0.00-4.00 / 37 / no error empty / 0 / no error 0.00-4.00 / 41 / no error
B, relabelled IDR_N_LP IDR_N_LP 0.00-4.00 / 0 / -12909 0.00-4.00 / 0 / -12909 0.00-4.00 / 0 / -12909
B, relabelled BLA_N_LP BLA_N_LP 0.00-4.00 / 37 / no error empty / 0 / no error 0.00-4.00 / 42 / no error
C, CRA entry, RASL kept CRA_NUT 0.08-4.36 / 0 / -17694 empty / 0 / no error 0.24-4.36 / 43 / no error

Frame counts vary by a frame or two between runs, since they are whatever the decoder produced inside the page's 1.5 s window. What does not vary is whether a row decodes at all.

Row B is the regression. The same 25 KB fragment decodes on Gecko 153 and on Chrome. Gecko 154 stores nothing and reports nothing.

Chrome and Gecko are both on VideoToolbox here. Chrome's error text in the IDR_N_LP row reads PipelineStatus::PIPELINE_ERROR_DECODE: Error Domain=NSOSStatusErrorDomain Code=-12909 "(null)" (-12909): VTDecompressionOutputCallback, and Gecko's reads AppleVTDecoder::OnDecodeError.

Three further measurements from the same table:

  • The IDR_N_LP relabel fails identically on all three engines with OSStatus -12909 (kVTVideoDecoderBadDataErr). An IDR slice header omits slice_pic_order_cnt_lsb, so the relabelled slice no longer parses as what it claims to be. This row is the control for the relabelling itself.
  • BLA_N_LP behaves exactly like CRA_NUT on all three engines.
  • Row C, a CRA entry point with its RASL pictures kept, decodes on Chrome (43 frames, no error) and fails on Gecko 153 with OSStatus -17694 (kVTVideoDecoderReferenceMissingErr). On Gecko 154 the samples do not reach the decoder, so the row reads empty for the same reason as row B.

The same behaviour reproduces on a real 1920x960 HEVC Main 10 open-GOP file at a CRA 178.1 s in: buffered empty on Gecko 154, 0.0-10.0 with 149 frames when the same pipeline starts at the t=0 IDR instead.

Container signalling

From the mid-stream fragment:

tfhd default_sample_flags = 0x01010000   (sample_depends_on=1, non-sync)
trun first_sample_flags   = 0x02000000   (sample_depends_on=2, sync)
first slice NAL           = 21 (CRA_NUT)

first_sample_flags marks the first sample sync and depending on nothing.

Mechanism

dom/media/platforms/agnostic/bytestreams/H265.h. H265NALU::IsIframe() returns true only for IDR_W_RADL and IDR_N_LP:

bool IsIframe() const {
  return mNalUnitType == NAL_TYPES::IDR_W_RADL ||
         mNalUnitType == NAL_TYPES::IDR_N_LP;
}

dom/media/mp4/MP4Demuxer.cpp, GetNextSample(). Added by bug 2049615, this overrides the container's sync flag with that answer:

} else if (mType == kHEVC && !sample->mCrypto.IsEncrypted()) {
#ifdef MOZ_APPLEMEDIA
  // VideoToolbox can return a bad data error if a CRA frame is the first
  // sample after a seek. Only IDR_W_RADL/IDR_N_LP are safe starting points.
  auto isIDR = H265::IsKeyFrame(sample);
  bool keyframe = isIDR.isOk() && isIDR.unwrap();
  if (sample->mKeyframe != keyframe) {
    NS_WARNING(...);
    sample->mKeyframe = keyframe;
  }
#endif

TrackBuffersManager demuxes appended segments through this same MP4Demuxer, so the override applies to MSE as well as to file playback.

What a page can do about it

Nothing. There is no signal to key a fallback on: successful appends, updateend, readyState at HAVE_METADATA, and buffered that never grows are also what a slow network looks like. The only trace anywhere is the NS_WARNING above, which is not present in a release build.

Suggested fix

Chromium handles the same decoder constraint on the same platform without demoting the CRA, and its code is worth comparing against.

1. Treat every IRAP picture as a random access point. media/formats/mp4/hevc.cc sets is_keyframe = true for the full IRAP set, BLA_W_LP through RSV_IRAP_VCL23, and false for the non-IRAP VCL types. H265NALU::IsIframe() currently answers for IDR_W_RADL and IDR_N_LP only.

2. Handle the VideoToolbox constraint in the VideoToolbox decoder. media/gpu/mac/video_toolbox_h265_accelerator.cc, in SubmitSlice:

if (pic->no_rasl_output_flag_ &&
    (slice_hdr->nal_unit_type == H265NALU::RASL_N ||
     slice_hdr->nal_unit_type == H265NALU::RASL_R)) {
  // Drop this RASL frame, otherwise VideoToolbox will fail to decode it.
  drop_frame_ = true;
  return Status::kOk;
}

// Update frame state.
frame_is_keyframe_ = slice_hdr->irap_pic;

That is H.265 section 8.1.3: when a CRA begins decoding, NoRaslOutputFlag is 1 and its associated RASL pictures are neither decoded nor output. Doing this in AppleVTDecoder, which is the component that has the constraint and knows when it has just been created or flushed, leaves the demuxer free to report the bitstream as it is.

The current shape puts a decoder-specific constraint into a container-level answer, and every consumer of that answer inherits it. MP4Demuxer reads it for seeking, TrackBuffersManager reads it for coded frame processing, and the #ifdef makes Gecko's own keyframe semantics differ between macOS and the other platforms.

The seek retry that bug 2049615 also added, extending the kH264 fallback to kHEVC, is unaffected by any of this.

3. Whatever the fix, surface a drop. A sample discarded during coded frame processing currently produces nothing observable in a release build. A MediaError, a SourceBuffer error event, or a console warning would let a page fall back instead of hanging.

Workarounds tried

result
Every tfhd / trun sync-flag combination no change, the demuxer overrides the container
Init segment split off from the first moof+mdat no change
appendWindowStart, timestampOffset, mode="sequence", duplicate init append, abort() before the media append no change
removeSourceBuffer + addSourceBuffer on seek next seek fails with MediaError 3 / "manager is detached"
Relabel the CRA IDR_N_LP buffers, then MediaError 3, OSStatus -12909; fails the same way on Chrome
Relabel the CRA BLA_N_LP buffered empty, as with the CRA
WebCodecs VideoDecoder isConfigSupported returns supported: false for hvc1.2.4.L120.B0, hev1.2.4.L120.B0 and hvc1.1.6.L93.B0; avc1.640028 returns true

We ship a workaround: a WASM HEVC decoder in a worker re-encodes to H.264 via VideoEncoder, so the SourceBuffer codec is avc1 and Gecko never sees HEVC. Seeking works. It replaces the hardware HEVC decoding the same browser uses for <video> playback with a software decode that runs between 0.7x and 2x real time depending on machine load.

Attachment

repro-standalone.html is attached: 112 KB, the three fragments embedded, no server and no sibling files needed. It prints the table above.

The unpacked form is A-idr-start.mp4, B-cra-clean.mp4, C-cra-with-rasl.mp4 and repro.html, which fetches them as siblings; those plus the two generator scripts can be attached on request. The page derives each file's hvc1 codec string from its own hvcC, so any HEVC fragmented MP4 dropped next to it works unchanged. A short excerpt of the real-world 1920x960 Main 10 file is available on request.

I don't have editbugs, so I can't set the fields myself. For whoever triages this:

  • Regressed by: bug 2049615
  • Keywords: regression
  • OS: macOS (26.6.2)
  • Hardware: ARM64

Gecko 153 and Chrome 151 on the same machine both decode the attached testcase's row B; Gecko 154 stores nothing and reports nothing.

Severity: -- → S3
Keywords: regression
Priority: -- → P2
Regressed by: 2049615

:padenot, since you are the author of the regressor, bug 2049615, could you take a look?

For more information, please visit BugBot documentation.

Flags: needinfo?(padenot)

Bug 2049615 worked around a VideoToolbox decode failure by demoting HEVC CRA
pictures to non-keyframes in MP4Demuxer::GetNextSample() under
#ifdef MOZ_APPLEMEDIA. That puts a decoder-specific constraint into a
container-level answer, and every consumer inherits it. On the file playback
path the demuxer can fall back to the preceding IDR, but TrackBuffersManager
demuxes appended segments through the same MP4Demuxer, and there the coded
frame group is simply never opened: the CRA is dropped, need random access point stays true, and every following sample is dropped with it. buffered
stays empty, with no error event and nothing in the console. For an open-GOP
stream whose only IDR is at t=0, that means MSE playback cannot be seeked or
resumed at all.

The constraint the original bug hit is narrower than "a CRA is unsafe". Per
the H265 spec, 8.1.3, when an IRAP picture starts the decoding process its
NoRaslOutputFlag is 1 and the RASL pictures associated with it reference
pictures that are not present, so they are neither decoded nor output.
VideoToolbox reports kVTVideoDecoderReferenceMissingErr (-17694) instead of
ignoring them. A CRA with no leading pictures always decoded fine.

So handle it where the constraint actually lives:

  1. H265NALU gains spec-accurate type predicates (IsVCLType, IsIRAPType,
    IsBLAType, IsRASLType) replacing IsIframe(), which only accepted
    IDR_W_RADL/IDR_N_LP. H265::IsKeyFrame() now reports any IRAP picture
    (spec 3.73) as a random access point, matching what Chromium's
    media/formats/mp4/hevc.cc does. This also fixes
    MatroskaDemuxer::CheckKeyFrameByExamineByteStream(), which was wrong on
    every platform, not just macOS.

  2. The MOZ_APPLEMEDIA keyframe demotion is removed from MP4Demuxer, so the
    demuxer reports the bitstream as it is and Gecko's keyframe semantics no
    longer differ between macOS and everything else.

  3. New H265RaslSkipper tracks NoRaslOutputFlag across a stream, and
    AppleVTDecoder::ProcessDecode() skips the RASL pictures that VideoToolbox
    cannot decode; ProcessFlush() resets it, since the next sample restarts
    the decoding process. The state machine lives in shared code rather than
    in the Apple backend so it is compiled and unit-tested on all platforms.

The seek retry that bug 2049615 also added is left in place.

Testing

MP4Demuxer.SeekHEVC previously asserted the demoted behaviour. It now asserts
that seeking to 2.0s in an open-GOP stream lands on the CRA rather than
falling back to the IDR at t=0, and that H265::IsKeyFrame() agrees. That
assertion fails before this patch and passes after:

TestMP4Demuxer.cpp:629
Value of: isKeyFrame.isOk() && isKeyFrame.unwrap()
Actual: false
Expected: true

Three new gtests cover H265::IsKeyFrame() over the whole IRAP range and the
non-IRAP VCL types, and H265RaslSkipper for the CRA-starts-decoding,
CRA-mid-stream, post-flush, BLA and unreadable-bitstream cases. AnnexB.,
H264.
, H265.* and MP4Demuxer.* are green (55 tests).

Reviewer note: the AppleVTDecoder part could not be compiled or exercised
here, only the cross-platform code was, so the RASL skipping needs a check on
macOS against an open-GOP HEVC stream. The reporter's standalone MSE testcase
on bug 2065822 (row B is the regression, row C exercises the RASL path)
covers both halves.

See Also: → 2075203

The bug has a release status flag that shows some version of Firefox is affected, thus it will be considered confirmed.

Status: UNCONFIRMED → NEW
Ever confirmed: true

Confirmed on macOS 27.2 (M4), Firefox 160.0b2. Repro: an x265 open-GOP video (IDR only at 0:00, CRA elsewhere) as fMP4 through MSE.

Entering or seeking at a segment that starts with a CRA:

  • Firefox on macOS: dropped, buffered empty, readyState 1, seeks hang
  • Chrome 155 and Safari 27.2 on the same Mac and Firefox on Windows play
  • IDR at every segment: plays
  • CRA with RASL, CRA without RASL, and CRA relabelled BLA: all dropped

D321849 has the right design (every IRAP is a random access point, only the undecodable RASL pictures are skipped, as in Chromium) but doesn't apply to current central and has gaps. I rebased it and reworked it, and it builds and the gtests pass (including new H265.RaslSkipper* tests) on macOS 27.2, Apple silicon.

Changes from D321849:

  • H265::IsKeyFrame() is untouched (the draft changed it to mean "any IRAP" and removed IsIframe()), and the code reuses SampleContainsNalu() instead of adding a new HVCC parser. SeekHEVC uses the existing IsRandomAccessPoint()
  • The "decoding starts" state was used up by the first sample after a reset even if it had no picture or couldn't be read, so the next CRA was treated as mid-stream and its RASL pictures failed. Only a sample that carries a picture uses it up now, and an unreadable sample leaves the state alone
  • An EOS/EOB NAL unit restarts decoding, so the next IRAP is handled as a start
  • Parameter sets in a skipped RASL sample were lost with it. They are now passed on with the next decoded sample and cleared on a flush
  • The MP4Demuxer seek retry still listed HEVC after HEVC stopped being demoted. This is H.264 only now
  • New gtests cover the skipper behaviours above, and SeekHEVC now asserts that it lands on the CRA

Before: entering at a CRA decodes nothing (0 frames, buffered empty) and seeks into CRA segments hang.
After: playback runs at the source frame rate with 0 dropped frames, except that a CRA followed by RASL pictures drops 1 frame in about half of the runs (3 of 6; an IDR-only version of the same video: 0 of 4). I haven't dug into that.

I'd like to take this bug and submit the fix.

This is a regression in Firefox 154 from bug 2049615. The cause is confirmed in the current tree. Marcus works on a fix (comment 5), based on the Hackbot patch D321849 (comment 3).

Analysis

On macOS, MP4TrackDemuxer::GetNextSample() replaces the container sync flag of each HEVC sample with the result of H265::IsKeyFrame() (MP4Demuxer.cpp). That function uses H265NALU::IsIframe(), which accepts only IDR_W_RADL and IDR_N_LP (H265.h). As a result, a CRA or BLA picture is not a random access point.

MSE demuxes appended video/mp4 data through the same MP4Demuxer. In TrackBuffersManager::ProcessFrames(), the need random access point flag stays true, so the code skips every sample of the segment (TrackBuffersManager.cpp). The only trace is a verbose MediaSourceSamples log line and a profiler marker. This explains rows B and C of the testcase and the BLA relabel row. Windows builds do not have the override, so Firefox on Windows plays the stream (comment 5).

The fix direction in D321849 and in comment 5 agrees with the code. H265NALU::IsIRAP() and H265::IsRandomAccessPoint() already exist (H265.cpp). Bug 2006226 added them after D321849 was posted, so the rework in comment 5 can use them.

Two more facts for the patch:

  • MatroskaDemuxer has the same IDR-only rule for HEVC under MOZ_APPLEMEDIA (MatroskaDemuxer.cpp). Comment 5 does not name it.
  • The cause of the occasional one-frame drop with RASL pictures in comment 5 is not known yet.

Bug 2075203 also lists bug 2049615 as its regressor. Its SourceBuffer.remove() symptom probably has the same cause, but nobody has verified this.

Next steps

The bug also waits for the needinfo on Paul from comment 2. Marcus asks to take the bug (comment 5). We suggest that a person with editbugs assigns the bug to him.


If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.

Bug 2049615 stopped MP4Demuxer from marking HEVC CRA samples as keyframes on
macOS because VideoToolbox fails to decode the RASL pictures that can follow a
CRA the decoder started from. The demotion also applies to MSE, where there is
no earlier IDR to fall back to, so a segment that starts on a CRA is silently
dropped and SourceBuffer.buffered stays empty.

Keep the keyframe flag of the container and skip the RASL pictures in
AppleVTDecoder instead: those that follow an IRAP picture the decoding started
from, or a BLA picture. The spec does not output them (8.1.3) and VideoToolbox
fails to decode them. Decoding starts again when the decoder is flushed and
after an end of sequence NALU. Only a sample that carries a picture uses up the
start of decoding, so a sample with only parameter sets, or one that can't be
read, doesn't affect it. The parameter sets of a skipped sample are prepended
to the next decoded one, as VideoToolbox applies in-band parameter sets to the
pictures that follow.

Assignee: nobody → git
Status: NEW → ASSIGNED

Since no one else had this assigned I figured it wouldn't be stepping on any toes to just submit a patch.

D338450 fixes HEVC seeking over MSE on macOS for me. Direct MKV playback still fails, because MatroskaDemuxer has the same IDR-only check (comment 6); applying the same change there fixes it too.

Setup: macOS 27.0.1, Apple M4, central e924f94175 (2026-11-01) with D338450 (diff of 2026-10-30), opt build. All clips are x265 Main 10, 640x360, 24 fps, keyint=96, with one IDR at 0:

Clip GOP After the IDR
C closed IDRs only (control)
D open CRAs with RASL
E open CRAs without RASL

Played as HLS fMP4 (4 s segments, hls.js) and as single MKV and MP4 files. 10 seeks per run; a seek passes when playback gets 1.5 s past the target and at least 24 new frames are decoded within 10 s.

central + D338450 + D338450 + MatroskaDemuxer change
MSE, C (IDR only) 10/10 10/10 10/10
MSE, D (CRA + RASL) 2/10 10/10 10/10
MSE, E (CRA, no RASL) 2/10 10/10 10/10
MKV, D and E 0/10 0/10 10/10
MP4 file, D and E 10/10 10/10

getVideoPlaybackQuality reported 0 dropped frames in every passing run. I did not check frame by frame for the drop in comment 5.

A real 1080p Main 10 episode (one IDR at 0, 241 CRAs, no RASL) that Jellyfin remuxes to HLS fMP4 freezes on every seek in 159 release and seeks normally with the patch.

In MKV, a seek advances currentTime but decodes no frames. The same happens in 158 release and a Nightly from May, so it is older than the MP4 regression. The change I tested:

--- a/dom/media/webm/MatroskaDemuxer.cpp
+++ b/dom/media/webm/MatroskaDemuxer.cpp
@@ -252,12 +252,7 @@ bool MatroskaDemuxer::CheckKeyFrameByExamineByteStream(
case NESTEGG_CODEC_HEVC: {
-#ifdef MOZ_APPLEMEDIA

  •  // VideoToolbox can only start from IDR, see MP4Demuxer.
    
  •  auto isKeyFrame = H265::IsKeyFrame(aSample);
    

-#else
auto isKeyFrame = H265::IsRandomAccessPoint(aSample);
-#endif

Clip E, for reproducing:
ffmpeg -f lavfi -i testsrc2=s=640x360:r=24:d=120 -c:v libx265 -pix_fmt yuv420p10le -tag:v hvc1 -x265-params keyint=96:min-keyint=1:open-gop=1:bframes=0 e.mkv

Since it is the same assumption as in MP4Demuxer, it may make sense to fold this into D338450.

Confirming on release Firefox 159.0 / macOS 27.0.1 / Apple M5 Max (cf_status_firefox159 can be set to affected).

Real-world impact: Emby web (hls.js 1.6.0-beta.2, MPEG-TS remux, video stream copy) hangs on every resume/seek of x265 open-GOP Main 10 files, since only the first frame of the file is IDR.

Controlled A/B with Emby’s own hls.js on the same 1080p Main 10 source: CRA-start segments decode 0 frames with no error, whether timestamps start at ~1188 s or are reset to 0; the same position re-encoded closed-GOP (IDR start, same timestamps) plays in 44 ms; H.264 at the same position plays.

Safari (native HLS) plays the CRA-start stream.

Happy to test a build with D338450 on Apple Silicon.

See Also: → 2091293

(In reply to polybjorn from comment #9)

D338450 fixes HEVC seeking over MSE on macOS for me. Direct MKV playback still fails, because MatroskaDemuxer has the same IDR-only check (comment 6); applying the same change there fixes it too.

Setup: macOS 27.0.1, Apple M4, central e924f94175 (2026-11-01) with D338450 (diff of 2026-10-30), opt build. All clips are x265 Main 10, 640x360, 24 fps, keyint=96, with one IDR at 0:

Clip GOP After the IDR
C closed IDRs only (control)
D open CRAs with RASL
E open CRAs without RASL

Played as HLS fMP4 (4 s segments, hls.js) and as single MKV and MP4 files. 10 seeks per run; a seek passes when playback gets 1.5 s past the target and at least 24 new frames are decoded within 10 s.

central + D338450 + D338450 + MatroskaDemuxer change
MSE, C (IDR only) 10/10 10/10 10/10
MSE, D (CRA + RASL) 2/10 10/10 10/10
MSE, E (CRA, no RASL) 2/10 10/10 10/10
MKV, D and E 0/10 0/10 10/10
MP4 file, D and E 10/10 10/10

getVideoPlaybackQuality reported 0 dropped frames in every passing run. I did not check frame by frame for the drop in comment 5.

A real 1080p Main 10 episode (one IDR at 0, 241 CRAs, no RASL) that Jellyfin remuxes to HLS fMP4 freezes on every seek in 159 release and seeks normally with the patch.

In MKV, a seek advances currentTime but decodes no frames. The same happens in 158 release and a Nightly from May, so it is older than the MP4 regression. The change I tested:

--- a/dom/media/webm/MatroskaDemuxer.cpp
+++ b/dom/media/webm/MatroskaDemuxer.cpp
@@ -252,12 +252,7 @@ bool MatroskaDemuxer::CheckKeyFrameByExamineByteStream(
case NESTEGG_CODEC_HEVC: {
-#ifdef MOZ_APPLEMEDIA

  •  // VideoToolbox can only start from IDR, see MP4Demuxer.
    
  •  auto isKeyFrame = H265::IsKeyFrame(aSample);
    

-#else
auto isKeyFrame = H265::IsRandomAccessPoint(aSample);
-#endif

Clip E, for reproducing:
ffmpeg -f lavfi -i testsrc2=s=640x360:r=24:d=120 -c:v libx265 -pix_fmt yuv420p10le -tag:v hvc1 -x265-params keyint=96:min-keyint=1:open-gop=1:bframes=0 e.mkv

Since it is the same assumption as in MP4Demuxer, it may make sense to fold this into D338450.

I agree it makes sense to add to this fix. I've added it and will push after some testing.

Attachment #9671895 - Attachment description: Bug 2065822 - Treat HEVC CRA as a random access point and skip undecodable RASL pictures in AppleVTDecoder. r=#media-playback-reviewers → Bug 2065822 - Treat HEVC CRA as a random access point and skip undecodable RASL pictures in AppleVTDecoder. r?#media-playback-reviewers

MatroskaDemuxer had the same IDR-only keyframe check for HEVC under
MOZ_APPLEMEDIA as MP4Demuxer, so seeking an open-GOP HEVC Matroska file on
macOS never landed on a CRA and decoded nothing. AppleVTDecoder now skips the
RASL pictures VideoToolbox can't decode, so report every IRAP picture as a
keyframe on all platforms.

Tested D338450 on a local build, and it fixes MSE playback from CRA-start segments.

Setup

  • Apple M3 Pro, macOS 27.2.1
  • mozilla-central 0e6902a13c (2026-11-17) with D338450 applied (it applied cleanly)

Content

  • HEVC Main 10, open-GOP (IDR only at 0:00, CRA elsewhere), with AAC
  • Served by Jellyfin 12.1 as HLS fMP4 (remux, played with hls.js)

Before (release 160.0)

  • Resuming or seeking mid-movie: appends succeed, but SourceBuffer.buffered stays empty and playback stalls at 0.
  • Raw MSE testcase: segments that start on a CRA buffer nothing, with any timestamps (tfdt rewritten to ~0, timestampOffset ±1794, video-only).
  • IDR-start segments buffer fine in all cases.

After (with D338450)

  • All CRA-start cases buffer and play.
  • Resuming in the real Jellyfin web UI works.
Flags: needinfo?(padenot)

Additional data for bug 2065822: reproduced on release Firefox 160.0.1 (macOS) with Jellyfin's HLS fMP4 remux, plus an observation that might help narrow the trigger

Environment

  • Firefox 160.0.1 (release), macOS 27.2, Apple silicon (M1)
  • Jellyfin server and web 12.4.0 (the bundled hls.js appears to be 1.6.16; the version string appears twice in the chunk)
  • The server remuxes HEVC Main 10 (video copy, audio transcoded to AAC) into HLS fMP4 with jellyfin-ffmpeg 8.1.3

Symptom (matches this bug)
Playback stops at the end of the first segment (video buffered end 9.843 s). Segment 1 is fully downloaded (HAR: HTTP 200, 9.0 MB, received at 5.3 s, no later segment requests). It never shows up in the buffer, and there is no error from the append:

video.currentTime   = 9.814958
video.readyState    = 2
video.buffered      = [[0, 9.843166]]

hls.js only reports a non-fatal bufferStalledError from its GapController. There is no bufferAppendError and no fragParsingError.

What segment 1 contains (segments regenerated with the same ffmpeg command line, then moof/trun and NAL types inspected)

  • tfdt = 236236 at timescale 24000 = 9.8432 s, i.e. contiguous with the end of segment 0
  • first sample flags 0x02000000 (sync sample); the first slice NAL is type 21 (CRA), preceded by in-band VPS/SPS/PPS (hevc_mp4toannexb), followed by RASL pictures (pts 9.84–9.97 s)
  • init segment: video edit list with media_time 2002 (0.083 s, B-frame delay), audio edit list 0

Synthetic repro, no copyrighted content

ffmpeg -f lavfi -i "testsrc2=s=1916x800:r=24000/1001:d=30,format=yuv420p10le" \
  -f lavfi -i "anullsrc=r=48000:cl=5.1:d=30" -map 0:v -map 1:a \
  -c:v libx265 -preset veryfast -crf 20 \
  -x265-params "keyint=240:min-keyint=240:scenecut=0:open-gop=1" \
  -pix_fmt yuv420p10le -tag:v hvc1 -c:a ac3 -b:a 640k -t 30 testbild.mp4

Played through Jellyfin (Firefox cannot decode the AC3 track, so the server remuxes). The ffmpeg command line Jellyfin runs, with paths removed:

ffmpeg -fflags +genpts -f mov,mp4,m4a,3gp,3g2,mj2 -i file:"<input>" -map_metadata -1 -map_chapters -1 \
  -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 copy -tag:v:0 hvc1 -bsf:v hevc_mp4toannexb \
  -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts \
  -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 \
  -hls_time 6 -hls_segment_type fmp4 -hls_fmp4_init_filename "<id>-1.mp4" -start_number 0 \
  -hls_segment_filename "<dir>/<id>%d.mp4" -hls_playlist_type vod -hls_list_size 0 \
  -hls_segment_options movflags=+frag_discont+skip_sidx -y "<dir>/<id>.m3u8"

The clip stalls at ~9 s every time. The same happens with real film content that has a 10 s keyframe interval.

Observation that may help
On the same machine and browser, several other HEVC Main 10 clips with a CRA (plus RASL pictures) at the start of every segment after the first do play through in the same Jellyfin pipeline. I confirmed this from the server side: for those clips Jellyfin received requests far beyond the first segment, and playback continued for more than 90 s. I did not check buffered for them. I could not find what distinguishes them from the stalling clips: bitrate, keyframe interval, CRA/RASL, parameter sets, timestamps and sync flags were compared and did not separate the two groups. If the trigger depends on something other than the NAL type of the first sample of the segment (for example when the append starts a new coded frame group), this may be a useful hint. I can provide the exact ffprobe and trace_headers output for both groups on request.

Happy to test patched builds.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: