2039853 - HEVC DASH-LL playback via MSE resulting in video stuttering

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Ryan McCartney]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:150.0) Gecko/20100101 Firefox/150.0

Steps to reproduce:

Good afternoon, I'd like to report an issue effecting playback of video using Media Source Extensions (MSE) and chunked media. The issue only appears to effect video playback when using HEVC/H.265.

Playback of this content was working as expected until around ~6 months ago, unfortunately I don't have a more accurate date available.

The issue manifests as stuttering in video playback, audio continues uninterrupted throughout. This can be clearly seen using the Dash.js reference player and the BBC R&D live test streams as follows:

Case 1

HEVC DASH Low Latency Live Stream (Chunked Media, 3.84s segments, x4 0.96s chunks)

• Manifest: https://rdmedia.bbc.co.uk/testcard/lowlatency/manifests/ll-hevc-ctv.mpd
• Reference Player: https://reference.dashif.org/dash.js/v5.1.1/samples/dash-if-reference-player/index.html

Stuttering observed here

Case 2

AVC DASH Low Latency Live Stream (Chunked Media, 3.84s segments, x4 0.96s chunks)

• Manifest: https://rdmedia.bbc.co.uk/testcard/lowlatency/manifests/ll-avc-full.mpd
• Reference Player: https://reference.dashif.org/dash.js/v5.1.1/samples/dash-if-reference-player/index.html

Expected Behaviour seen in this case, no stuttering observed

Case 3

HEVC DASH (Media delivered to the source buffer as whole segments, 3.84s segments)

In this case the underlying media is fundamentally the same as offered in case one. However, in this case media is delivered as whole segments rather than chunk delivered using Chunk Transfer Encoding (CTE).

• Manifest: https://rdmedia.bbc.co.uk/testcard/simulcast/manifests/hevc-ctv.mpd
• Reference Player: https://reference.dashif.org/dash.js/v5.1.1/samples/dash-if-reference-player/index.html

Expected Behaviour seen in this case, no stuttering observed

Actual results:

When playing DASH Low Latency content (chunked media) encoded using HEVC Firefox stutters when presenting video.

The browser appears to show frames from the first chunk appended to the MSE source buffer as normal but video playback then freezes until the first chunk of the next segment is appended to the source buffer.

No errors are observed from MSE.

Expected results:

When playing DASH low latency content (or any chunked media) encoded using HEVC Firefox should render and present all frames from all chunks appended to the MSE source buffer.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[Ryan McCartney]

This may be a duplicate of — #2012108

Some further digging show this regression was introduced between 147.0b5 and 147.0b6 in this commit, as part of solving #1981503.

The effect is on the above streams the decoder reinitialised on every fragment of the fmp4 it receives even where there are no changes to the underlying config. The result on the test streams listed above is that only 1 in 4 chunks is successfully decoded.

Comment 4

[C.M.Chang[:chunmin]]

Thanks for the careful report and regression range!

A plausible assumption is that each LL-CTE chunk carries its own moof with slightly differing OOB extradata, which defeats the H265::CompareExtraData check in HEVCChangeMonitor::CheckForChange introduced in bug 2012108's fix.

Comment 5

[Ryan McCartney]

Thank-you for the suggestions, it looks like H265::CompareExtraData is indeed returning false. It's triggered on line 1645 in H265::CompareExtraData where numSPS is equal to 0.

  if (numSPS == 0 || numSPS != config2.NumSPS()) {
    return false;
  }

In the case of the reference streams above I would expect a single chunk within a segment to contain its own MOOF and MDAT. The segments and chunks in the media above are formed according to the CMAF specification (ISO/IEC 23000-19:2024).

Comment 6

[Ryan McCartney]

Is a check for numSPS == 0 valid in this comparison? mPreviousExtraData also has a numSPS equal to zero, this suggests H265::CompareExtraData is doing a bit more than named. Removing numSPS == 0 || would fix this issue from my point of view, but I'd like to be certain there isn't a reason for this existing before proposing it as a patch.

As it stands the current implementation matches the H264 one. The only other place H265::CompareExtraData is called is later in HEVCChangeMonitor::CheckForChange.

Comment 7

[Ryan McCartney]

Attachment: Bug 2039853 - Stop checking for a non-zero numSPS when comparing extra data. r=chunmin,jolin

Comment 8

[Ryan McCartney]

I've submitted the following patch, happy to amend or adjust strategy as needed - https://phabricator.services.mozilla.com/D303104

Comment 9

[C.M.Chang[:chunmin]]

Attachment: Bug 2039853 - Add gtest for H265::CompareExtraData equality of SPS-less configs

Comment 10

[C.M.Chang[:chunmin]]

Attachment: Bug 2039853 - Add MSE HEVC SPS-less CMAF test

test_MediaSource_hevc_synthetic_spsless.html appends a single SPS-less hev1 init
segment (hvcC numOfArrays==0; parameter sets in-band) followed by several CMAF
chunks appended individually: chunk0 is the keyframe (in-band VPS/SPS/PPS), the
rest are non-keyframe slices. The MSE pipeline hands each separately-appended
segment a pointer-distinct SPS-less out-of-band extradata buffer; at each segment
boundary H265::CompareExtraData returns false for the NumSPS()==0 buffers, so
HEVCChangeMonitor::CheckForChange reports a spurious configuration change and the
decoder is recreated, dropping the queued frames.

The fixture is generated from ffmpeg's testsrc (redistributable). The test fails
before the fix (only the keyframe chunk's frames decode) and passes after it.

Comment 11

[C.M.Chang[:chunmin]]

Attachment: Bug 2039853 - Add generator script for HEVC test fixtures under dom/media/mediasource/test/

generate_hevc_synth_fixtures.py runs ffmpeg+libx265 on a short testsrc clip
to emit a fragmented MP4 and splits the result into an init segment plus
per-fragment moof+mdat .m4s files. CLI flags expose the HEVC config knobs
that affect parser and decoder behavior: sample entry type (hev1 vs hvc1),
in-band vs out-of-band parameter sets, and an optional surgery that zeros
the hvcC numOfArrays byte to produce a spec-valid SPS-less out-of-band
record.

The default invocation produces a conventional hvc1 / out-of-band fragmented
MP4. The bug 2039853 SPS-less hev1 fixtures are produced via the cookbook
invocation documented in the module docstring:

python3 generate_hevc_synth_fixtures.py
--tag hev1 --in-band-params --sps-less
--name-prefix bug2039853_hevc_

The module docstring records the exact ffmpeg command line and the reference
toolchain (ffmpeg 6.1.1, libx265 3.5), so fixtures produced by the cookbook
invocations are byte-for-byte reproducible. The script is documentation only
-- it is not run by CI.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a05e3ad05dc3
https://hg.mozilla.org/integration/autoland/rev/a5a0807ad250
Add gtest for H265::CompareExtraData equality of SPS-less configs r=media-playback-reviewers,aosmond
https://github.com/mozilla-firefox/firefox/commit/6881c07fcc29
https://hg.mozilla.org/integration/autoland/rev/1f758e5e8135
Add generator script for HEVC test fixtures under dom/media/mediasource/test/ r=media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/b2965b341cea
https://hg.mozilla.org/integration/autoland/rev/b35f535c498e
Add MSE HEVC SPS-less CMAF test r=media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/964a57f25358
https://hg.mozilla.org/integration/autoland/rev/7e21c0fa635c
Stop checking for a non-zero numSPS when comparing extra data. r=chunmin

Comment 13

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/a5a0807ad250
https://hg.mozilla.org/mozilla-central/rev/1f758e5e8135
https://hg.mozilla.org/mozilla-central/rev/b35f535c498e
https://hg.mozilla.org/mozilla-central/rev/7e21c0fa635c

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/8033cbca4853
https://hg.mozilla.org/integration/autoland/rev/72b9a0a0d301
Revert "Bug 2039853 - Stop checking for a non-zero numSPS when comparing extra data. r=chunmin" for causing mda failures at test_MediaSource_hevc_spsless.html

Comment 15

[rperta]

Backed out for causing mda failures at test_MediaSource_hevc_spsless.html
Backout link
Push with failures
Failure log(s)

Comment 16

[Iulian Moraru]

Backout merged to central: https://hg-edge.mozilla.org/mozilla-central/rev/72b9a0a0d301

Comment 17

[C.M.Chang[:chunmin]]

(In reply to rperta from comment #15)

Backed out for causing mda failures at test_MediaSource_hevc_spsless.html
Backout link
Push with failures
Failure log(s)

It's probably because I accidentally corrupt the bug2039853_hevc_chunk*.m4s files. I've re-uploaded them

Comment 18

[C.M.Chang[:chunmin]]

(In reply to C.M.Chang[:chunmin] from comment #17)

(In reply to rperta from comment #15)

Backed out for causing mda failures at test_MediaSource_hevc_spsless.html

It's probably because I accidentally corrupt the bug2039853_hevc_chunk*.m4s files. I've re-uploaded them

No, I was wrong. It's caused by other issues. I'll check.

Comment 19

[C.M.Chang[:chunmin]]

It looks like macos 10.15 on CI doesn't support hardware decoding or HEVC. See Bug 1942396.

Comment 20

[Ryan McCartney]

Thanks for the work on producing test cases for this issue, it's a good approach and will hopefully make future occurrences of this issue detectable. Thanks, Ryan

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e82e0b12529e
https://hg.mozilla.org/integration/autoland/rev/a375bdbb0098
Add gtest for H265::CompareExtraData equality of SPS-less configs r=media-playback-reviewers,aosmond
https://github.com/mozilla-firefox/firefox/commit/cf312a88e9bf
https://hg.mozilla.org/integration/autoland/rev/40a305be71f2
Add generator script for HEVC test fixtures under dom/media/mediasource/test/ r=media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/e986181a5b3a
https://hg.mozilla.org/integration/autoland/rev/1fecacd92073
Add MSE HEVC SPS-less CMAF test r=media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/b876ad8e1b6e
https://hg.mozilla.org/integration/autoland/rev/7217252d4f97
Stop checking for a non-zero numSPS when comparing extra data. r=chunmin

Comment 22

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/a375bdbb0098
https://hg.mozilla.org/mozilla-central/rev/40a305be71f2
https://hg.mozilla.org/mozilla-central/rev/1fecacd92073
https://hg.mozilla.org/mozilla-central/rev/7217252d4f97