1524500 - [MSE] Partial frames aren't removed when they should be

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect)

Description

[Jean-Yves Avenard [:jya]]

The MSE spec states in the Coded Frame Removal algorithm:

https://w3c.github.io/media-source/#sourcebuffer-coded-frame-removal
step 3:
"Remove all media data, from this track buffer, that contain starting timestamps greater than or equal to start and less than the remove end timestamp."

When adding overlapping frame, we read :
https://w3c.github.io/media-source/#sourcebuffer-coded-frame-processing

step 13:
"If last decode timestamp for track buffer is unset and presentation timestamp falls within the presentation interval of a coded frame in track buffer, then run the following steps: "

while we do not implement the audio splice frame algorithm which has a special handling and remove partial audio.
For video, deciding if the overlapped window is to be removed depends on the frame timestamp only.

Yet, we only remove the frame if the removal internal includes the frame interval.

That it, it's only removed if:
frame.startTime >= removeInterval.startTime AND frame.endTime <= removal_interval.endTime

This can lead to problems where a single frame can be left off in the source buffer, and end up being in the wrong position as that frame is now ignored as unaccounted for in the index.

This is the cause for stalls when using Netflix interactive videos.

Comment 1

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

Per spec:
https://w3c.github.io/media-source/#sourcebuffer-coded-frame-removal
Step 3 of the Coded Frame Removal Interval:
"Remove all media data, from this track buffer, that contain starting timestamps greater than or equal to start and less than the remove end timestamp. "

So to decide if a frame should be removed from a track buffer, its start time is the only information to be used.

Comment 2

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1524500 - P2. Simplify logic to break loop early. r?bryce

The logic was redundant with the next step that will already remove all until the next keyframe.

Depends on D18321

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/945469549521
P1. Remove partial frames when the starting time falls within the removal interval. r=bryce
https://hg.mozilla.org/integration/autoland/rev/ea48bec24f70
P2. Simplify logic to break loop early. r=bryce

Comment 4

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/945469549521
https://hg.mozilla.org/mozilla-central/rev/ea48bec24f70

Comment 5

[Jean-Yves Avenard [:jya]]

Comment on attachment 9040655 [details]
Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

None

User impact if declined

This is seen with netflix interactive videos where playback will stall with no ability to recover.
The fix was confirmed by Netflix as working as intended.

Is this code covered by automated tests?

No

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

We apply the spec more correctly, we evict overlapped video frames more easily. The type of typical playback would make it unlikely to be seen with other video sites (which only stream straight content)

String changes made/needed

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

Ryan, want this for next week's potential dot release?

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

jya, how was this verified? By Netflix? It's nice to have QA confirm it if you can give STR. But if Netflix verified, that's fine too.

Comment 8

[Jean-Yves Avenard [:jya]]

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #7)

jya, how was this verified? By Netflix? It's nice to have QA confirm it if you can give STR. But if Netflix verified, that's fine too.

Yes, by Netflix, sent to my private email.

The STR are:
0 - go to netflix.com/watch/80228265?t=14343
1 - Select “Chest”
2 - Select “Bookcase”
3 - Select “Redstone”
4 - Select “Gabriel”
let it play through the next few choice points without any interaction, letting the default choices to be picked; playback freezes eventually.

After selecting Gabriel, it will take about 1 minute for 2 choices to be selected, it will freeze shortly after the 2nd choice.
Reproduction rate is about 80%

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9040655 [details]
Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

Verified in Nightly, fix for interactive videos.
Let's uplift this for beta 6.

Comment 10

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/releases/mozilla-beta/rev/6eb6ad4198da
https://hg.mozilla.org/releases/mozilla-beta/rev/16b643f92fbf

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Yeah, we can track this for 65.0.1. Jean-Yves, is this something we should consider taking on ESR60 as well? Not sure how widespread interactive video is on Netflix.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9040655 [details]
Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

[Triage Comment]
Fixes Netflix playback issues for some videos. Approved for 65.0.1.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/f8e028cb3d83
https://hg.mozilla.org/releases/mozilla-release/rev/1b42fcc6c253

Comment 14

[Catalin Sasca, Desktop Test Engineering [:csasca]]

I successfully reproduced the issue on Firefox Nightly 67.0a1 (2019-02-01) under Windows 10 (x64) using the STR from Comment 8.

The issue is no longer reproducible on Firefox Beta 66.0b6 and latest Nightly 67.0a1 (2019-02-11). Tests were performed under Windows 10 (x64), macOS 10.12 and Ubuntu 18.04 (x64).

I tried verifying it on treeherder build 65.0.1 (20190208225306), but as the video loads, the following message is shown on screen: "Streaming Error, The title is not available to watch instantly. Please try another title." with the error code: F7121-1331.

I will eventually verify it when 65.0.1 release will be out.

Comment 15

[Catalin Sasca, Desktop Test Engineering [:csasca]]

I verified the issue on Firefox 65.0.1 under Windows 10 (x64), Ubuntu 18.04 (x64) and macOS 10.12.