[MSE] Partial frames aren't removed when they should be

VERIFIED FIXED in Firefox 65

Status

()

VERIFIED FIXED
23 days ago
11 days ago

People

(Reporter: jya, Assigned: jya)

Tracking

(Blocks: 1 bug)

unspecified
mozilla67
Points:
---
Bug Flags:
in-qa-testsuite ?

Firefox Tracking Flags

(relnote-firefox 65+, firefox-esr60 affected, firefox65+ verified, firefox66+ verified, firefox67+ verified)

Details

(Whiteboard: [qa-triaged])

Attachments

(2 attachments)

(Assignee)

Description

23 days ago

The MSE spec states in the Coded Frame Removal algorithm:

https://w3c.github.io/media-source/#sourcebuffer-coded-frame-removal
step 3:
"Remove all media data, from this track buffer, that contain starting timestamps greater than or equal to start and less than the remove end timestamp."

When adding overlapping frame, we read :
https://w3c.github.io/media-source/#sourcebuffer-coded-frame-processing

step 13:
"If last decode timestamp for track buffer is unset and presentation timestamp falls within the presentation interval of a coded frame in track buffer, then run the following steps: "

while we do not implement the audio splice frame algorithm which has a special handling and remove partial audio.
For video, deciding if the overlapped window is to be removed depends on the frame timestamp only.

Yet, we only remove the frame if the removal internal includes the frame interval.

That it, it's only removed if:
frame.startTime >= removeInterval.startTime AND frame.endTime <= removal_interval.endTime

This can lead to problems where a single frame can be left off in the source buffer, and end up being in the wrong position as that frame is now ignored as unaccounted for in the index.

This is the cause for stalls when using Netflix interactive videos.

(Assignee)

Comment 1

23 days ago

Created attachment 9040655 [details]
Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

Per spec:
https://w3c.github.io/media-source/#sourcebuffer-coded-frame-removal
Step 3 of the Coded Frame Removal Interval:
"Remove all media data, from this track buffer, that contain starting timestamps greater than or equal to start and less than the remove end timestamp. "

So to decide if a frame should be removed from a track buffer, its start time is the only information to be used.

(Assignee)

Updated

23 days ago
Assignee: nobody → jyavenard
(Assignee)

Comment 2

23 days ago

Created attachment 9040656 [details]
Bug 1524500 - P2. Simplify logic to break loop early. r?bryce

The logic was redundant with the next step that will already remove all until the next keyframe.

Depends on D18321

Comment 3

22 days ago
Pushed by jyavenard@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/945469549521
P1. Remove partial frames when the starting time falls within the removal interval. r=bryce
https://hg.mozilla.org/integration/autoland/rev/ea48bec24f70
P2. Simplify logic to break loop early. r=bryce

Comment 4

21 days ago
bugherder
Status: NEW → RESOLVED
Last Resolved: 21 days ago
status-firefox67: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
(Assignee)

Updated

20 days ago
See Also: → bug 1524890
(Assignee)

Comment 5

19 days ago

Comment on attachment 9040655 [details]
Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

None

User impact if declined

This is seen with netflix interactive videos where playback will stall with no ability to recover.
The fix was confirmed by Netflix as working as intended.

Is this code covered by automated tests?

No

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

We apply the spec more correctly, we evict overlapped video frames more easily. The type of typical playback would make it unlikely to be seen with other video sites (which only stream straight content)

String changes made/needed

Attachment #9040655 - Flags: approval-mozilla-release?
Attachment #9040655 - Flags: approval-mozilla-beta?
status-firefox65: --- → affected
status-firefox66: --- → affected

Ryan, want this for next week's potential dot release?

tracking-firefox65: --- → ?

jya, how was this verified? By Netflix? It's nice to have QA confirm it if you can give STR. But if Netflix verified, that's fine too.

Flags: needinfo?(jyavenard)
(Assignee)

Comment 8

18 days ago

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #7)

jya, how was this verified? By Netflix? It's nice to have QA confirm it if you can give STR. But if Netflix verified, that's fine too.

Yes, by Netflix, sent to my private email.

The STR are:
0 - go to netflix.com/watch/80228265?t=14343
1 - Select “Chest”
2 - Select “Bookcase”
3 - Select “Redstone”
4 - Select “Gabriel”
let it play through the next few choice points without any interaction, letting the default choices to be picked; playback freezes eventually.

After selecting Gabriel, it will take about 1 minute for 2 choices to be selected, it will freeze shortly after the 2nd choice.
Reproduction rate is about 80%

Flags: needinfo?(jyavenard)

Comment on attachment 9040655 [details]
Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

Verified in Nightly, fix for interactive videos.
Let's uplift this for beta 6.

Attachment #9040655 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9040656 - Flags: approval-mozilla-beta+

Yeah, we can track this for 65.0.1. Jean-Yves, is this something we should consider taking on ESR60 as well? Not sure how widespread interactive video is on Netflix.

status-firefox-esr60: --- → affected
tracking-firefox65: ? → +
tracking-firefox66: --- → +
tracking-firefox67: --- → +

Comment on attachment 9040655 [details]
Bug 1524500 - P1. Remove partial frames when the starting time falls within the removal interval. r?bryce

[Triage Comment]
Fixes Netflix playback issues for some videos. Approved for 65.0.1.

Attachment #9040655 - Flags: approval-mozilla-release? → approval-mozilla-release+
Flags: qe-verify+
relnote-firefox: --- → 65+
Whiteboard: [qa-triaged]

I successfully reproduced the issue on Firefox Nightly 67.0a1 (2019-02-01) under Windows 10 (x64) using the STR from Comment 8.

The issue is no longer reproducible on Firefox Beta 66.0b6 and latest Nightly 67.0a1 (2019-02-11). Tests were performed under Windows 10 (x64), macOS 10.12 and Ubuntu 18.04 (x64).

I tried verifying it on treeherder build 65.0.1 (20190208225306), but as the video loads, the following message is shown on screen: "Streaming Error, The title is not available to watch instantly. Please try another title." with the error code: F7121-1331.

I will eventually verify it when 65.0.1 release will be out.

Status: RESOLVED → VERIFIED
status-firefox66: fixed → verified
status-firefox67: fixed → verified
Flags: qe-verify+

I verified the issue on Firefox 65.0.1 under Windows 10 (x64), Ubuntu 18.04 (x64) and macOS 10.12.

status-firefox65: fixed → verified
Flags: in-qa-testsuite?
You need to log in before you can comment on or make changes to this bug.