1140974 - Video stalls if audio and video streams have different durations

Status: RESOLVED FIXED

(Core :: Audio/Video, defect, P2)

Description

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

When using separate source buffers for audio and video, and invoking endOfStream() after all buffering is complete, playback stalls if the stream have different-enough* durations.
* A difference of 0.1s didn't cause the issue (got the 'ended' event), but differences of 0.7s and 1.08s caused a stall (got the 'stalled' event).

Will attach proposed unit test showing the issue.

Could this have the same root cause as bug 1139271 and/or bug 1137118?

Comment 1

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: 1140974-tests.patch

Patch with 3 unit tests:
- test_Ended_mp4.html - Audio and video with similar sizes (within 0.1s), ends correctly
- test_Ended_mp4_short_audio.html - Shorter audio (by 1.08s), stalls
- test_Ended_mp4_short_video.html - Shorter video (by 0.7s), stalls
Shortening is done by just appending 1 fewer fragment file to one of the source buffers.
'ms.endOfStream()' is called in all tests.

Comment 2

[Paul Adenot (:padenot)]

(moved to Video/Audio and set Gerald as assignee).

This probably needs a review flag so it does not get forgotten. Anthony, can you pick a victim?

Comment 3

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Sorry for the wrong assignment, I must have mis-clicked.

Not sure it the tests need a formal review now, until either: we dismiss the bug; or we add a fix, at which point the tests may be reviewed as well.

Added: Blocking bug 778617 (Tracking bug for MSE), this should ensure we won't forget this bug.

Comment 4

[Jean-Yves Avenard [:jya]]

Why do you think it should behave differently?

I've made it so a .3s difference will ignored and ended will be fired. But anymore than that means a stall. 0.7s is a big difference. That playback stalls is the normal behaviour, crap in, crap out

Comment 5

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

I naively expected the player to know that nothing else was coming after the explicit endOfStream(), as opposed to an unknown potential network delay when loading from a standard URL.

So playback could just end normally after having played whatever it could, either by 1. playing the shortest time, 2. filling the shorter track(s) with soul-crushing emptiness, or even 3. disabling the shorter track(s) -- as if it had received a single mp4 container with tracks of varying sizes (if such a thing is possible, e.g. a movie commentary track that ends at the titles, minutes before the video actually ends?)

Part of the end-of-stream algorithm (https://w3c.github.io/media-source/#end-of-stream-algorithm) is to set the duration to the highest end time of all source buffers, pointing towards my 2nd intuition, and suggesting that having different end times is acceptable.
Side question: Can the start time of separate tracks be different as well? How is it handled?

Also at end-of-stream we have: "Notify the media element that it now has all of the media data", but 'stalled' means "The user agent is trying to fetch media data" (http://www.w3.org/TR/html5/embedded-content-0.html#mediaevents), but the agent shouldn't try to fetch because it should know it has all the data -- though I guess you could interpret that to mean 'I need more data but I know I won't get more, so stall now'.

Finally your ability to handle a 0.3s difference (arbitrary number, or coming from a spec?) suggests that you would be able to handle any difference.


But despite all this, if my expectations are wrong (very possible), then please just close this bug as invalid; at least now we have a record of this behavior and we'll know we need to build tests that have audio&video streams with durations within 0.3s.

Providers should probably create content with matching durations anyway, so in the end it's a minor issue and I'm fine with punishing "crap in", or choosing to handle it much later in the project.

Comment 6

[Jean-Yves Avenard [:jya]]

(In reply to Gerald Squelart [:gerald] from comment #5)
> I naively expected the player to know that nothing else was coming after the
> explicit endOfStream(), as opposed to an unknown potential network delay
> when loading from a standard URL.

The problem is that it's difficult to know when EndOfStream() means nothing more will be coming, or it's temporary.
http://w3c.github.io/media-source/#end-of-stream-algorithm
2.4.7 End of stream algorithm

Can be called in different circumstances than just an explicit call to EndOfStream().
EndOfStream() can also be called with an "error" argument.

Maybe what we could do is distinguish case where End Of Stream Algorithm was called with an error, and those without. And if called without error know so, and such that the MediaSourceReader returns EndOfStream

We currently can't distinguish a stream when its readyState is "ended" and how it got into "ended"

Comment 7

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Paul - nothing to review yet.

Comment 8

[Jean-Yves Avenard [:jya]]

For uplift, YouTube has a few videos where audi != video duration. the workaround currently in place is nasty

Comment 9

[Ralph Giles (:rillian)]

There's no patch to uplift here. Did you mean to nominate something else? If not, please needinfo me again once something's in tree.

Comment 10

[Jean-Yves Avenard [:jya]]

Actually, was just a dependency to another.

so can say it's just fixed by bug 1141333