Video stalls if audio and video streams have different durations

RESOLVED FIXED

Status

()

defect
P2
normal
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: gerald, Assigned: jya)

Tracking

(Blocks 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Reporter

Description

4 years ago
When using separate source buffers for audio and video, and invoking endOfStream() after all buffering is complete, playback stalls if the stream have different-enough* durations.
* A difference of 0.1s didn't cause the issue (got the 'ended' event), but differences of 0.7s and 1.08s caused a stall (got the 'stalled' event).

Will attach proposed unit test showing the issue.

Could this have the same root cause as bug 1139271 and/or bug 1137118?
Reporter

Comment 1

4 years ago
Patch with 3 unit tests:
- test_Ended_mp4.html - Audio and video with similar sizes (within 0.1s), ends correctly
- test_Ended_mp4_short_audio.html - Shorter audio (by 1.08s), stalls
- test_Ended_mp4_short_video.html - Shorter video (by 0.7s), stalls
Shortening is done by just appending 1 fewer fragment file to one of the source buffers.
'ms.endOfStream()' is called in all tests.
(moved to Video/Audio and set Gerald as assignee).

This probably needs a review flag so it does not get forgotten. Anthony, can you pick a victim?
Component: Web Audio → Video/Audio
Flags: needinfo?(ajones)
Reporter

Comment 3

4 years ago
Sorry for the wrong assignment, I must have mis-clicked.

Not sure it the tests need a formal review now, until either: we dismiss the bug; or we add a fix, at which point the tests may be reviewed as well.

Added: Blocking bug 778617 (Tracking bug for MSE), this should ensure we won't forget this bug.
Blocks: MSE
Assignee

Comment 4

4 years ago
Why do you think it should behave differently?

I've made it so a .3s difference will ignored and ended will be fired. But anymore than that means a stall. 0.7s is a big difference. That playback stalls is the normal behaviour, crap in, crap out
Reporter

Comment 5

4 years ago
I naively expected the player to know that nothing else was coming after the explicit endOfStream(), as opposed to an unknown potential network delay when loading from a standard URL.

So playback could just end normally after having played whatever it could, either by 1. playing the shortest time, 2. filling the shorter track(s) with soul-crushing emptiness, or even 3. disabling the shorter track(s) -- as if it had received a single mp4 container with tracks of varying sizes (if such a thing is possible, e.g. a movie commentary track that ends at the titles, minutes before the video actually ends?)

Part of the end-of-stream algorithm (https://w3c.github.io/media-source/#end-of-stream-algorithm) is to set the duration to the highest end time of all source buffers, pointing towards my 2nd intuition, and suggesting that having different end times is acceptable.
Side question: Can the start time of separate tracks be different as well? How is it handled?

Also at end-of-stream we have: "Notify the media element that it now has all of the media data", but 'stalled' means "The user agent is trying to fetch media data" (http://www.w3.org/TR/html5/embedded-content-0.html#mediaevents), but the agent shouldn't try to fetch because it should know it has all the data -- though I guess you could interpret that to mean 'I need more data but I know I won't get more, so stall now'.

Finally your ability to handle a 0.3s difference (arbitrary number, or coming from a spec?) suggests that you would be able to handle any difference.


But despite all this, if my expectations are wrong (very possible), then please just close this bug as invalid; at least now we have a record of this behavior and we'll know we need to build tests that have audio&video streams with durations within 0.3s.

Providers should probably create content with matching durations anyway, so in the end it's a minor issue and I'm fine with punishing "crap in", or choosing to handle it much later in the project.
Flags: needinfo?(jyavenard)
Assignee

Comment 6

4 years ago
(In reply to Gerald Squelart [:gerald] from comment #5)
> I naively expected the player to know that nothing else was coming after the
> explicit endOfStream(), as opposed to an unknown potential network delay
> when loading from a standard URL.

The problem is that it's difficult to know when EndOfStream() means nothing more will be coming, or it's temporary.
http://w3c.github.io/media-source/#end-of-stream-algorithm
2.4.7 End of stream algorithm

Can be called in different circumstances than just an explicit call to EndOfStream().
EndOfStream() can also be called with an "error" argument.

Maybe what we could do is distinguish case where End Of Stream Algorithm was called with an error, and those without. And if called without error know so, and such that the MediaSourceReader returns EndOfStream

We currently can't distinguish a stream when its readyState is "ended" and how it got into "ended"
Flags: needinfo?(jyavenard)
Assignee

Updated

4 years ago
Depends on: 1141333

Updated

4 years ago
Priority: -- → P2
Paul - nothing to review yet.
Flags: needinfo?(ajones)
Assignee

Updated

4 years ago
Assignee: nobody → jyavenard
Assignee

Comment 8

4 years ago
For uplift, YouTube has a few videos where audi != video duration. the workaround currently in place is nasty
Flags: needinfo?(giles)
There's no patch to uplift here. Did you mean to nominate something else? If not, please needinfo me again once something's in tree.
Flags: needinfo?(giles) → needinfo?(jyavenard)
Assignee

Comment 10

4 years ago
Actually, was just a dependency to another.

so can say it's just fixed by bug 1141333
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Assignee

Updated

4 years ago
Flags: needinfo?(jyavenard)
You need to log in before you can comment on or make changes to this bug.