Computing buffered ranges as the intersection of audio and video tracks can leave gaps

RESOLVED INVALID

Status

()

defect
P5
normal
RESOLVED INVALID
4 years ago
4 years ago

People

(Reporter: bholley, Assigned: bholley)

Tracking

(Blocks 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

4 years ago
This is the source of the issue I brought up in the meeting yesterday. The basic problem is that the time boundaries of fragments don't necessarily line up perfectly between audio and video. Consider the following two fragments:

Fragment A:
Audio: [0, 1.005)
Video: [0, 1)

Fragment B:
Audio: [1.005, 2)
Video: [1, 2)

We have all the data we need here, but we currently don't make full use of it, because GetBuffered returns the following:

Union(Intersection(A.audio, A.video), Intersection(B.audio, B.video)) = Union([0, 1), [1.005, 2))

Which has a discontinuity. What we should instead be doing is:

Intersection(Union(A.audio, B.Audio), Union(A.video, B.video)) = Intersection([0, 2), [0, 2)) = [0, 2).

Which has no discontinuity.

The way to do this is to allow subdecoders to do GetBuffered on a per-track basis. This means that MediaSourceReader::GetBuffered can do the algorithm above, and MediaSourceReader::Select{Audio,Video}Reader can pick the right decoder, which might be different for audio and video given a single timestamp.

This might be the cause (or one of the causes) of bug 1103357.
(Assignee)

Updated

4 years ago
Summary: Computed buffered ranges as the intersection of audio and video tracks can leave gaps → Computing buffered ranges as the intersection of audio and video tracks can leave gaps
(Assignee)

Comment 1

4 years ago
I'm working on patches for this, which are coming along nicely.
(Assignee)

Comment 2

4 years ago
Per IRC discussion - it sounds like YouTube always uses separate source buffers for audio/video, so this should be an on-issue for youtube (though still a problem for other sites).

I've got the patches done (with a test!) and will land them, but the timestampOffset stuff is first priority.
No longer blocks: 1103357
Depends on: 1064128
(Assignee)

Updated

4 years ago
Blocks: 1118533
We don't currently support video/audio in the same source buffer.
(Assignee)

Comment 4

4 years ago
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #3)
> We don't currently support video/audio in the same source buffer.

Well, it kinda sorta works - better with the patches that I have. P5 though.
Priority: -- → P5
Component: Audio/Video → Audio/Video: Playback
Jean-Yves - can we close this?
yes.. the problem was only with the old MSE and combined audio+video track.

Also the specs are clear on how a buffered range is to be calculated with MSE. and Bobby's description was incorrect.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.