1118370 - Computing buffered ranges as the intersection of audio and video tracks can leave gaps

Status: RESOLVED INVALID

(Core :: Audio/Video: Playback, defect, P5)

Description

[Bobby Holley (:bholley)]

This is the source of the issue I brought up in the meeting yesterday. The basic problem is that the time boundaries of fragments don't necessarily line up perfectly between audio and video. Consider the following two fragments:

Fragment A:
Audio: [0, 1.005)
Video: [0, 1)

Fragment B:
Audio: [1.005, 2)
Video: [1, 2)

We have all the data we need here, but we currently don't make full use of it, because GetBuffered returns the following:

Union(Intersection(A.audio, A.video), Intersection(B.audio, B.video)) = Union([0, 1), [1.005, 2))

Which has a discontinuity. What we should instead be doing is:

Intersection(Union(A.audio, B.Audio), Union(A.video, B.video)) = Intersection([0, 2), [0, 2)) = [0, 2).

Which has no discontinuity.

The way to do this is to allow subdecoders to do GetBuffered on a per-track basis. This means that MediaSourceReader::GetBuffered can do the algorithm above, and MediaSourceReader::Select{Audio,Video}Reader can pick the right decoder, which might be different for audio and video given a single timestamp.

This might be the cause (or one of the causes) of bug 1103357.

Comment 1

[Bobby Holley (:bholley)]

I'm working on patches for this, which are coming along nicely.

Comment 2

[Bobby Holley (:bholley)]

Per IRC discussion - it sounds like YouTube always uses separate source buffers for audio/video, so this should be an on-issue for youtube (though still a problem for other sites).

I've got the patches done (with a test!) and will land them, but the timestampOffset stuff is first priority.

Comment 3

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

We don't currently support video/audio in the same source buffer.

Comment 4

[Bobby Holley (:bholley)]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #3)
> We don't currently support video/audio in the same source buffer.

Well, it kinda sorta works - better with the patches that I have. P5 though.

Comment 5

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Jean-Yves - can we close this?

Comment 6

[Jean-Yves Avenard [:jya]]

yes.. the problem was only with the old MSE and combined audio+video track.

Also the specs are clear on how a buffered range is to be calculated with MSE. and Bobby's description was incorrect.