1294412 - [MSE] After endOfStream(), buffered video unplayable

Status: RESOLVED INVALID

(Core :: Audio/Video: Playback, defect)

Description

[CoolCmd]

Attachment: testcase

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160726073904

Steps to reproduce:

load testcase and open console


Actual results:

audio segments and video segments are added, and endOfStream() called. audio and video ranges not intersect (part of audio stream is lost). HTMLMediaElement.buffered contain only video range (this is OK). but Firefox fail to seek: 'seeked' event never fired.

Chrome playing back this testcase without problems. if you comment any using g_oAudioSourceBuffer and append only video segments (m4v), Firefox also playing back this testcase without problems.


Expected results:

a) Firefox must seek (and later playback) buffered stream.
- or -
b) HTMLMediaElement.buffered.length must be equal 0 after endOfStream().

Comment 1

[Jean-Yves Avenard [:jya]]

First, that it works in chrome, doesn't mean it's the right thing to do or that it's per the w3c spec.

if you have a gap of more than 125ms, playback will stall, or if you attempt to seek in a position where no data is buffered +/- 125ms seek won't complete. Then instead "waiting" will be fired instead.

the buffered range and seekable range is there to show where you can seek and where seek will immediately succeed if no data is to be added to the sourcebuffers.

it is up to your player to detect gaps and adjust where it will seek. 

Our buffered MSE implementation is 100% per spec. https://w3c.github.io/media-source/index.html#widl-SourceBuffer-buffered. The sourcebuffer buffered attribute is the intersection of each track in the sourcebuffer and the media element buffered range is the intersection of the sourcebuffer attached.
https://w3c.github.io/media-source/index.html#widl-SourceBuffer-buffered

I recommend that you install the about:media add-on which will help greatly understanding what is going on:
https://addons.mozilla.org/en-US/firefox/addon/about-media/?src=api (note it's currently not compatible with e10s, so you must disable multiprocess first)

http://localhost/~jyavenard/tests/coolcmd/coolcmd.html
	blob:http://localhost/08dae279-73df-4d38-97b5-5a8f8b4aba64
	currentTime: 844.911
	Quality: -- (total:0 dropped:0 corrupted:0)
	Buffered ranges: [(844.911, 845.119455)]
		SourceBuffer 0
			start=844.911 end=845.119455
		SourceBuffer 1
			start=842.024666 end=844.542
	Internal Data:
	audio decoder: shutdown
	audio frames decoded: 0
	audio state: ni=0 no=0 ie=0 demuxr:0 demuxq:0 decoder:0 tt:-1.000000 tths:-1 in:0 out:0 qs=0 pending:0 waiting:1 sid:4294967295
	video decoder: shutdown
	hardware video decoding: disabled
	video frames decoded: 0 (skipped:0)
	video state: ni=0 no=0 ie=0 demuxr:0 demuxq:0 decoder:0 tt:-1.000000 tths:-1 in:0 out:0 qs=0 pending:0 waiting:0 sid:4294967295
	Dumping data for demuxer 7fedfab60c00:
		Dumping Audio Track Buffer(audio/mp4a-latm): - mLastAudioTime: 842.024666
			NumSamples:118 Size:79014 NextGetSampleIndex:0 NextInsertionIndex:118
			Buffered: ranges=[(842.024666, 844.542000)]
		Dumping Video Track Buffer(video/avc) - mLastVideoTime: 844.944000
			NumSamples:6 Size:15758 NextGetSampleIndex:1 NextInsertionIndex:6
			Buffered: ranges=[(844.911000, 845.119455)]

here you can see that your audio buffered range is 842.02-844.54, while the video is 844.911-845.11
that gives an intersection of [(844.911, 845.119455)] that's around .2s of playable data. not much!

If you're missing data, appending them will automatically change the mediasource readyState from "ended" to "open"

If you believe the specs are wrong, then please open a bug there. Maybe the spec could be changed so that the seekable range is also the buffered range when the readyState is ended

Comment 2

[Jean-Yves Avenard [:jya]]

BTW, Chrome/Blink buffered range calculation in MSE is wrong, you can track the problem and how they are about to fix it there: https://bugs.chromium.org/p/chromium/issues/list?q=label:MSEptsdtsCleanup

Comment 3

[CoolCmd]

I found in spec why seek may fail: https://w3c.github.io/media-source/index.html#mediasource-seeking
>If one or more of the objects in activeSourceBuffers is missing media segments for the new playback position...

And I found in spec why HTMLMediaElement.buffered is not empty: https://w3c.github.io/media-source/index.html#htmlmediaelement-extensions
>4.2 If readyState is "ended", then set the end time on the last range in source ranges to highest end time.
Really strange behavior...

Jean-Yves Avenard, thanks for 125ms. spec now says that gap threshold is implementation-specific, so this info can be useful.

Comment 4

[CoolCmd]

i think, workaround is simple: check HTMLMediaElement.buffered before endOfStream(), not after.

Comment 5

[Jean-Yves Avenard [:jya]]

(In reply to CoolCmd from comment #3)
> I found in spec why seek may fail:
> https://w3c.github.io/media-source/index.html#mediasource-seeking
> >If one or more of the objects in activeSourceBuffers is missing media segments for the new playback position...

As you will see, we are exactly conforming to the spec.

> 
> And I found in spec why HTMLMediaElement.buffered is not empty:
> https://w3c.github.io/media-source/index.html#htmlmediaelement-extensions
> >4.2 If readyState is "ended", then set the end time on the last range in source ranges to highest end time.
> Really strange behavior...

It can be weird at first, but this just ensures that you can play videos to the end so that currentTime can reach duration.
This makes me realise that we should allow seeking to the end of the file even if we have no audio or video if attempting to seek between the track highest end time and duration, if the mediasource is ended; this is not 100% spec conformant, but IMHO we should be able to always successfully seek where the buffered range attribute reports that we have content. I will lodge a MSE bug for that too.

> 
> Jean-Yves Avenard, thanks for 125ms. spec now says that gap threshold is
> implementation-specific, so this info can be useful.

We are thinking of increasing this value to 500ms so that you don't get unexpected stalls due to bad content.

If you find anything that is not per spec with our MSE implementation, please do not hesitate to report it.

You may find that you should test your code with Firefox first, as it's likely that if it works there, it will work with all other browsers; plus that guarantees that your code is properly using the MSE specs as defined.

Comment 6

[Jean-Yves Avenard [:jya]]

I lodged https://github.com/w3c/media-source/issues/147