Media Source Extensions appendBuffer of WebM stream in random order
Categories
(Core :: Audio/Video: Playback, defect)
Tracking
()
People
(Reporter: darin.dimitrov, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(2 obsolete files)
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36
Steps to reproduce:
I am trying to play a fragmented webm video using Media Source Extensions. If I append the segments sequentially, then everything works, but if I append them in random order, Firefox seems to ignore already appended frgaments.
I created a sample repro here: https://jsfiddle.net/k65s8zwg/
Here are the steps to reproduce the issue:
- Start the video
- Immediately after starting forward towards the end of the video so that the last segment is fetched and appended to the source buffer
- After the segment is buffered rewind back to the middle of the video
The middle segment will play until it reaches the point in time of the last segment and then it will stop as if the last segment was never buffered.
Tested this behaviour in Chrome, Opera and Brave and the video seems to play without interruption.
Actual results:
The buffer that was already appended is ignored
Expected results:
The video plays smoothly
|
||
Comment 1•4 years ago
|
||
The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.
|
||
Comment 2•4 years ago
|
||
I think I can reproduce that by (1) seeking to around 28s (2) seeking to the middle (3) then the playback would stop because of lacking of data. I will check this issue later, set NI on myself.
Looks like the buffered ranges have discontinuities when seeking (you'll need to switch your dev tools to the iframe to run the code below, see https://developer.mozilla.org/en-US/docs/Tools/Working_with_iframes).
If I let the video play through and run the following in the console
{
let buffered = $('video').buffered;
for (let i = 0; i < buffered.length; i++) {console.log(`Range: ${buffered.start(i)} - ${buffered.end(i)}`);}
}
I get Range: 0.003 - 30.544
If I seek I see results like
Range: 0.003 - 25.604
Range: 29.87 - 30.544
so we're getting discontinuities.
I can get the same result in Chromium based browsers if I jump forward then scrub back:
Range: 0 - 8.528
Range: 25.594 - 30.036
Edit: for clarificaiton, I also see Chromium stall in some of these cases. Depending on where I scroll to it may load the next segment, but it will also get stuck at times.
Not sure exactly why, but I suspect this is a case where the JS is not handling all the different buffering contingencies.
Thanks for looking into this issue. Does this mean that when I use appendBuffer, there shouldn’t be any discontinuities between the current video runtime and the added segment timestamps? If this is the case, if the user starts the video and immediately seeks to the end, do I need to load the entire video buffer?
I'd need to look more to give an exact answer, I wanted to add some quick thoughts to the bug to help with investigation, but I won't be able to look more today.
There can be discontinuities, but we need to have media buffered at the time we're trying to play back. So I should have said "it looks like some data is missing that we need after seeking back, I would expect the page to buffer here rather than having a discontinuity." My guess would be that there's an issue with detecting when needing to buffer in JS, but that is just a guess.
Browsers are also allowed to evict data from the buffers, I don't think that's what's happening here, but I can't be sure without more debugging. This means that even if data is buffered at some point, it may not be later. So the player logic needs to accommodate that.
Well, the thing is that if I load all the previous segments up to the point we are seeking, then it works fine. But this is exactly what I wanted to avoid as it would be inefficient.
When we seek towards the end of the video we can see the last segment buffered up to the end, and when we seek back, we can see this last segment discarded, and only the current segment being buffered (as if the last segment was never buffered).
You shouldn't need to load all the segments up to the current seek, but if you wish to debug further I'd verify
- That data needed to play back after a seek is being appended -- stalling after seeks suggests it's not.
- That data is rebuffered if evicted as needed. If you seek to the end and buffer that data, then seek elsewhere and that data is evicted, it will need to be added back in.
|
|
||
Comment 8•4 years ago
|
||
Using a non-null image can prevent element from changing its ready state incorrectlt to HAVE_METADATA. See more detailed analysis in [1].
|
|
||
Comment 9•4 years ago
|
||
|
|
||
Updated•4 years ago
|
|
||
Comment 11•4 years ago
|
||
Comment on attachment 9219597 [details]
Bug 1707846 - part1 : use empty image in null video data.
Revision D114005 was moved to bug 1707242. Setting attachment 9219597 [details] to obsolete.
|
|
||
Comment 12•4 years ago
|
||
Comment on attachment 9219598 [details]
Bug 1707846 - part2 : add test.
Revision D114006 was moved to bug 1707242. Setting attachment 9219598 [details] to obsolete.
|
|
||
Comment 13•4 years ago
•
|
||
I did some investigation and I think that is JS player's issue. As Bryce said in comment3, there was a gap between 25.604 and 29.87, and that missing part was removed by MSE spec, so the JS player should be responsible to add them back.
Let's go through the process steps by steps,
- seeked to near the end and got the buffered range like that
Video : (25.603000, 29.869000), (29.870000, 30.036000)
Audio : (25.594000, 30.544000)
- seeked to the place around the middle part of the video, and we would got the new buffeded range
Video : (12.803000, 17.069000), (25.603000, 29.869000), (29.870000, 30.036000)
Audio : (12.794000, 17.060000), (25.594000, 30.544000)
When appending the data that belongs to the range of 12s to 17s, we detected the discontinuty that caused us to reset the state per the spec step 6.
So here we reset the highest end timestamp for track buffer, which was 30.036000 for video and 30.544000 for audio, and that would be updated later to the new end of the new coded frames group, which was 17.069000 for video and 17.060000 for audio.
- Video was continually playing, then something happened
When JS player started appending the data which range is 21.336000, 25.604000 that overlapped with the previous data (25.603000, 29.869000) that had been appended. Then triggered a process of removing frames.
Per the spec step14,
If highest end timestamp for track buffer is set and less than or equal to presentation timestamp,
=> Remove all coded frames from track buffer that have a presentation timestamp greater than or equal to highest end timestamp and less than frame end timestamp
According to the step2, the highest end timestamp had been reset before, so currently the highest end timestamp was 25.604000. All coded frames have a presentation timestamp greater than or equal to 25.604000 would be removed.
Description
•