Closed Bug 1237629 Opened 9 years ago Closed 9 years ago

Media source extensions fail to play a WebM file with BlockGroup elements

Categories

(Core :: Audio/Video: Playback, defect, P2)

Product:
Core
Component:
Audio/Video: Playback
Version:
46 Branch
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla46
Tracking Flags:
Tracking Status
firefox46 --- fixed

People

(Reporter: rudolfs.bundulis, Assigned: jya)

References

Details

Attachments

(2 files)

Image of the hex contents of the faulty webm file demonstrating that the cluster has a valid length
21.76 KB, image/png
Details
[webm] Parse into BlockGroup element.
1.95 KB, patch
kinetik
: review+
Details | Diff | Splinter Review
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36 Steps to reproduce: I tried to load this webm file through media source extensions by using the following html markup: <html> <head> <script src="https://code.jquery.com/jquery-2.1.4.min.js"></script> <script type="text/javascript"> var video; function play() { var webmSupported = MediaSource.isTypeSupported("video/webm; codecs=\"vorbis, vp8\""); console.log("webm supported: " + webmSupported); var conentType = "video/webm; codecs=\"vp8\""; var url = '/arch.webm'; if (MediaSource.isTypeSupported(conentType)) { var mediaSource = new MediaSource(); video.src = URL.createObjectURL(mediaSource); mediaSource.addEventListener('sourceopen', function() { var sourceBuffer = mediaSource.addSourceBuffer(conentType); sourceBuffer.addEventListener('error', function() { console.log('error:' + sourceBuffer.error); }); var xhr = new XMLHttpRequest; xhr.open('get', url); xhr.responseType = 'arraybuffer'; xhr.onload = function () { sourceBuffer.addEventListener('updateend', function () { if (mediaSource.readyState == 'open') { console.log("signalling end of stream"); mediaSource.endOfStream(); } }); console.log("Loaded a chunk of " + xhr.response.byteLength + " bytes"); var chunk = new Uint8Array(xhr.response); sourceBuffer.appendBuffer(chunk); }; xhr.send(); }); } } $(function() { video = document.getElementById("video"); video.addEventListener('loadedmetadata', function() { console.log('loadedmetadata'); console.log("video is " + video.duration + " seconds long"); }); video.addEventListener('paused', function() { console.log('paused'); }); video.addEventListener('seeking', function() { console.log('seeking'); }); video.addEventListener('loadeddata', function() { console.log('loadeddata'); }); video.addEventListener('canplay', function() { console.log('canplay'); console.log("video is " + video.duration + " seconds long"); console.log("buffered ranges: " + video.buffered.length); for (var bufferedRange = 0; bufferedRange < video.buffered.length; ++bufferedRange) { console.log(bufferedRange + ": from " + video.buffered.start(bufferedRange) + " to " + video.buffered.end(bufferedRange)); } video.play(); }); video.addEventListener('timeupdate', function() { console.log("playback time is " + video.currentTime); }); play(); }); </script> </head> <body> <video id="video"></video> </body> </html> This markup works fine on Chrome and the file plays fine with VLC, ffplay and if simply dragged into Firefox in an open tab. But when using MSE, a duration of 0 seconds is reported and nothing happens. I tried to debug and found possible pitfalls but not sure which one actually causes this. 1) ContainerParser.cpp(184): if (mLastMapping && (initSegment || IsMediaSegmentPresent(aData))) { // The last data contained a complete cluster but we can only detect it // now that a new one is starting. // We use mOffset as end position to ensure that any blocks not reported // by WebMBufferParser are properly skipped. What does this mean - if I generate a fragment which is short enough to fit in a single cluster it will never be detected when using MSE (dragging the file in an open tab does not hit this code path)? 2) WebMBufferedParser.cpp - the method WebMBufferedParser::Append does not look for BlockGroup elements and thus it will also skip the Block elements inside this (indeed the same file but generated with SimpleBlock elements instead of BlockGroup/Block elements works). Actual results: Nothing was played. Expected results: The video should be played.
Component: Untriaged → Audio/Video: Playback
Product: Firefox → Core
Since the file is 21 MBs here is a link to Google drive for the faulty file - https://drive.google.com/file/d/0BxZxYDtgrhYKWkxYRkZBbWctSjg/view?usp=sharing
Jean-Yves, can you take a look and see if this is by spec or not?
Flags: needinfo?(jyavenard)
Priority: -- → P2
Can you provide a URL that doesn't work?
This will only happen if your webm is encoded such that a cluster doesn't have a size explicitly set (eg. size is -1). Which per spec means it ends when a new one start. So it doesn't matter if your cluster is small: if there's only one it just needs a size. YouTube's is currently the only site I know using webm/vp9 and all their clusters have a size. To me, our behaviour is correct and is per spec. http://w3c.github.io/media-source/index.html#sourcebuffer-segment-parser-loop describe on what to do when we receive a media segment: here is the description of what a media segment is: http://w3c.github.io/media-source/webm-byte-stream-format.html#webm-media-segments "A WebM media segment is a single Cluster element." "The Cluster header may contain an "unknown" size value. If it does then the end of the cluster is reached when another Cluster header or an element header that indicates the start of an WebM initialization segment is encountered." So if your cluster has an unknown size, and there's only one cluster: that particular cluster doesn't have an end: it is as such incomplete. I would be keen as closing this bug as invalid. However, there's probably work around we could do, even though not required per spec for such content.
Assignee: nobody → jyavenard
Flags: needinfo?(jyavenard)
Hi Anthony, I pasted the link to the file and the markup to reproduce this issue above - sorry that I can't provide a working url, if you can point me to a free popular hosting site I can set it up, but it should be trivial to host it locally on Apache. Hi Jean-Yves, did you look at the file I shared? It has a single cluster with a specified size, so the case of a cluster with an unknown length is not relevant here. At least as far as I debugged the code it seems that for some reason Block elements are not seen by the parser. I have attached a screenshot from Notepad++ hex edit mode to demonstrate that the size is fine. It is set as 0x08015d4dbe, taking of the b00001xxx mask it gives 0x15d4dbe and the size ends at offset 0x73, and file ends at 0x15d4e31 which equals 0x73 + 0x15d4dbe, so the size should be fine.
I found a workaround for this if I mix Block elements with SimpleBlock elements - Chrome had an issue that it could not translate timestamps across clusters that had only SimpleBlock elements so I started to terminate each Cluster with a Block and this approach seems to make both Firefox and Chrome happy. However, I'd still say this is a bug in Firefox, since the provided WebM should be valid - it has all sizes, timings etc.
See Also: → 1240201
what did you use to generate this webm ???
The file is muxed by my own code, the company I work for is developing a html5 based video streaming solution and the file was generated using the first protoype for that. Why is that relevant?
The issue appears to be the use of BlockGroup which our parser do not handle.. So I was just curious on what produced this file, seeing that our webm support is pretty old, but it BlockGroup had never been a problem. So obviously we will fix it, but in the mean time, you may want to use SimpleBlock instead.
Excuse me if I seemed impolite with the "relevant" sentence - I was just afraid this would get closed since I am not using ffmpeg or libwebm:) As I said with only SimpleBlock elements I get issues with Chrome, but I have found a compromise - using all SimpleBlocks and one BlockGroup for the last frame, this seems to work and even if I loose the last frame on Firefox that is not such a big issue.
(In reply to rudolfs.bundulis from comment #11) > Excuse me if I seemed impolite with the "relevant" sentence - I was just > afraid this would get closed since I am not using ffmpeg or libwebm:) > > As I said with only SimpleBlock elements I get issues with Chrome, but I > have found a compromise - using all SimpleBlocks and one BlockGroup for the > last frame, this seems to work and even if I loose the last frame on Firefox > that is not such a big issue. What issues would you get with Chrome? Most of our regression tests are using webm with SingleBlock or plain Block ; that's also what YouTube is using. I've never experienced problem with chrome there. Are you certain the issue isn't with what you're generating? I'd be very surprised if there was a problem in chrome when it comes to webm
We could have more detailed parsing into the BlockGroup and read the information applicable to the blocks it contains (like duration). We could also ensure as per the spec that if there are blocks contained within the BlockGroup they all have the same timecode. But for now, simply scanning into it is all that is required. Much easier that way and can't cause regressions.
(In reply to rudolfs.bundulis from comment #11) > and even if I loose the last frame on Firefox > that is not such a big issue. that shouldn't be the case if all elements have a size. Could you please open another bug and provide a sample? Thank you
Attachment #8708955 - Flags: review?(kinetik)
> What issues would you get with Chrome? https://code.google.com/p/chromium/issues/detail?id=574760 I was using SimpleBlock elements with non monotonic timestamps and also provided a DefaultDuration and that made Chrome divide the video in fragments. But that was more or less my bad since I had misinterpreted the meaning of the DefaultDuration field. That is why I tried to switch to BlockGroup elements and use durations there, but then I ran into this. But eventually I managed to sort it out as explained above. > Most of our regression tests are using webm with SingleBlock or plain Block > ; that's also what YouTube is using. Maybe I misunderstood the specification - but can you use Block without BlockGroup? From the spec (http://www.matroska.org/technical/specs/index.html) Block is a level 3 element, and BlockGroup is a level 2 element (which at lest IMHO implies that a BlockGroup is always a parent of a Block). How would one use plain Block elements? > and even if I loose the last frame on Firefox > that is not such a big issue. Before bugging you further I'll check the duration of that video on Chrome and Firefox and make sure I really do loose the last frame, and if so I'll open a bug.
(In reply to rudolfs.bundulis from comment #15) > > What issues would you get with Chrome? > > https://code.google.com/p/chromium/issues/detail?id=574760 > > I was using SimpleBlock elements with non monotonic timestamps and also > provided a DefaultDuration and that made Chrome divide the video in but that is invalid. timestamps in webm *must* be monotonic.
> timestamps in webm *must* be monotonic. Maybe is have overlooked something, but is there any place in the specification that states so? My particular use case is a stream for IP camera, which can be recorded when being triggered by motion and other alarms. In that case I can have lets say a video segment that is 60 seconds long but actually has lets say 3 video blocks each of which is 10 seconds with some gaps between them. And in that case of course the time stamp intervals at the end of those blocks will not be monotonic. So far it seems that this actually works on both browsers.
A gap doesn't prevent timestamps to be monotonic: monotonically increasing: timestamps never go back. The spec is on the matroska website. There's also a few bugs that have been lodged here and closed. With some older version of Firefox it will assert, in newer ones either the data won't be added to the source buffer or you'll get a decoding error.
Yes - excuse me I did not make myself clear :D I am never going backwards - what I mean with "monotonic" (which was an improper use of this term) was that the intervals among frames are not of a constant size - which I guess is not a very popular use case. Sorry for misleading. The issue was that Chrome could not handle this and split the webm file in multiple buffered randges and worst of all - it did not emit pause or something else to let me know that a fragment had ended. But yeah - removing DefaultDuration and adding Block at the end of the cluster solved this. But on the subject of Block elements - still, how should I format the stream to use BlockElements without BlockGroup.
https://w3c.github.io/media-source/webm-byte-stream-format.html#webm-media-segments "Block & SimpleBlock elements are in time increasing order consistent with the WebM spec." http://www.webmproject.org/docs/container/ "absolute (block + cluster) timecodes MUST be monotonically increasing. All timecodes are associated with the start time of the block."
Multiple buffered range if you have gaps, seem the right thing to do... What else did you expect? Under some circumstances, you may have a frame that has the duration of the frame+gap May I ask why you are using webm over MP4? MP4 is supported by all browsers, that's not the case with webm. We have plans to support vp9 or opus within MP4 container.
Summary: Media source extensions fail to play a WebM file with a single cluster and block elements → Media source extensions fail to play a WebM file with BlockGroup elements
Well I mean, I set the durations of the last frames to "frame+gap" like you said, but Chrome still divided the whole video in multiple ranges even though there were actually no breaks. I mean I would not even mind that if only after calling play() Chrome would give me a paused() event to know that the first range has been played, but that did not happen. But as I stated - this was fixed with the introduction of the Block element. > May I ask why you are using webm over MP4? The source video is in MPEG4 (which is sadly not supported, if it was we would not use webm). We can't encode that into H.264 since H.254 is a proprietary codec and we do not wish to purchase licenses, so here VP8 comes into play. Also, in a weird way, iOS does not support fragmented mp4 but it can play webm files in Safari (go figure). But yeah, for the content that is already encoded with H.264 we are streaming MP4. I actually reported a bug whece Firefox incorrectly reads base data offset and submitted a patch ( https://bugzilla.mozilla.org/show_bug.cgi?id=1235427 ). We had other issues with MP4 on Chrome so from my experience I'd say that MP4 is not much more stable - there still are quirks to be found. I'm hope I'm not spamming to much but for example we fought quite long to find out that Firefox requires the tfdt box which was not needed for Chrome (and VLC does not even recognize it) - the ISO spec I read was from 2005 and the tfdt was introduced in later amendments. And the worst thing is that when tfdt comes into play I cannot share the same tcp data (as in zero memcpy) among multiple clients (which I could do without it) since the ones that connect later and get a tfdt that is not 0 start to buffer - now I can share only the mdat payload but need to generate individual moof headers for each client.
Attachment #8708955 - Flags: review?(kinetik) → review+
https://hg.mozilla.org/mozilla-central/rev/985507e90616
Status: UNCONFIRMED → RESOLVED
Closed: 9 years ago
status-firefox46: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: