1138992 - media source extension is not playing

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P5)

Description

[guy paskar]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36

Steps to reproduce:

go to http://download.tsi.telecom-paristech.fr/gpac/mp4box.js/
follow the instruction to play a video (make sure it has CORS) you may use this video
http://storage.googleapis.com/peer5_vod/bbb_ld.mp4




Actual results:

Nothing is playing. No errors, media is being appended.



Expected results:

Load the same page in Chrome and see what should happen.
It used to work in firefox too a while back.

Comment 1

[Ralph Giles (:rillian)]

MSE is only enabled for general testing in the developer and nightly channels, and for youtube-only in 37 beta. Does the page work for you in current Firefox 38 and 39?

Comment 2

[guy paskar]

Happens also on 39 (nightly) same issue.

On 37 it was with youtubeonly = false.

Again, it used to work a few weeks back.

btw, I read that 37 will have youtubeonly = false when it will be released to stable channel , is it true?

Comment 3

[Ralph Giles (:rillian)]

(In reply to guy paskar from comment #2)

> On 37 it was with youtubeonly = false.

Ok.

> btw, I read that 37 will have youtubeonly = false when it will be released
> to stable channel , is it true?

I don't think that's correct. Our implementation isn't complete yet. We haven't decided when to remove the youtube domain restriction on release, but I don't expect it will be before firefox 39 or 40.

Comment 4

[cyril.concolato]

I confirm that it doesn't work on Nightly 39.0a1 (2015-03-03) on Windows 8.1 64bits and that a similar demo was working some time ago. It may be a regression in mp4box.js in the way the media segments are generated but the fact that it works in Chrome is strange. I'd be happy to modify mp4box.js (if necessary) to make it work with FF.

Comment 5

[guy paskar]

(In reply to cyril.concolato from comment #4)
> I confirm that it doesn't work on Nightly 39.0a1 (2015-03-03) on Windows 8.1
> 64bits and that a similar demo was working some time ago. It may be a
> regression in mp4box.js in the way the media segments are generated but the
> fact that it works in Chrome is strange. I'd be happy to modify mp4box.js
> (if necessary) to make it work with FF.

Found anything?

Comment 6

[guy paskar]

Any news here?

Comment 7

[guy paskar]

Ping

Comment 8

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Does it work with media.mediasource.format-reader=true ?

Comment 9

[Jean-Yves Avenard [:jya]]

Several problems at play here:

1- You're not listening to the sourcebuffer nor media element error events.

2- The issue is in the MP4 init segment you're appending. The ctts is of version 1 (signed offset) which the stagefright demuxer doesn't support.

3- You use a single moof+mdat for every single video frame. This is highly inefficient, and will lead to subpar performance on most browsers (not including on how much bigger the data the JS append will actually be).

4- Every video frame is marked as a keyframe. This will likely lead to decoding error when seeking or visual artefacts.  Note that we don't properly handle media segment not starting with keyframe

Implementing 1) would have shown you the error in 2)

As a quick workaround for 2): being fragmented mp4; you do not use a sample table in the moov atom; its sample count is 0. You should exclude those atoms (ctts, stsc, stts, stsz, stco etc..) they serve no purpose.

Adding a workaround for item 2) the stream plays well ; however seeking on mac leads to the expected visual artefacts ; this is likely going to be hardware dependent, I'd assume you'll get decoding errors (Safari choke on those)

Comment 10

[Jean-Yves Avenard [:jya]]

Attachment: Ignore ctts version if we have 0 samples in the table.

The versioning is irrelevant when we have 0 samples in our table so we can ignore it instead.

Comment 11

[cyril.concolato]

Thanks for the detailed analysis.

About 1: when I started implementing mp4box.js, not much was reported by the error events, I had more information using tools as chrome://media-internals. I've just tried using the error events in Firefox on the media element and the sourcebuffer. I do get an error on the SourceBuffer, but without any specific information. Is there a way to get an equivalent of media-internals? 

About 2: I should probably not force using version 1 all the time, indeed. There are more and more files using that version, that's why I started using it. 

About 3: Sure. It was easier to generate this way, introduced no additional latency, and I did not see any performance issue so far. It's on my TODO list.

About 4: The current marking is because I have a problem when using Chrome if I use sample flags to mark only key frames, probably because of the problems you have with segments not starting with key frames. Hopefully, this should be fixed soon.

Regarding the use of the boxes ctts, stsc, stts, stsz, stco etc, the ISOBMFF spec mandates the presence of 'stsc', 'stts', 'stco'. They serve indeed no real purpose except for legacy systems like QuickTime I think. For ctts, the spec is ambiguous I think because it says it must be used if CT differs from DT. Since it documents only the initialization segment which contains no data, the comparison CT vs. DT does not make sense. I guess I could not put it. It's just that since it's present in the original file, it is emptied and copied in the output initialization segment.

Comment 12

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Can someone produce a segment we can check in as a unit test?

Comment 13

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Comment on attachment 8663475 [details] [diff] [review]
Ignore ctts version if we have 0 samples in the table.

Review of attachment 8663475 [details] [diff] [review]:
-----------------------------------------------------------------

r+ if you include a test

Comment 14

[Jean-Yves Avenard [:jya]]

(In reply to cyril.concolato from comment #11)
> About 3: Sure. It was easier to generate this way, introduced no additional
> latency, and I did not see any performance issue so far. It's on my TODO
> list.

It makes our new MSE performs very poorly unfortunately ; we implemented the spec to the letter. So when we have a moof with a single frame ; we re-create for every single frames a new demuxer, re-parse the init segment from 0 etc...
That's why you see on a debug build that firefox wait until you've reached 25% download to start playback (that's how long it takes to only treat the first 10s of video).

I'm not sure how we could make our implementation fast there and still be fully spec compliant.


> 
> About 4: The current marking is because I have a problem when using Chrome
> if I use sample flags to mark only key frames, probably because of the
> problems you have with segments not starting with key frames. Hopefully,
> this should be fixed soon.

Actually, upon double checking, not having the first frame of a moof be a keyframe is no long a problem (it was in version 41 and earlier)

So, so long that the keyframes are properly marked, you'll be fine.

It seems that chrome totally ignore that container information to detect keyframes and rely on the h264 internal data. (which they can do thanks to using ffmpeg). But we can't

Comment 15

[Jean-Yves Avenard [:jya]]

(In reply to cyril.concolato from comment #11)
> Thanks for the detailed analysis.
> 
> About 1: when I started implementing mp4box.js, not much was reported by the
> error events, I had more information using tools as
> chrome://media-internals. I've just tried using the error events in Firefox
> on the media element and the sourcebuffer. I do get an error on the
> SourceBuffer, but without any specific information. Is there a way to get an
> equivalent of media-internals? 

unfortunately, the only thing would be to run a debug build and enable some logging (we have a bug to provide JS message for when you have demuxing errors). The log would have told you exactly which bytes was wrong in that particular stream.


You don't see a video element error here ? only sourcebuffer ?
you should have :(

Comment 16

[cyril.concolato]

(In reply to Jean-Yves Avenard [:jya] from comment #14)
> (In reply to cyril.concolato from comment #11)
> > About 3: Sure. It was easier to generate this way, introduced no additional
> > latency, and I did not see any performance issue so far. It's on my TODO
> > list.
> 
> It makes our new MSE performs very poorly unfortunately ; we implemented the
> spec to the letter.  So when we have a moof with a single frame ; we
> re-create for every single frames a new demuxer, re-parse the init segment
> from 0 etc...
Where did you read that in the spec? 
According to [1], each moof indeed starts a new media segment, but the segment parser loop [2] does not force using a new demuxer nor parsing again the init segment ... You should just go from step 6 to step 4 because you're never in the  PARSING_INIT_SEGMENT state.

[1] https://w3c.github.io/media-source/isobmff-byte-stream-format.html#iso-media-segments
[2] http://w3c.github.io/media-source/#sourcebuffer-segment-parser-loop

Comment 17

[cyril.concolato]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #12)
> Can someone produce a segment we can check in as a unit test?

The "debug settings" tab of the MP4Box.js demo app has a checkbox that allows saving the segments that are being generated.

Comment 18

[Jean-Yves Avenard [:jya]]

(In reply to cyril.concolato from comment #16)
> Where did you read that in the spec? 
> According to [1], each moof indeed starts a new media segment, but the
> segment parser loop [2] does not force using a new demuxer nor parsing again
> the init segment ... You should just go from step 6 to step 4 because you're
> never in the  PARSING_INIT_SEGMENT state.

the spec doesn't have anything on even using a demuxer. If the samples are contiguous and monotonically increasing ; then we'll use a single demuxer during the entire operation.

however, as we deal with one moof at a time ; rather than working via a block of samples ; we must work on one sample at a time . And for each sample (rather than each block of samples) we have to search if it's going to override an existing frame, check where it needs to be inserted in the array of frames, insert them etc.. etc...

By grouping multiple frames in a single moof ; our code will run that many times faster.

The spec only describe on how to work frame by frame in the Coded Frame Processing algorithm http://w3c.github.io/media-source/index.html#sourcebuffer-coded-frame-processing

and this is what we do. However, we have optimised the common cases so the spec aren't completely unusable. And the common case is multiple samples per media segment. So rather than following the spec and working frame by frame, we only need to check the first and last frame and that they are contiguous. From that we only have to perform a single pass search and a O(1) insertion. But by forcing a single frame in a moof; you prevent any type of optimisation.

Your content will work mind you, it's just impossible to optimise.

Comment 19

[Jean-Yves Avenard [:jya]]

(In reply to cyril.concolato from comment #17)
> (In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #12)
> > Can someone produce a segment we can check in as a unit test?
> 
> The "debug settings" tab of the MP4Box.js demo app has a checkbox that
> allows saving the segments that are being generated.

I have a similar mode in Firefox, that will save all data ever appended to a sourcebuffer.

Comment 20

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3842384f730a
https://hg.mozilla.org/integration/mozilla-inbound/rev/178cdfac3315

Comment 21

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/3842384f730a
https://hg.mozilla.org/mozilla-central/rev/178cdfac3315