881512 - Implement enough of MSE to run the YouTube MSE player

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Matthew Gregan [:kinetik]]

At a minimum this requires support for multiple decoders and separate audio and video tracks.

Comment 2

[avada]

Now that all depending bugs are fixed is this far off?
How come https://www.youtube.com/html5 shows all green except for "MSE & H.264"? Isn't the same system h.264 codec available with MSE?

Comment 3

[Feross Aboukhadijeh]

Can we get an update on MediaSource in Firefox? ETA for release?

Comment 4

[alex_mayorga]

(In reply to Feross Aboukhadijeh from comment #3)
> Can we get an update on MediaSource in Firefox? ETA for release?

From https://wiki.mozilla.org/Platform/MediaSourceExtensions :

"Timeline wise we are hoping to support enough of the spec to support the YouTube production player by the end of the year but will need to collaborate with YouTube to ensure that we're focussing on the right things. We also expect that dash.js compatibility will require a similar feature set."

Comment 5

[avada]

That's what it said last year already... So actually youtube should work by now. (It doesn't)

Comment 6

[owpard]

(In reply to Feross Aboukhadijeh from comment #3)
> Can we get an update on MediaSource in Firefox? ETA for release?

It's still a work in progress. Bug 976037 comment 1 gives an update on what is playable and what isn't. Basically some 'DASH in JavaScript' implementations are working but the YouTube demo player is not yet.

Comment 7

[Matthew Gregan [:kinetik]]

Attachment: bug881512_p1_v0.patch

First patch in a series, more coming soon.

Comment 8

[Matthew Gregan [:kinetik]]

Attachment: p1: Add start (and rename end) time helper to TimeRanges.

Comment 9

[Matthew Gregan [:kinetik]]

Attachment: p2: Allow a SourceBuffer to own multiple subdecoders. Switch decoders on abort().

Comment 10

[Matthew Gregan [:kinetik]]

Attachment: p3: Hide MediaSourceResource implementation details from HTMLMediaElement.

Comment 11

[Matthew Gregan [:kinetik]]

Attachment: p4: Add hidden pref to ignore MIME type in AddSourceBuffer and IsTypeSupported. For experimental/debugging use only.

Comment 12

[Matthew Gregan [:kinetik]]

Attachment: p5: Start processing multiple decoders.

Comment 13

[Matthew Gregan [:kinetik]]

Attachment: p6: Switch video readers on EOF.

Comment 14

[Matthew Gregan [:kinetik]]

Attachment: p7: Crude attempt at segment parser for WebM.

Comment 15

[Matthew Gregan [:kinetik]]

The r? patches are ready, pending any review comments.  The f? patches have at least one known bug, so I expect to post updated versions soon.  There's probably at least one more patch to come, too.

Comment 16

[cajbir (:cajbir)]

Comment on attachment 8406660 [details] [diff] [review]
p7: Crude attempt at segment parser for WebM.

Review of attachment 8406660 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/media/mediasource/SourceBuffer.cpp
@@ +111,5 @@
>    int64_t offset = (aTime / (double(mMediaDuration) / USECS_PER_S)) * length;
>    return offset;
>  }
>  
> +class ContainerParser {

Does this need a virtual destructor?

Comment 17

[Matthew Gregan [:kinetik]]

Attachment: p5: Start processing multiple decoders.

Comment 18

[Matthew Gregan [:kinetik]]

Attachment: p6: Switch video readers on EOF.

Comment 19

[Matthew Gregan [:kinetik]]

Attachment: p7: Crude attempt at segment parser for WebM.

Comment 20

[Matthew Gregan [:kinetik]]

Attachment: p8: Unpref Media Source Extensions (for WebM) in non-release builds.

Comment 21

[Matthew Gregan [:kinetik]]

Attachment: p9: Temp fix: don't call mParentDecoder functions while SBD runs off decode thread. Revert when SBD runs on state machine's task queue.

Comment 22

[Matthew Gregan [:kinetik]]

Attachment: p10: Don't pretend the entire resource is always completely cached.

Comment 23

[Matthew Gregan [:kinetik]]

Attachment: p11: Use the base video and audio queues. Currently works by shifting media data from the sub-reader queues on each decode call. Future work could allow the sub-readers to use the base queues directly.

Comment 24

[Matthew Gregan [:kinetik]]

Attachment: p12: Remove useless static_casts.

Comment 25

[Matthew Gregan [:kinetik]]

Attachment: p13: Don't mix refcounting with auto ptr use. Rather than try to hang on to the reader, punch the appropriate holes through via the state machine class.

Comment 26

[Matthew Gregan [:kinetik]]

Attachment: p14: Delay decoder initialization by indicating that we're waiting for subdecoders to be created.

Comment 27

[Matthew Gregan [:kinetik]]

Attachment: p15: Fix bogus indexing of audio/video readers by ditching audio/video reader arrays and using the parent decoder directly.

Comment 28

[Matthew Gregan [:kinetik]]

Attachment: p16: Explicitly initialize ReadMetadata's aTags parameter to null. Also add some null deref tests.

Comment 29

[Matthew Gregan [:kinetik]]

Comment on attachment 8410099 [details] [diff] [review]
p8: Unpref Media Source Extensions (for WebM) in non-release builds.

Obsoleting pref change patch (p8), it has been moved to bug 1000686.

Comment 30

[Matthew Gregan [:kinetik]]

Try push (also includes seeking patches from bug 1000608): https://tbpl.mozilla.org/?tree=Try&rev=533fd846f2b7

Comment 31

[Matthew Gregan [:kinetik]]

Attachment: p17: Make sure any released decoders are destroyerd on the main thread.

Comment 32

[Matthew Gregan [:kinetik]]

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=d75260ca4ed2

	44ad5896ff61	Matthew Gregan — Bug 881512 - Make sure any released decoders are destroyed on the main thread. r=cajbir
	15fe0db2ff8d	Matthew Gregan — Bug 881512 - Explicitly initialize ReadMetadata's aTags parameter to null. Also add some null deref tests. r=cajbir
	81403176036e	Matthew Gregan — Bug 881512 - Fix bogus indexing of audio/video readers by ditching audio/video reader arrays and using the parent decoder directly. r=cajbir
	6c69283f7713	Matthew Gregan — Bug 881512 - Delay decoder initialization by indicating that we're waiting for subdecoders to be created. r=cajbir
	1eddd13262de	Matthew Gregan — Bug 881512 - Don't mix refcounting with auto ptr use. Rather than try to hang on to the reader, punch the appropriate holes through via the state machine class. r=cajbir
	5fbc68a34f62	Matthew Gregan — Bug 881512 - Remove useless static_casts. r=cajbir
	199c040d09f0	Matthew Gregan — Bug 881512 - Use the base video and audio queues. Currently works by shifting media data from the sub-reader queues on each decode call. Future work could allow the sub-readers to use the base queues directly. r=cajbir
	c116a4499d82	Matthew Gregan — Bug 881512 - Don't pretend the entire resource is always completely cached. r=cajbir
	5ea033583b20	Matthew Gregan — Bug 881512 - Temp fix: don't call mParentDecoder functions while SBD runs off decode thread. Revert when SBD runs on state machine's task queue. r=cajbir
	a14f9763bf8f	Matthew Gregan — Bug 881512 - Crude attempt at segment parser for WebM. r=cajbir
	cf3e31d25944	Matthew Gregan — Bug 881512 - Switch video readers on EOF. r=cajbir
	59ffec98e96f	Matthew Gregan — Bug 881512 - Start processing multiple decoders. r=cajbir
	18754facd070	Matthew Gregan — Bug 881512 - Add hidden pref to ignore MIME type in AddSourceBuffer and IsTypeSupported. For experimental/debugging use only. r=cajbir
	f8ece490919d	Matthew Gregan — Bug 881512 - Hide MediaSourceResource implementation details from HTMLMediaElement. r=cajbir
	d978fe16633a	Matthew Gregan — Bug 881512 - Allow a SourceBuffer to own multiple subdecoders. Switch decoders on abort(). r=cajbir
	c570bcfd29b4	Matthew Gregan — Bug 881512 - Add start (and rename end) time helper to TimeRanges. r=cajbir

Comment 33

[Matthew Gregan [:kinetik]]

Attachment: p18: Use MediaTaskQueue to run MSE tasks rather than a per-MSE thread.

Comment 34

[Matthew Gregan [:kinetik]]

Comment on attachment 8413148 [details] [diff] [review]
p18: Use MediaTaskQueue to run MSE tasks rather than a per-MSE thread.

p18 was planned for a near future cleanup in a different bug, but it turned out to be needed to fix an orange on TBPL debug builds:

3587 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/mediasource/test/test_MediaSource.html | Assertion count 1 is greater than expected range 0-0 assertions.

...which didn't show up on try because the same test failed for a different (innocuous) reason.

Comment 35

[Matthew Gregan [:kinetik]]

Landed p18: https://hg.mozilla.org/integration/mozilla-inbound/rev/30b52597454b

Comment 36

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c570bcfd29b4
https://hg.mozilla.org/mozilla-central/rev/d978fe16633a
https://hg.mozilla.org/mozilla-central/rev/f8ece490919d
https://hg.mozilla.org/mozilla-central/rev/18754facd070
https://hg.mozilla.org/mozilla-central/rev/59ffec98e96f
https://hg.mozilla.org/mozilla-central/rev/cf3e31d25944
https://hg.mozilla.org/mozilla-central/rev/a14f9763bf8f
https://hg.mozilla.org/mozilla-central/rev/5ea033583b20
https://hg.mozilla.org/mozilla-central/rev/c116a4499d82
https://hg.mozilla.org/mozilla-central/rev/199c040d09f0
https://hg.mozilla.org/mozilla-central/rev/5fbc68a34f62
https://hg.mozilla.org/mozilla-central/rev/1eddd13262de
https://hg.mozilla.org/mozilla-central/rev/6c69283f7713
https://hg.mozilla.org/mozilla-central/rev/81403176036e
https://hg.mozilla.org/mozilla-central/rev/15fe0db2ff8d
https://hg.mozilla.org/mozilla-central/rev/44ad5896ff61
https://hg.mozilla.org/mozilla-central/rev/30b52597454b

Comment 37

[Masatoshi Kimura [:emk]]

YouTube still says that the browser does not support Media Source Extensions even with the patched build. Is it YouTube's issue?

Comment 38

[Matthew Gregan [:kinetik]]

The functionality is hidden behind a pref for now; opening that up (for WebM only) is bug 1000686.

Comment 39

[AnAkkk]

With both media.mediasource.enabled and media.mediasource.ignore_codecs set to true, youtube videos just play for a few seconds and skip to the end, not sure if this is normal.
With media.mediasource.enabled set to true and media.mediasource.ignore_codecs set to false, they seem to play normally but only 360p/720p is available, which I guess is normal.

I'm using the latest nightly 31.0a1 (2014-04-27) on Arch Linux.

Comment 40

[cajbir (:cajbir)]

(In reply to AnAkkk from comment #39)
> With both media.mediasource.enabled and media.mediasource.ignore_codecs set
> to true, youtube videos just play for a few seconds and skip to the end, not
> sure if this is normal.
> With media.mediasource.enabled set to true and
> media.mediasource.ignore_codecs set to false, they seem to play normally but
> only 360p/720p is available, which I guess is normal.

If you right click on the video, choose "stats for nerds", do you see it using the Dash player and WebM?

Comment 41

[AnAkkk]

With media.mediasource.ignore_codecs = true, DASH:yes, video/mp4 and it plays for exactly 3 seconds before skipping to the end.
With media.mediasource.ignore_codecs = false, DASH:no, video/mp4 and it works fine.

Comment 42

[AnAkkk]

Sorry, it is only supposed to work on WebM videos?

Comment 43

[AnAkkk]

Nevermind. On some videos it does show DASH: yes, video/webm and I have the exact same issue.

Comment 44

[Brian Carpenter [:geeknik]]

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1002019 because setting media.mediasource.enabled to true and enabling YouTube's HTML5 player causes Nightly to freeze/hang/quit responding.

Comment 45

[cajbir (:cajbir)]

(In reply to AnAkkk from comment #42)
> Sorry, it is only supposed to work on WebM videos?

The current implementation was built and tested with WebM. It is planned to improve support to MP4. It may work already on some platforms or combinations of decoder but I wouldn't expect it.

Comment 46

[Matthew Gregan [:kinetik]]

(In reply to AnAkkk from comment #41)
> With media.mediasource.ignore_codecs = true, DASH:yes, video/mp4 and it
> plays for exactly 3 seconds before skipping to the end.

media.mediasource.ignore_codecs is for developer-only testing, please don't set it unless you have a good reason, and expect problems (but don't report them) when you do.

(In reply to AnAkkk from comment #43)
> Nevermind. On some videos it does show DASH: yes, video/webm and I have the
> exact same issue.

Please file a new bug and CC me.  Reporting new issues in a closed bug has a high probability of the issue being overlooked.

Comment 47

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/bc6a9d65b84f

Comment 48

[Matthew Gregan [:kinetik]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #47)
> https://hg.mozilla.org/mozilla-central/rev/bc6a9d65b84f

Oops, that's actually for bug 1000686, but I forgot to fix the commit message when splitting the bugs.

Comment 49

[AnAkkk]

(In reply to Matthew Gregan [:kinetik] from comment #46)
> Please file a new bug and CC me.  Reporting new issues in a closed bug has a
> high probability of the issue being overlooked.

This won't be necessary, my issue seem to be fixed with the latest nightly.

Comment 50

[Kostas]

In my case, (Nightly 32.0a1, BuildID=20140505030202 )
before setting media.mediasource.enabled to true, 3/6 are green in https://www.youtube.com/html5 :
"HTMLVideoElement", "H.264" and "WebM VP8".

After setting media.mediasource.enabled to true, 5/6 are green :
all except for "MSE & H.264".
Is this expected?
Shouldn't the "MSE & H.264" be green now too?

Comment 51

[Matthew Gregan [:kinetik]]

(In reply to Kostas from comment #50)
> Shouldn't the "MSE & H.264" be green now too?

MP4 (H.264, etc.) aren't supported in MSE yet.  We're concentrating on WebM on YouTube right now, other functionality will follow.

Comment 52

[Kostas]

(In reply to Matthew Gregan [:kinetik] from comment #51)
> (In reply to Kostas from comment #50)
> > Shouldn't the "MSE & H.264" be green now too?
> 
> MP4 (H.264, etc.) aren't supported in MSE yet.  We're concentrating on WebM
> on YouTube right now, other functionality will follow.
Thanks for the quick reply.

Comment 53

[user277536]

So, the target milestone for this bug - Firefox 31 has been released, but MSE on YouTube still doesn't work properly. Seeking sometimes takes a lot of time, seeking back doesn't work at all and repeat also doesn't work.
I tried the latest Nightly, but it seems that it only got worse - sometimes seeking just hangs.
Is bug 1024858 the only reason why it's still broken or there're some other YouTube related bugs?

Comment 54

[Matthew Gregan [:kinetik]]

Follow bug 1000686, which is tracking enabling this functionality by default in nightly builds.  This bug covered the initial implementation, but there are a number of bugs remaining which are being worked on now.

Comment 55

[Chris]

2 days ago youtube deprecated flash (now max 720p) and upped html5 functionality significantly, adding 60fps and super high resolutions to html5.

I think the slow approach to this needs changing I consider it now urgent to have fully working mp4 support on html5, the internet is plastered with "chrome only for 6ofps youtube"

It also works on IE11 in windows 8 although the internet is not making that clear its just saying it works on IE11 full stop but it does require windows 8.

Comment 56

[lars]

I just switched from Firefox to Nightly and I manually enabled MSE & h264 so I can use YouTube.

The two biggest problems I have with it right now are:

- Seeking doesn't work. Seeking within the 'buffered range' is slow and takes ~10 seconds. Seeking beyond the buffered range doesn't work at all.

- Videos with inVideo branding (example: https://srv.tuxplace.nl/hosted/screenshots/2014-11-03-13-36-00.png , icon in the top right corner) will freeze at the exact moment the icon will appear.

Comment 57

[DrDumbass]

I wanted to provide some help to anyone frustrated by this. Here is an article about a very helpful plugin that forces flash in HD to run in nightly instead of html5, but you can select between the two. http://www.ghacks.net/2014/10/20/force-flash-video-on-youtube-in-firefox/ and a link to the plugin http://barisderin.com/?p=2225 . For the time being, this makes everything run smoothly.

Comment 58

[daniel.2345]

(In reply to DrDumbass from comment #57)
> I wanted to provide some help to anyone frustrated by this. Here is an
> article about a very helpful plugin that forces flash in HD to run in
> nightly instead of html5, but you can select between the two.
> http://www.ghacks.net/2014/10/20/force-flash-video-on-youtube-in-firefox/
> and a link to the plugin http://barisderin.com/?p=2225 . For the time being,
> this makes everything run smoothly.

This comment is not quite related but I want to say using Flash was obsoleted because of slowness and lag. I'd rather see 320p than using Flash on my slow computer.

Comment 59

[Simon Howes]

(In reply to daniel.2345 from comment #58)
> (In reply to DrDumbass from comment #57)
> > I wanted to provide some help to anyone frustrated by this. Here is an
> > article about a very helpful plugin that forces flash in HD to run in
> > nightly instead of html5, but you can select between the two.
> > http://www.ghacks.net/2014/10/20/force-flash-video-on-youtube-in-firefox/
> > and a link to the plugin http://barisderin.com/?p=2225 . For the time being,
> > this makes everything run smoothly.
> 
> This comment is not quite related but I want to say using Flash was
> obsoleted because of slowness and lag. I'd rather see 320p than using Flash
> on my slow computer.

I'm on the opposite side. I have a fast laptop and I'm fed up being forced webm which doesn't use hardware acceleration. I'd much rather have Flash and h.264 with HW acceleration over webm eating battery life and burning my lap. Anyway, bugzilla isn't an Internet forum so let's end this.

Comment 60

[fireattack [:fireattack]]

Hi all, in bug #833023 Jan Gerber said that:

>> - http://www.youtube.com/html5 states that the latest Aurora does not
>> support "MSE & WebM VP9".
>For MSE support, check Bug 881512

And I came here, saw this bug is marked as "fixed", but in https://www.youtube.com/html5 it still shows "MSE & WebM VP9" not supported. (I'm using latest Nightly) 

How could I fix this or it has not been implemented into Firefox yet?

Comment 61

[Masatoshi Kimura [:emk]]

It is temporarily disabled (see bug 1097436 and bug 1108059).

Comment 62

[Chris]

On win7 youtubne reports all codecs available (without ignore_codecs) but when youtube tries to start playing it spins for ages and eventually falls back to non 60fps codec.

beta testers are reporting its been working fine and are astonished its not in the release version

so whast the issue with ff36?

code removed?
not working yet on windows?