Open Bug 778617 (MSE) Opened 8 years ago Updated 1 year ago

[meta] Tracking bug for Media Source Extensions API implementation

Categories

(Core :: Audio/Video: Playback, defect, P2)

defect

Tracking

()

tracking-b2g +
Tracking Status
relnote-firefox --- 36+

People

(Reporter: kinetik, Unassigned)

References

(Depends on 33 open bugs, )

Details

(4 keywords, Whiteboard: [shumway] [dependency: marketplace-partners])

Attachments

(1 obsolete file)

This proposal extends HTMLMediaElement to allow JavaScript to generate media streams for playback. Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams.
Attached patch WebIDL/DOM stubs - patch v0 (obsolete) — Splinter Review
I plan to work on this in the near future, but I've got a couple of things to finish first.  In the mean time, I stubbed out the DOM side of this to give myself an excuse to play with the new WebIDL bindings.
Attachment #647050 - Attachment description: patch v0 → WebIDL/DOM stubs - patch v0
is there a bug filed for handling mediaStream decoding/decryption?
(In reply to David Dahl :ddahl from comment #2)
> is there a bug filed for handling mediaStream decoding/decryption?

You mean like the webm encryption stuff? Not that I'm aware of. Of course, the MediaSource API makes it possible to implement many scrambling and encryption methods in javascript.
And if you're referring to the Encrypted Media Extensions (http://dvcs.w3.org/hg/html-media/raw-file/default/encrypted-media/encrypted-media.html), there's no bug that I'm aware of, and I don't think there is a plan to implement it at this stage.
Any progress? Is there any estimation on when this API will be implemented?
Depends on: 792675
Attachment #647050 - Attachment is obsolete: true
(In reply to Matthew Gregan [:kinetik] from comment #1)
> I plan to work on this in the near future

Matthew - are you still planning to work on this? If not, can this be reassigned? With WebRTC support around the corner, I think the MediaSource API will allow the creation of some pretty amazing applications.

Please let me know the current status of this bug. Thanks!
Yes, I picked it up again recently.  There are a first set of patches coming shortly.
Excellent!
Anything new? when should we expect first patch?
Hey Matthew, what's the latest on the Media Source Extensions API implementation? You mentioned that a first set of patches would be out around 1 month ago. Any progress?
Depends on: 855130
Alias: mediasource
Whiteboard: [shumway]
List things that we may need to do after Bug 855130 landing.

1. Relation between MediaSource & SourceBuffer should be 1-to-N
2. MediaSource::addSourceBuffer: param(type) MIME type checking.
3. Depend on Bug 744896: to enable/ disable audio track/ to select a video.
4. HTMLMediaElement behavior change 
https://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html#htmlmediaelement-extensions
Depends on: 744896
Items 2 and 4 are partially done in the latest patch attached to bug 855130.  I'll file bugs for the others (and a number of other things) that block this bug over the next day or so.
Depends on: 881512
Depends on: 881514
Depends on: 886194
Depends on: 886196
Depends on: 889201
Depends on: 889202
Depends on: 889205
Depends on: 889712
Depends on: 895754
Depends on: 896866
Depends on: 907468
Depends on: 833023
Hi, 

How is this going? Are you working on it or waiting for some of the dependencies to clear? Can you give a hint of when it will be milestone?

Thank you

Dennis
It's being worked on, most of the action is in the dependent bugs.  The proposed schedule is here: https://wiki.mozilla.org/Platform/MediaSourceExtensions
Blocks: 932516
No longer blocks: 932516
Depends on: 932516
Depends on: 939612
Depends on: 939613
Depends on: 943186
This API is needed if you want to see youtube video on 1080p:
http://googlesystem.blogspot.fr/2013/10/youtube-drops-support-for-1080p-streams.html

" YouTube removed the non-DASH streams for 480p and 1080p. This means that you'll no longer be able to watch 1080p videos in YouTube's HTML5 player if you use Firefox or Safari. "
Depends on: 950120
Blocks: 962353
Blocks: 974362
Depends on: 976037
Whiteboard: [shumway] → [shumway:m2]
Blocks: 979096
No longer blocks: 962353
Depends on: 962353
No longer blocks: 974362
Depends on: 974362
No longer blocks: 979096
Depends on: 979096
Depends on: 979099
Depends on: 979100
Whiteboard: [shumway:m2] → [shumway]
Depends on: 989888
The proposed schedule is from the last year. Is the schedule still actual or not? What is the timeline when Firefox would be able to play videos using DASH reference implementation at dashif.org?
(In reply to anvar.sosnitski from comment #17)
> The proposed schedule is from the last year. Is the schedule still actual or
> not? What is the timeline when Firefox would be able to play videos using
> DASH reference implementation at dashif.org?

Some DASH implementations are working in nightly builds - those that use WebM/VP9. See for example:

http://cd.pn/mse/webm/dash-player.html

Support for YouTube's DASH player is ongoing. Nightly build plus bug 989888 allows playback of VP9 Dash streams on YouTube. More work on that is needed for seeking and stream switching however.

I have not yet investigated dashif.org support due to it requiring (AFAIK) H.264 support and we've had muxer issues to deal with in that regard. I would hop ewe can get on to testing and getting it working in the near future.
Depends on: 238041
Shipping Chrome supports the MSE API:
https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/RG1dEbQEu8E
As does IE 11:
http://msdn.microsoft.com/en-us/library/ie/dn594470%28v=vs.85%29.aspx
Whiteboard: [shumway] → [shumway] [blink-parity] [ie11-parity]
Depends on: 996368
Depends on: 77992
Depends on: 1000608
Depends on: 1000180
Depends on: 1000686
Depends on: 1002019
Depends on: 1002290
Depends on: 1002294
Depends on: 1002320
Depends on: 1002987
Depends on: 1002994
(In reply to cajbir (:cajbir) from comment #18)
> (In reply to anvar.sosnitski from comment #17)
> > The proposed schedule is from the last year. Is the schedule still actual or
> > not? What is the timeline when Firefox would be able to play videos using
> > DASH reference implementation at dashif.org?
> 
> Some DASH implementations are working in nightly builds - those that use
> WebM/VP9. See for example:
> 
> http://cd.pn/mse/webm/dash-player.html
> 
> Support for YouTube's DASH player is ongoing. Nightly build plus bug 989888
> allows playback of VP9 Dash streams on YouTube. More work on that is needed
> for seeking and stream switching however.
> 
> I have not yet investigated dashif.org support due to it requiring (AFAIK)
> H.264 support and we've had muxer issues to deal with in that regard. I
> would hop ewe can get on to testing and getting it working in the near
> future.

@cajbir - I help manage the dash.js project https://github.com/Dash-Industry-Forum/dash.js which is the reference client for the DASH Industry Forum and also the preferred MSE implementation for many members including Microsoft and Akamai. We have good implementations under Chrome and IE11 and would love to add Firefox to that mix, not just for our companies but for the MSE ecosystem in general. If there is anything we can do from our end to help accelerate testing and/or code changes for H.264 ISOBMFF support please don't hesitate to reach out. 

Cheers

Will Law
Depends on: 931388
Depends on: 975782
Depends on: 978613
Depends on: 1008542
Depends on: 1013032
Is there a first FF stable release where MSE can be used or is it still in Nightly?
Dash on youtube doesn't work even in Nightly 32. I think we have to wait for all issues on which this one depends until get it working in nightly. It is too early to talk about stable.
(In reply to Aymeric Vitte from comment #21)
> Is there a first FF stable release where MSE can be used or is it still in
> Nightly?

(In reply to Nazar Mokrynskyi from comment #22)
> Dash on youtube doesn't work even in Nightly 32. I think we have to wait for
> all issues on which this one depends until get it working in nightly. It is
> too early to talk about stable.

It works with VP9/WEBM video streams on youtube. But videos that have them are by far the minority. Also they have around half the bitrate and are more resource demanding to decode.
For whatever (irrational) reason h.264 - which is overwhelmingly the dominant format - is ignored until everything else is done.
(In reply to avada from comment #23)
> For whatever (irrational) reason h.264 - which is overwhelmingly the
> dominant format - is ignored until everything else is done.

This is an inappropriate comment. Please don't pass judgement (using words like irrational) on why things are done, assume good intent from the people doing the work. H.264 support relies on other work which is being done in parallel.
Do you have any idea when this could be available?
Depends on: 1022479
Depends on: 1024858
Depends on: 1025768
Depends on: 1025770
Depends on: 1028642
Depends on: 1028742
Depends on: 1028748
Depends on: 1028144
Depends on: 1030539
Depends on: 1030527
Depends on: 1035622
Afaik h264 in general meanwhile works. Does that mean support for dash/h264 as well (or maybe soon down the road)?
Yes, see bug 1027875.
Depends on: 1031532
Blocks: 1031121
As of today, Google has locked Nightly users out of being able to switch back an forth between HTML5 and Flash. 

HTML5 is now potentially the permanent default on YouTube for Fx33+. This should probably become a higher priority if not already YouTube's policy sticks.
Whiteboard: [shumway] [blink-parity] [ie11-parity] → [shumway] [blink-parity] [ie11-parity][dependency: marketplace-partners]
Depends on: 1041374
Depends on: 1041387
Depends on: 1041396
Depends on: 1041405
Depends on: 1041919
Depends on: 1044505
Depends on: 1044498
Depends on: 1044498
Depends on: 1044762
Depends on: 1043696
Blocks: 1049327
Blocks: 1049326
Blocks: 1049323
Blocks: 1049321
Blocks: 1049318
Blocks: 1049317
Blocks: 1050083
Blocks: 1050099
Depends on: 1050580
Depends on: 1050652
Blocks: 1053657
Blocks: 1053658
Blocks: 1053674
Blocks: 1053682
Depends on: 1040552
Depends on: 1054970
Sandip, how can we nominate this as a b2g 2.2 feature?
Flags: needinfo?(skamat)
Depends on: 1055899
Summary: Implement proposed Media Source Extensions API → Tracking bug for Media Source Extensions API implementation
No longer blocks: 1049317
Depends on: 1049317
No longer blocks: 1049318
Depends on: 1049318
No longer blocks: 1049321
Depends on: 1049321
No longer blocks: 1049323
Depends on: 1049323
No longer blocks: 1049326
Depends on: 1049326
No longer blocks: 1049327
Depends on: 1049327
No longer blocks: 1050083
Depends on: 1050083
No longer blocks: 1050099
Depends on: 1050099
No longer blocks: 1053658
Depends on: 1053658
No longer blocks: 1053674
Depends on: 1053674
No longer blocks: 1053682
Depends on: 1053682
No longer blocks: 1053657
Depends on: 1053657
No longer depends on: 238041
Depends on: 1056440
Depends on: 1056452
Depends on: 1056534
Depends on: 1057181
Depends on: 1058418
Depends on: 1058422
Assignee: kinetik → nobody
Status: ASSIGNED → NEW
Alias: mediasource → MSE
Depends on: 1059569
Depends on: 1059625
Depends on: 1061007
Depends on: 1054783
Now that there is support for H264 in FirefoxNightly when I'm trying to use H264 with MSE I'm getting a 'NotSupportedError: Operation is not supported' error when I'm calling var sourceBuffer = ms.addSourceBuffer('video/mp4;codecs="avc1.42c00d"');
Of course, have you ever look to this site (youtube.com/html5)?
MSE with H264 is not implemented enough today. Just stay tuned, when this bug are resolved, it should work:)
Depends on: 1062017
Depends on: 1062018
Depends on: 1062020
Depends on: 1062023
Depends on: 1062055
Depends on: 1062101
Depends on: 1062657
Depends on: 1062661
Depends on: 1062664
Depends on: 1062666
Depends on: 1062669
Depends on: 1062670
Depends on: 1022075
Blocks: shumway-1.0
No longer blocks: shumway-m4
feature-b2g: --- → 2.2?
Flags: needinfo?(skamat)
Depends on: 1063323
Depends on: 1063365
> feature-b2g: --- → 2.2?

We are currently reviewing the bug backlog for this feature, and the resulting schedule changes. Please don't set this flag until after that review. Thanks!
Depends on: 1064113
Depends on: 1064128
No longer depends on: 1064128
Depends on: 1064160
Depends on: 1064570
Depends on: 1064699
Depends on: 1064705
Depends on: 1065207
Depends on: 1065215
Depends on: 1065218
Depends on: 1065219
Depends on: 1065221
Depends on: 1065235
Depends on: 1065250
Depends on: 1065855
Depends on: 1068483
Depends on: 1069671
Depends on: 1074614
Depends on: 1075980
Depends on: 1066467
(In reply to info from comment #30)
> Now that there is support for H264 in FirefoxNightly when I'm trying to use
> H264 with MSE I'm getting a 'NotSupportedError: Operation is not supported'
> error when I'm calling var sourceBuffer =
> ms.addSourceBuffer('video/mp4;codecs="avc1.42c00d"');

I have the same error. 
Does Anyone know how to fix it?
(In reply to John from comment #33)
> (In reply to info from comment #30)
> > Now that there is support for H264 in FirefoxNightly when I'm trying to use
> > H264 with MSE I'm getting a 'NotSupportedError: Operation is not supported'
> > error when I'm calling var sourceBuffer =
> > ms.addSourceBuffer('video/mp4;codecs="avc1.42c00d"');
> 
> I have the same error. 
> Does Anyone know how to fix it?

Oh, don't support yet, video/webm is OK.
(In reply to John from comment #33)
> (In reply to info from comment #30)
> > Now that there is support for H264 in FirefoxNightly when I'm trying to use
> > H264 with MSE I'm getting a 'NotSupportedError: Operation is not supported'
> > error when I'm calling var sourceBuffer =
> > ms.addSourceBuffer('video/mp4;codecs="avc1.42c00d"');
> 
> I have the same error. 
> Does Anyone know how to fix it?

set media.mediasource.ignore_codecs to true
Depends on: 1081766
Depends on: 1081817
Blocks: 1083089
Depends on: 1086525
Depends on: 1087165
Depends on: ship-MSE
Depends on: 1091774
Depends on: 1092025
Depends on: 1092915
Depends on: 1092932
Depends on: 1093020
Depends on: 1094271
Depends on: 1094501
Depends on: 1096089
Depends on: 1095778
Depends on: 1096752
Depends on: 1096790
Depends on: 1097252
Depends on: 1097441
Depends on: 1098022
Depends on: 1098674
Depends on: 1098678
Depends on: 1098680
Depends on: 1098990
Depends on: 1098637
Depends on: 1100176
Depends on: 1097723
Depends on: 1101062
Depends on: 1101247
Depends on: 1100913
No longer depends on: 1096717
Depends on: 1102666
Depends on: 1104410
Depends on: 1104426
No longer depends on: 1104410
Depends on: 1105066
Depends on: 1102647
Depends on: 1105132
Depends on: 1105553
No longer depends on: 1105514
Depends on: 1100063
Depends on: 1105771
Depends on: 1107611
Depends on: 1107737
Depends on: 1108846
Depends on: 1108917
Depends on: 1109431
Depends on: 1109437
Depends on: 1109987
Depends on: 1103357
Depends on: 1096755
Depends on: 1110534
Depends on: 1111311
Depends on: 1111328
Depends on: 1111413
Release Note Request (optional, but appreciated)
[Why is this notable]: A new area for videos
[Suggested wording]: Media Source Extensions API implemented
[Links (documentation, blog post, etc)]: Any link?

AFAIK, this will ship with 36.
relnote-firefox: --- → ?
Depends on: 1111966
Depends on: 1112424
Depends on: 1112444
Depends on: 1112445
Depends on: 1112446
It doesn't have to be in 2.2. But, need to continue working on it. 
Bobby, please follow up this topic.
feature-b2g: 2.2? → ---
Flags: needinfo?(bchien)
(In reply to Ralph Giles (:rillian) from comment #39)
> Specification is https://w3c.github.io/media-source/
Thanks but I was hoping for something more general and less technical.
continue work and mark as tracking-b2g:+ for future release.
tracking-b2g: --- → +
Flags: needinfo?(bchien)
Priority: -- → P1
Depends on: 1113776
Depends on: 1091325
Depends on: 1113924
Depends on: 1114802
Depends on: 1114840
Depends on: 1114849
Depends on: 1114847
Depends on: 1114844
Depends on: 1114885
Blocks: MSE-FxOS
Depends on: 1108086
Depends on: 1115096
Depends on: 1115190
Depends on: 1115835
(In reply to Sylvestre Ledru [:sylvestre] from comment #38)
> Release Note Request (optional, but appreciated)
> [Why is this notable]: A new area for videos
> [Suggested wording]: Media Source Extensions API implemented
> [Links (documentation, blog post, etc)]: Any link?
> 
> AFAIK, this will ship with 36.

See bug 1112761.  API use in 36 will be limited.
Depends on: 1116043
Depends on: 1116056
Depends on: 1116284
Depends on: 1116353
Depends on: 1116382
Depends on: 1116384
Depends on: 1116626
Depends on: 1062993
Depends on: 1082974
Depends on: 1109292
Depends on: 1117881
Depends on: 1117903
Depends on: 1117991
Depends on: 1118123
Depends on: 1118126
Depends on: 1108847
Depends on: 1118370
Depends on: 1118528
Depends on: 1118533
Depends on: 1118589
Depends on: 1119033
Depends on: 1119119
Depends on: 1119208
Depends on: 1119456
Depends on: 1119757
Depends on: 1120079
Depends on: 1120084
Depends on: 1120086
Depends on: 1120128
Depends on: 1120266
Depends on: 1120282
Depends on: 1120453
Depends on: 1120701
Depends on: 1121288
Depends on: 1121342
Added to the release notes with "Media Source Extensions API implemented (Youtube only)" as wording.
Let me know if someone has an issue with this.
Update to "Implemented a subset of the Media Source Extensions (MSE) API to allow native HTML5 playback on YouTube. Full support is on the way". Thanks to Bobby for the new version!
Depends on: 1121661
Depends on: 1121135
Depends on: 1121668
Depends on: 1121747
Depends on: 1121750
Depends on: 1121757
Depends on: 1122149
Depends on: 1122250
Depends on: 1122358
Depends on: 1122803
Depends on: 1122802
Depends on: 1122873
Depends on: 1123198
Depends on: 1123202
Depends on: 1123203
Depends on: 1123492
Depends on: 1123507
Depends on: 1123923
No longer depends on: 1121747
Depends on: 1124316
Depends on: 1124487
Depends on: 1124952
Depends on: 1125469
Depends on: 1125581
Depends on: 1125776
Depends on: 1125911
Depends on: 1125915
Depends on: 1125917
Depends on: 1125936
Depends on: 1125993
No longer depends on: 1125993
Depends on: 1126052
Depends on: 1126332
Depends on: 1126465
Blocks: 1124024
Depends on: 1127203
Depends on: 1127122
Depends on: 1127448
Depends on: 1127554
Depends on: 1127775
Depends on: 1127920
Depends on: 1128069
Depends on: 1125621
Depends on: 1128115
Depends on: 1128171
Depends on: 1128179
Depends on: 1128013
Depends on: 1128295
Depends on: 1128332
Depends on: 1128410
Depends on: 1114976
Depends on: 1128409
Depends on: 1081911
Depends on: 1128649
Depends on: 1128170
Depends on: 1128742
Depends on: 1129224