Closed Bug 577084 (HLS) Opened 14 years ago Closed 6 years ago

Implement HTTP Live Streaming (HLS) for audio and video on Android

Categories

(Core :: Audio/Video: Playback, defect)

Product:
Core
Component:
Audio/Video: Playback
Platform:
All
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: usenet, Unassigned)

References

Details

(Keywords: dev-doc-complete) 

Apple has released the spec of their HTTP Live Streaming protocol, which is already implemented in their Safari browser, as an Internet-Draft. 

See http://tools.ietf.org/html/draft-pantos-http-live-streaming-04 and http://developer.apple.com/iphone/library/documentation/networkinginternet/conceptual/streamingmediaguide/introduction/introduction.html for more information on this protocol.

I believe that streaming protocol is elegant and lightweight, bandwidth-adaptive, and independent of any particular media file format or codec, and seems an ideal fit with Mozilla's support for WebM. By putting the burden of streaming adaptation on the client, and only using standard HTTP GET requests for communication, it also allows the use of standard webservers to support streaming content.

If Firefox is to support a streaming protocol, it seems to me that it should be this one.
A quick glance through the current draft shows that WebM wont be supported by the spec:

4.  Media files

   Each media file URI in a Playlist file MUST identify a media file
   which is a segment of the overall presentation.  Each media file MUST
   be formatted as an MPEG-2 Transport Stream or an MPEG-2 audio
   elementary stream [ISO_13818].
True, but it seems possible to create something similar using WebM. (I'm not sure what it would take to get that spec updated possibly.) And using a proposed extension to the MediaSource-API even JavaScript could theoretically be used to make HLS for WebM possible.
http://www.ioncannon.net/utilities/1515/segmenting-webm-video-and-the-mediasource-api/
Playing HLS actually works on firefox, for WebM. I can't say much about it, it just surprised (and confused) me to see this bug.
Luciana: it does? Can you give an example stream?

We are implementing DASH, which supports WebM more officially than the Apple 'Live Streaming' spec. That's bug 702122, but the code hasn't landed yet.
How does it work on Firefox Android with HLS H.264 ?
Perhaps this issue should be reconsidered, as Firefox supports H.264 on Windows now. \

I see no practical reason in NOT supporting HLS: it's a simple spec, it is widespread on mobile (every android device newer than 3.0 supports it), and is the de-facto standard of streaming media with http.
As I understand it, HLS requires MPEG-2 TS, which we don't plan to support at this stage.  See https://bugzilla.mozilla.org/show_bug.cgi?id=799315#c44 for some discussion.
Afaik MPEG-TS around the H.264-data is demanded, yes - as says http://tools.ietf.org/html/draft-pantos-http-live-streaming-10 in chapter 4 (page 18, "Media segments"). But where exactly is the problem? Imho TS is what you'd usually encapsulate that data in, isn't it?
The plan is to implement the standardized version of HLS: DASH
with MP4 as container. See https://bugzilla.mozilla.org/show_bug.cgi?id=799315#c48
Blocks: youtube.com
Youtube Live provide HLS as Live video stream to mobile platform. On android, you can watch Youbube Live by Youtube app. But on B2G, without the b2g's support, you can not watch Youtube Live on Firefox OS device.

Is it possible to support HLS only on B2G?
For Apple HTTP Live Streaming, please check out my JavaScript proof-of-concept experiment at http://rreverser.github.io/mpegts. It allows to play HLS in realtime by converting MPEG-TS chunks to already supported MP4 in realtime in Web Worker using jBinary (http://github.com/jDataView/jBinary) library with described binary structures according to standards.

Currently there are noticable lags on switching videos since those are still separate chunks, but I believe that with coming Media Source Extensions it could be continued to generate MP4 in parts with `mfra` atoms and sending that as single MP4 stream to <video> element.

So even if HLS wouldn't be implemented on browser level, it still can be done in JavaScript.
(In reply to me from comment #15)
> For Apple HTTP Live Streaming, please check out my JavaScript
> proof-of-concept experiment at http://rreverser.github.io/mpegts. 

Awesome!
Thanks Ingvar for your work on mpegts (and JBinary)!. I tested it and I could manage to play live content from the DVB-T using ffmpeg for hls streaming. There were some blocking issues when playing a new fragment so I tried to use MediaSource to make it run properly.

Still it looks like both Firefox and Chromium don't support h264 content injected on the (MediaSource) video buffers even when h264 is active (through gstreamer on Linux for example). While I prefer much more to default to oepn WebM/VP8/VP9 videos, one big issue is that digital TV broadcasting at this moments is monopolized by h264 and it will imposse a (very) heavy h264-to-VP8-transcoding work to avoid the h264 trap. (I'm speaking about the use-case of a set-top box streaming live-TV to the browser. For VoD there is no problem, since transcoding can be made offline). I already placed a new bug in the Chromium track system (http://code.google.com/p/chromium/issues/detail?id=349445&can=4&colspec=ID%20Pri%20M%20Iteration%20ReleaseBlock%20Cr%20Status%20Owner%20Summary%20OS%20Modified). Maybe someone more experienced can open a similar one for Firefox.

(Note: Chromium TS parsing runs about 3 times slower than Firefox in my tests. It looks it's due to its ArrayBuffer implementation)
Thanks guys for feedback! Probably anybody wants to contribute so we could let JavaScript rock for HTTP Live Streaming?
With the H.264 codec release from Cisco did that help the traction for HSL with Firefox?
(In reply to Tim from comment #19)
> With the H.264 codec release from Cisco did that help the traction for HSL
> with Firefox?

It makes sense to use Ingvar's Javascript based solution when MSE is ready. MP4 support for MSE is not ready yet. Windows and Mac OSX use the platform decoders and on Linux can be tested with ffmpeg in places where patents don't apply.

OpenH264 needs high profile support and support for AAC audio before it is widely applicable for video playback.
Blocks: 1170774
Microsoft added native HLS (and DASH) support to Edge on Windows 10. Microsoft already had MSE support from IE11 on Windows 8.1.

http://blogs.msdn.com/b/ie/archive/2015/01/29/simplified-adaptive-video-streaming-announcing-support-for-hls-and-dash-in-windows-10.aspx
Alias: HLS
Summary: Implement HTTP Live Streaming for audio and video → Implement HTTP Live Streaming (HLS) for audio and video
Depends on: 1071132
Blocks: 1071132
No longer depends on: 1071132
Component: Audio/Video → Audio/Video: Playback
(In reply to MrX1980 from comment #22)
> Some HLS example videos:
> http://www.jwplayer.com/html5/hls/
> https://walterebert.com/playground/video/hls/

And Twitch live stream (works on Ms Edge browser) : http://www.twitch.tv/ogamingtv/html5
What's the current plan? Could we support HLS using a JavaScript addon (like pdf.js for PDFs)?
We don't currently plan to implement native support for HLS in Firefox. Safari and Edge implement HLS on desktop platforms, but neither Chrome nor IE do. Firefox supports MSE, which allows third-party libraries like Dailymotion's HLS.js to support HLS:

https://github.com/dailymotion/hls.js
Should we close this bug then and think about alternatives for all the sites that are depending on this?
I should clarify: we are no longer considering HLS for desktop Firefox, but we may need to implement it on Android or FFOS for mobile compatibility.
OS: All → Android
Summary: Implement HTTP Live Streaming (HLS) for audio and video → Implement HTTP Live Streaming (HLS) for audio and video on Android
(In reply to Chris Peterson [:cpeterson] from comment #27)
> We don't currently plan to implement native support for HLS in Firefox.
> Safari and Edge implement HLS on desktop platforms, but neither Chrome nor
> IE do. Firefox supports MSE, which allows third-party libraries like
> Dailymotion's HLS.js to support HLS:
> 
> https://github.com/dailymotion/hls.js

How do I install it as a plugin? I looked at description of this DailyMotion and similar RReverser library, but I could not find information on how to install it so playback would work on any site that now says that FireFox fails to playback videos.
https://github.com/mangui/firefox-hls

Once the xpi is installed, clicking directly on a m3u8 link will automatically play.
(In reply to Jean-Yves Avenard [:jya] from comment #37)
> https://github.com/mangui/firefox-hls
> 
> Once the xpi is installed, clicking directly on a m3u8 link will
> automatically play.

It loads build-in player, but doesn't play a thing (No media in info). Does it need firefox to be restarted?

Test links played in VLC, but failing to play with the extension:

    http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/sl.m3u8
    http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8
    http://devimages.apple.com/iphone/samples/bipbop/gear1/prog_index.m3u8
    http://playertest.longtailvideo.com/adaptive/oceans_aes/oceans_aes.m3u8 (AES encrypted)
    http://playertest.longtailvideo.com/adaptive/captions/playlist.m3u8 (HLS stream with CEA-608 captions)
    http://playertest.longtailvideo.com/adaptive/wowzaid3/playlist.m3u8 (with metadata)
    http://content.jwplatform.com/manifests/vM7nH0Kl.m3u8
    http://cdn-fms.rbs.com.br/hls-vod/sample1_1500kbps.f4v.m3u8
    http://cdn-fms.rbs.com.br/vod/hls_sample1_manifest.m3u8
    http://vevoplaylist-live.hls.adaptive.level3.net/vevo/ch1/appleman.m3u8 (LIVE TV)
    http://vevoplaylist-live.hls.adaptive.level3.net/vevo/ch2/appleman.m3u8 (LIVE TV)
    http://vevoplaylist-live.hls.adaptive.level3.net/vevo/ch3/appleman.m3u8 (LIVE TV)
    http://www.nacentapps.com/m3u8/index.m3u8 (VOD)
    http://srv6.zoeweb.tv:1935/z330-live/stream/playlist.m3u8 (LIVE TV)
    http://content.jwplatform.com/manifests/vM7nH0Kl.m3u8 (link protection, video not encrypted)
    http://sample.vodobox.net/skate_phantom_flex_4k/skate_phantom_flex_4k.m3u8 (4K)
    http://cdn3.viblast.com/streams/hls/airshow/playlist.m3u8 (4K)
Retiring MP4 video output support on August 1st 2016
https://twittercommunity.com/t/retiring-mp4-video-output-support-on-august-1st-2016/66045

"Beginning August 1, 2016, MP4 URLs will be removed from the payload of Tweets that contain video. Twitter website, iOS and Android applications as well as the Fabric SDK have been updated to solely use adaptive streaming. HLS URLs (.m3u8) will remain unchanged and applications currently using MP4 URLs for video playback should move to using these HLS URLs instead. We are making these changes because adaptive streaming provides a higher quality experience for users by delivering more reliable playback, faster start times, and more efficient data usage across variable network conditions.

One week after MP4 URLs have been removed from the Tweet payload, requests for MP4 videos will no longer be served."
This is just plain silly... HLS is patent encumbered and unsupported by most web browsers.

If there wasn't an alternative I could understand but we have DASH and MSE.
Flags: needinfo?(cpeterson)
(In reply to Jean-Yves Avenard [:jya] from comment #40)
> HLS is patent encumbered and unsupported by most
> web browsers.

Browser charts about native HLS and MPEG-DASH support seem to be difficult to get but if I'm not wrong only Microsoft Edge, Safari and Chrome on Android do support native HLS. Native MPEG-DASH support seems to be even less widespread. Firefox does not support native HLS or MPEG-DASH but for the latter the ticket #1263150 already exists.


(In reply to Jean-Yves Avenard [:jya] from comment #40)
> If there wasn't an alternative I could understand but we have DASH and MSE.

MSE but not MPEG-DASH (interestingly most users I see on the web seem to be confused by that). I wonder if MPEG-DASH would be more adopted by the industry instead of HLS if there would be reliable native browser support.
Anthony and Blake are investigating options for supporting HLS on Android.

AFAIK, Safari and Edge are the only desktop browsers with native support for HLS. Edge also supports MPEG-DASH:

https://blogs.windows.com/msedgedev/2015/01/29/simplified-adaptive-video-streaming-announcing-support-for-hls-and-dash-in-windows-10/
Flags: needinfo?(cpeterson)
A possible approach is like bug 780379 to use Android's streaming media player to play HLS data. As bug 941351 mentioned as well, bug 780379 cannot fix the case, media element's src = xxx.m3u8. To solve it, an idea would be to send src's url to AndroidBridge.cpp[1]from media element and send the url to Android world and play it with Android's media player. 

[1]https://dxr.mozilla.org/mozilla-central/source/widget/android/AndroidBridge.cpp?from=AndroidBridge.cpp
Alastor, 
This is this bug we discussed yesterday. 
Please help check it for comment 45.
Thanks!
Flags: needinfo?(alwu)
Assignee: nobody → alwu
Flags: needinfo?(alwu)
Depends on: 1282410
We can open HLS with external app after landing bug1282410.
But now, we don't have any plan to do native support for HLS, so free this bug,
Assignee: alwu → nobody
In case it gets planned to add native support for HLS wouldn't this be a potential solution for all platforms? If yes maybe this report should then not get limited to Android anymore or alternatively it should get marked as FIXED or INVALID as HLS is now indirectly supported on Android.
HLS is mostly needed for Android compatibility with mobile websites designed for iOS Safari. Few websites use HLS on desktop. Edge and Safari support HLS natively on desktop, but Chrome and IE (and Firefox) do not.

We recommend that desktop websites needing HLS use Dailymotion's HLS.js library:

http://engineering.dailymotion.com/introducing-hls-js/
I think that this is one of the reasons why websites still need Flash. Twitch.tv recently added HTML5 player and they're streaming video using HLS. On desktop, you can't watch Twitch in Firefox without Flash, but you shouldn't install Flash because it is one big security gap which forgot to die few years ago.

I also talked with few guys from Czech television and they told me that they will enable HTML5 player on desktop version of their website when major browsers will support HLS.
(In reply to Michal Vašíček from comment #50)
> I think that this is one of the reasons why websites still need Flash.
> Twitch.tv recently added HTML5 player and they're streaming video using HLS.
> On desktop, you can't watch Twitch in Firefox without Flash, but you
> shouldn't install Flash because it is one big security gap which forgot to
> die few years ago.
> 
> I also talked with few guys from Czech television and they told me that they
> will enable HTML5 player on desktop version of their website when major
> browsers will support HLS.

As Chris said, they can use https://github.com/dailymotion/hls.js.
As someone that recently had to implement a video player with HLS support, I will firmly state that I do not believe that recommending a JavaScript polyfill is the right approach. We fell back to a Flash solution for IE, and I believe we are going to eventually go with the same for desktop Firefox. The JS solutions resulted in too many situations where the video got stuck when a new segment was requested. No browser with native HLS support ran into this issue.
(In reply to Chris Peterson [:cpeterson] from comment #49)
> HLS is mostly needed for Android compatibility with mobile websites designed
> for iOS Safari. Few websites use HLS on desktop. Edge and Safari support HLS
> natively on desktop, but Chrome and IE (and Firefox) do not.
> 
> We recommend that desktop websites needing HLS use Dailymotion's HLS.js
> library:
> 
> http://engineering.dailymotion.com/introducing-hls-js/

I think you missed what I was pointing to. Based on comment #47 we should get HLS support on Android with Firefox 50 by using an external approach. With Alastors's freeing for this ticket he implies that the next step is now to implement HLS natively on Android (so I'm requesting to change the title from "Implement HTTP Live Streaming (HLS) for audio and video on Android" to "Implement native HTTP Live Streaming (HLS) for audio and video on Android"). My point was as soon as we have a native solution for HLS on Android in the codebase porting it to the desktop might be not that much effort anymore and thus could be acceptable.
QA Contact: mihai.ninu
jacheng and I studied for few days to understand current status about HLS on Fennec.
Several options were discussed and would like to see if there's any comment here.
There will be a further discussion with CPearce & AJones next week in TPE.
Any feedback is welcome, thanks !

Opt 1 :
Based on current front-end solution which HLS playback starts with an overlay VideoView on top of Fennec App, we should provide as many HTMLMediaElement's DOM events as possible for JS player to synchronize its playing state while operating on VideoView (or use Android MediaPlayer instead, if we want playback in the background).

====
Pros : a) All unsupported formats do it in the same way.
       b) Rely on no other libraries but only components from Android framework.
       c) Low implementation cost.
Cons : a) Inconsistent UX while playing unsupported media (Always in fullscreen mode)
=========================================================

Opt 2 :
Leverage ExoPlayer java library's HLS related components.

1. Leverage ManifestFetcher & HlsPlaylistParser to build HlsMediaPlaylist.
2. Download packets via DefaultUriDataSource and Loadable object "TSChunk". The samples are extracted by HlsExtractorWrapper(TSExtractor) and are spliced inside each extractor to make sure data starting with a keyframe, then stored into SampleQueue.
3. Samples are requested by MFR, and are placed into SampleHolder. (We could either deliver these samples to Codec process and create FakeMediaData for MFR or download these samples in Codec and update information back to main process for making FakeMediaData)

====
Pros : a) Playback for HLS is fully controlled by HTMLMediaElement.
Cons : a) Huge implementation effort
       b) Library Maintenance cost
       c) Device may not have this library.

=========================================================
Opt 3 :
Implement hls.js-like functions (Download & Convert TS to MP4) , then leverage MSE API.

====
Pros : a) Native-support-like UX.
Cons : a) The cost of implementation
=========================================================
 
What we have to confirm at first is ...
do we need to pay royalty if Fennec is unpacking TS either via ExoPlayer(Opt. 2) or by itself (Opt. 3)?

Ref : M2TS patent pool, http://www.mpegla.com/main/programs/m2s/Documents/m2sCrossRefChart.pdf
      It seems that application will be charged if it decodes/unpacks/creates TS file.
QA Contact: mihai.ninu → sorina.florean
Depends on: 1375474
Depends on: 1375478
Blocks: HLS_on_Fennec
(In reply to Chris Peterson [:cpeterson] from comment #49)
> HLS is mostly needed for Android compatibility with mobile websites designed
> for iOS Safari. Few websites use HLS on desktop. Edge and Safari support HLS
> natively on desktop, but Chrome and IE (and Firefox) do not.Thanks,

Is this still true? I think the UK Parliament site is using HLS; it works in Chrome but not in Firefox.

http://parliamentlive.tv/Commons

Gerv
(In reply to Gervase Markham [:gerv] from comment #55)
> (In reply to Chris Peterson [:cpeterson] from comment #49)
> > HLS is mostly needed for Android compatibility with mobile websites designed
> > for iOS Safari. Few websites use HLS on desktop. Edge and Safari support HLS
> > natively on desktop, but Chrome and IE (and Firefox) do not.Thanks,
> 
> Is this still true? I think the UK Parliament site is using HLS; it works in
> Chrome but not in Firefox.
> 
> http://parliamentlive.tv/Commons
> 
> Gerv

It is working for me in Firefox 59 on Windows 10.
(In reply to Gervase Markham [:gerv] from comment #55)
> (In reply to Chris Peterson [:cpeterson] from comment #49)
> > HLS is mostly needed for Android compatibility with mobile websites designed
> > for iOS Safari. Few websites use HLS on desktop. Edge and Safari support HLS
> > natively on desktop, but Chrome and IE (and Firefox) do not.Thanks,
> 
> Is this still true? I think the UK Parliament site is using HLS; it works in
> Chrome but not in Firefox.
> 
> http://parliamentlive.tv/Commons
> 
> Gerv

I truthfully believe this comment (i.e. few use it on desktop) was not true when it was even made. Most sites (e.g. UK Parliament) are implementing a polyfill to make it work on desktops. Truthfully native HLS support would be a welcome edition to all platforms, not just Android. That said the original comment is still correct that it HLS is only natively supported by Edge and Safari.
> (In reply to Gervase Markham [:gerv] from comment #55)
>Truthfully native HLS support would be a welcome edition to all platforms, not just Android

What would be even more welcome would be for sites to collectively put pressure on Apple instead to adopt the Dash web standard natively on iOS. But I guess changing the rest of the world (and delivering a potentially inferior experience to them in the meantime) is the only way out if we've so given up on Apple.
(In reply to Thomas Wisniewski from comment #58)
> > (In reply to Gervase Markham [:gerv] from comment #55)
> >Truthfully native HLS support would be a welcome edition to all platforms, not just Android
> 
> What would be even more welcome would be for sites to collectively put
> pressure on Apple instead to adopt the Dash web standard natively on iOS.
> But I guess changing the rest of the world (and delivering a potentially
> inferior experience to them in the meantime) is the only way out if we've so
> given up on Apple.

While Dash may be nice, unless HTML5test is mistaken or outdated, Edge is currently the only browser to support a native implementation of it. At least HLS has a bit more support given that it has Edge, Safari (desktop and mobile), and Chromium-based Android browsers.
Sure, but then why not just ask everyone to support native Dash instead of the splintering that has occurred over time?

(But I'll stop playing Devil's advocate, as sites have already made their choice).
I see parliamentlive.tv downloading m3u8 and ts files, but it plays for me in Firefox on macOS as well, so they must be using a player which supports HLS sources on browsers with the MediaSource Extensions API.

Separately, I'm not sure what advantage native dash or hls implementation would have, when it can easily be done on the client side, leaving site authors freedom to optimize for their needs.
I recommend we close this bug now, I've gone through and verified HLS is working in Firefox 59+, and the few (8/120) where it's not I'll file follow up bugs -- half seem like UA sniffing things.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
So, to be clear... we support Apple's HLS on Android, but only on Android, as of 59?
Documentation updated:

https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats#Protocols
https://developer.mozilla.org/en-US/Firefox/Releases/59

Please let me know if there's any change to support for desktop; both because I'll need to update docs and because there are a couple of sites I use that I have to crank up Safari for on my Mac because of this. :)
Keywords: dev-doc-needed → dev-doc-complete
Depends on: 1552145
You need to log in before you can comment on or make changes to this bug.