1057646 - HTML5 <video> support for H.264 via OpenH264

Status: RESOLVED WONTFIX

(Core :: Audio/Video: Playback, enhancement)

Description

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

We have an H.264 codec now, by way of OpenH264, and it's already being used for WebRTC.  It could be useful to allow using it for HTML5 <video>, as an alternative to the existing support for OS-provided media facilities (e.g., bug 886181 comment #7).

Comment 1

[cajbir (:cajbir)]

We'd need a solution for the audio side of things (ie. AAC) for this to be useful I think.

Comment 2

[Chris Pearce [:cpearce (Not reading bugmail)]]

OpenH264 also needs to support Main and High Profile H.264 before it's really useful for <video>.

Comment 3

[Ethan Hugg [:ehugg]]

The OpenH264 project is working on including Main and High Profile, but adding support or licensing for AAC is not on the roadmap.

Comment 4

[Dave Garrett]

(In reply to cajbir (:cajbir) from comment #1)
> We'd need a solution for the audio side of things (ie. AAC) for this to be
> useful I think.

Actually, it can be useful without that. Some services are re-encoding animated GIFs as audioless video using h.264. (e.g. Twitter)

Video decoding via OpenH264 could also be useful to provide a consistent codec implementation for video, even if a platform audio codec is needed. Opus or another free audio codec could also be used with h.264 video.

Comment 5

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Gecko has the support. We just need a reason to enable it.

Comment 6

[Daniele "Mte90" Scasciafratte]

To improve the performance of videos?
To avoid problems on pc that don't have the codecs?

If OpenH264 is only for WebRTC is useless for the major people that use the browser for the mainly stuff like social networks and see videos.

Comment 7

[Dave Garrett]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #5)
> Gecko has the support. We just need a reason to enable it.

As far as I'm concerned, the primary reason to use OpenH264 for HTML5 <video> is that it would give a universal and _consistent_ codec to all platforms. The current support is patchy and buggy and reliant on whatever the system has. This is not a reliable system. It's impossible to test properly on all platforms, as real world systems are messy, and using a system codec means relying on that mess.

Comment 8

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to Daniele "Mte90" Scasciafratte from comment #6)
> To improve the performance of videos?

It doesn't do that.

> To avoid problems on pc that don't have the codecs?

Are you talking about machines that have an AAC decoder but no H.264 decoder. How common are those?

(In reply to Dave Garrett from comment #7)
> As far as I'm concerned, the primary reason to use OpenH264 for HTML5
> <video> is that it would give a universal and _consistent_ codec to all
> platforms. The current support is patchy and buggy and reliant on whatever
> the system has. This is not a reliable system. It's impossible to test
> properly on all platforms, as real world systems are messy, and using a
> system codec means relying on that mess.

OpenH264 doesn't get away from system decoders because you still need an AAC decoder. The advantage of system decoders is that they support hardware acceleration. The system software decoders are generally reliable.

Comment 9

[Dave Garrett]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #8)
> OpenH264 doesn't get away from system decoders because you still need an AAC
> decoder. The advantage of system decoders is that they support hardware
> acceleration. The system software decoders are generally reliable.

h.264 is unfortunately ubiquitous for video; AAC, on the other hand, does not have to be used. Opus is available for free to all and is currently the best lossy audio codec in existence for virtually all use-cases. There isn't a video codec that can do this, yet.

The system decoders are reliable until they aren't. At minimum, this should be available for when the system ones break.

Also, why doesn't OpenH264 support hardware acceleration? That seems like something worth doing (at least, of course, for mainstream hardware).

Comment 10

[u493872]

Excuse my ignorance but Twitter states "We currently support MP4 and MOV video formats on mobile apps.

On the web, we support the MP4 video format with H264 format with AAC audio. You can upload videos up to 512MB, however you will be prompted to edit videos to 30 seconds or less in length."

Is the lack of ACC audio support the reason video will not play in firefox? If that is so is there not a way to get the video at least to play?

Comment 11

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to crivera99 from comment #10)
> Is the lack of ACC audio support the reason video will not play in firefox?

What platform are you referring to? Which version of Firefox?

> If that is so is there not a way to get the video at least to play?

Which video?

Comment 12

[u493872]

I have windows xp, version 42 I guess they are not supporting XP anymore and that is the problem. Twitter video not working.

Comment 13

[Dave Garrett]

(In reply to crivera99 from comment #12)
> I have windows xp, version 42 I guess they are not supporting XP anymore and
> that is the problem. Twitter video not working.

This bug is about adding a new feature, not existing video playback support. However, this new feature would help you, as I don't think we ever added support for h.264 <video> on WinXP. I think it's Vista and up due to WinXP lacking the needed support, itself. That said, supporting WinXP isn't a sufficient reason to do anything anymore, by itself, at least.

Comment 14

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to Dave Garrett from comment #13)
> This bug is about adding a new feature, not existing video playback support.
> However, this new feature would help you, as I don't think we ever added
> support for h.264 <video> on WinXP. I think it's Vista and up due to WinXP
> lacking the needed support, itself. That said, supporting WinXP isn't a
> sufficient reason to do anything anymore, by itself, at least.

Supporting MP4 playback on Windows requires both H.264 and AAC support to get video and audio. We are looking for a solution for Windows XP.

Comment 15

[Yorgos]

May this topic be related with
https://bugzilla.mozilla.org/show_bug.cgi?id=1230785

?

Comment 18

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

OpenH264 isn't useful for video playback so I'm going to close this bug.

Comment 19

[Yorgos]

may it be useful in case like this (desperate) one?
https://bugzilla.mozilla.org/show_bug.cgi?id=1222272#c86

Comment 20

[KOLANICH]

> OpenH264 isn't useful for video playback so I'm going to close this bug.

Could you clarify why do you think it is not useful?

Comment 21

[Antony Jones]

(In reply to KOLANICH from comment #20)
> > OpenH264 isn't useful for video playback so I'm going to close this bug.
> 
> Could you clarify why do you think it is not useful?

I'm curious why this comment has been associated with me...
I just got a 'Need Info' email in my work account: I can't remember using Bugzilla, and at first assumed it was a phishing attack...
In any case, I'm not [e-mail]

Comment 23

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to KOLANICH from comment #20)
> > OpenH264 isn't useful for video playback so I'm going to close this bug.
> 
> Could you clarify why do you think it is not useful?

OpenH264 is not useful because it is too slow for playback purposes and doesn't support b-frames. I'm not aware of any platforms that have an AAC decoder but are missing an H.264 decoder. You could encode adverts so that OpenH264 can play them but it would make more sense to use WebM.

Comment 24

[Jan Horak [:jhorak]]

Attachment: openh264.patch

For those who might be interested in using openh264 for h264 playback, I've attached the patch. With current openh264 version, only baseline profile works, also there's missing aac codec for audio playback.

There's some move forward regarding B-frame support: https://github.com/cisco/openh264/issues/2981

Comment 25

[Jan Horak [:jhorak]]

Attachment: libopenh264+libfdk+ffmpeg-pref.patch

Added also fdk-aac for audio playback and media.ffmpeg.preferred preference for giving system ffmpeg lib higher.

Comment 26

[Zach Lym]

Just to explain/update Jan's comments:

As of v2.0 OpenH264 supports the main and high profiles and should work for most H.264 files. AAC support can be handled by FDK-AAC-Stripped, which is a version of FDK-AAC with all of the patented bits stripped out.

FDK-AAC has a non-standard license, but Redhat legal and the FSF consider it to be free software, although Debian has placed it in their non-free repo. WRT patents, Redhat legal has green-lit the use of FDK-AAC-Stripped in mainline.

It might be time to reconsider using OpenH264 for videos, given that H264 patents will be around until ~2027. But it is more code to maintain for a feature that is best handled using system libraries or hardware decoders....

Comment 27

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

I'm no longer involved in media playback, but I can give some commentary.

The forward direction for this (ignoring the long tail) is to get people using VAAPI through Wayland. The openh264 path exists but last time I checked (years ago) the performance was poor. Playback is more demanding on the CPU than video calls and openh264 was only single threaded (that may not still be true).

Widevine includes support for encrypted H.264, so once you get AAC you get most of the premium (DRM-based) services working. YouTube uses free codecs so that will also work. What you're left with is a bunch of sites that all have different player implementations and you can't rely on them responding to dropped frames should openh264 fail to keep up.

Any decisions about this are up to Nils.

Nils - any thoughts or has anything changed since last I looked at this?

Comment 28

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Folks, can be this patch included as a Tier-3 which means Mozilla will not use this code path but it will be used by distros which can't use ffmpeg by default like Fedora/RHEL?

Comment 29

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

I'm going to work on that. This is important to have it as Chromium already ships it and can play H.264 clips out of the box but Firefox fails so people are directed to use Chrome/Chromium.

The OpenH264 codec already supports the high profile and should be used only when system ffmpeg is missing and we're requested to play H.264 clips.

Comment 30

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Also I think the final solution should be incorporated with Bug 1660336 as we may:

1. use system ffmpeg first if VA-API is enabled and recent enough
2. fall back to bundled (ffvpx) for VP9/8
3. try system ffmpeg
4. fallback to openh264 when system ffmpeg is missing

Comment 31

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1057646 [Linux] Add OpenH264 decoder files to bundled ffvpx, r?jya

Comment 32

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1057646 [Linux] Add FDK AAC Decoder files to bundled ffvpx, r?jya

Depends on D89343

Comment 33

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1057646 [Linux] Add OpenH264 / FDK AAC build config to bundled ffvpx, r?jya

Depends on D89344

Comment 34

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1057646 [Linux] Add OpenH264 / FDK AAC build options to configure files, r?glandium

Depends on D89345

Comment 35

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1057646 [Linux] Add media.ffvpx.openh264.enabled pref to control OpenH264 usage, r?jya

Depends on D89346

Comment 36

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1057646 [Linux] Add OpenH264 control to FFmpegDecoderModule, r?jya

• Add empty EnableOpenH264Decoder() method to PlatformDecoderModule.
• Implement it for ffvpx FFmpegDecoderModule module by mIsOpenH264Enabled attribute.

Depends on D89347

Comment 37

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1057646 [Linux] Use OpenH264 as a fallback when system ffmpeg is missing, r?jya

Depends on D89348

Comment 38

[Jean-Yves Avenard [:jya]]

We already have a OpenH264 decoder via the GMP.

It can be used for HTML5 video so long as it's prefed on. The lack of support for the high profile was an issue in the past, we would need to update the GMP component.

However, this doesn't solve the case where OpenH264 is provided by Cisco and needs to be installed regardless, and on Android it appears we will no longer be able to install the OpenH264 plugin unless we use the playstore.

Comment 39

[Jean-Yves Avenard [:jya]]

(In reply to Martin Stránský [:stransky] from comment #29)

I'm going to work on that. This is important to have it as Chromium already ships it and can play H.264 clips out of the box but Firefox fails so people are directed to use Chrome/Chromium.

I don't believe that's the case.

Chrome can play h264 out of the box, but not Chromium.

The OpenH264 codec already supports the high profile and should be used only when system ffmpeg is missing and we're requested to play H.264 clips.

Then we need to update the various OpenH264 plugin and let the GMPDecoderModule be a default for video decoding
https://searchfox.org/mozilla-central/source/dom/media/platforms/PDMFactory.cpp#398-404

and for that to work the pref media.gmp.decoder.enabled needs to be turned on.
https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml#7149-7152