Closed Bug 1562862 Opened 6 years ago Closed 6 years ago

Webm support for h.264 codec

Categories

(Core :: Audio/Video: Playback, enhancement)

Product:
Core
Component:
Audio/Video: Playback
Version:
69 Branch
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: jeroen, Unassigned)

References

(
URL
)

Details

Attachments

(2 files, 3 obsolete files)

Screenshot_2019-07-11 Screenshot.png
121.73 KB, image/png
Details
webm_mp4.diff
8.57 KB, patch
Details | Diff | Splinter Review
Attached patch webm_mp4.diff (obsolete) — — Splinter Review

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0

Steps to reproduce:

Tried to play a webm file with h.264 content.
Also tried to use this format with the media extensions.
Being able to stream h.264 with webm and media extensions allows me to create livestreams and send them to multiple users without re-encoding. The webm is extremely suited for this purpose.

mimetype support test:
http://dmlinking.net/~pe1rxq/video.html
simple website with videofile with h.264 content
http://dmlinking.net/~pe1rxq/testmp4.webm.html

Same h.264 content in mp4 container:
http://dmlinking.net/~pe1rxq/test.mp4.html

Website using media extensions to feed a livestream.
(Video streams are in blue)
-click on 'connect & play' for a stream
-vp8 based stream 'testvideo.pe1rxq.ampr.org' works
-h.264 based stream 'webcam.buurtschapcrescendo.nl' does not work
http://dmlinking.net:8800/

Actual results:

File was not played.
Also mimetype was reported as unsupported.

Expected results:

Since H.264 is supported but with another container and the webm container (based on matroska) can handle h.264 just fine it is possible to modify the demuxer for webm slightly to support is.

I have tested the attached patch to add h.264 support to the firefox nightly, and it works for all my mentioned use cases.

Component: Untriaged → Audio/Video: Playback
Product: Firefox → Core
Type: defect → enhancement

A webm is by definition restricted to a narrow set of codec as a subset of mkv.

We don't have plan to add support for mkv

Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX

Indeed technically the file is a matroska file.
But I would really like you to reconsider the plan not to support any mkv.

First the requested codec (h.264) is already supported by firefox: so no change here.
Second firefox should be liberal in the files it support.
Especially since third: All the puzzle pieces are already present. The attached patch is less than 30 lines of code. It is barely a new feature.

And on a personal note: it would really help me to make live streaming using a media extensions possible.
I prefer VP8/9 myself as well, but some of my streams are generated using hardware encoders producing h.264 unfortunately.
This 30 line patch made everything possible, the alternative is producing a unique bytestream per client to get it through the mp4 demuxer. Which makes the decentralized streaming project I am working on a lot harder.

Google are the ones who set the spec on what is allowed in a webm. So that they could generate something against their own spec is really baffling.

Seeing that webm support isn't universal across browsers, the path you describe will be needed regardless.

What about raising a bug with chrome so that their recorder generate valid files instead, compliant with the spec they have written?

(In reply to Jean-Yves Avenard [:jya] from comment #3)

Google are the ones who set the spec on what is allowed in a webm. So that they could generate something against their own spec is really baffling.

Seeing that webm support isn't universal across browsers, the path you describe will be needed regardless.

What about raising a bug with chrome so that their recorder generate valid files instead, compliant with the spec they have written?

Where in the WebM specification does the language state that WebM can ONLY support certain codecs?

The original WebM specification supported VP8 and Vorbis, correct? Now MediaRecorder does not support Vorbis, but rather supports Opus, and also supports VP9.

Chromium/Chrome currently supports video/x-matroska;codecs=h264, video/x-matroska;codecs=avc1 and video/webm;codecs=h264.

Specifically the W3C MediaRecorder specification https://w3c.github.io/mediacapture-record/#MediaRecorderAPI does not contain any language whatsoever as to a limitation on container or codecs that can be used.

(In reply to Jean-Yves Avenard [:jya] from comment #1)

A webm is by definition restricted to a narrow set of codec as a subset of mkv.

We don't have plan to add support for mkv

Why should mkv not be supported? Support for mkv SHOULD be within whatever plan is being referenced. If not a reason why support WILL NOT be planned needs to be clearly stated, other than because Google said so (though actually does something else). The video encoded using avc1 or h264 has far greater quality and less size compared to VP8, VP9 using MediaRecorder API.

navigator.userAgent
"Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/77.0.3824.6 Chrome/77.0.3824.6 Safari/537.36"

navigator.vendor
"Google Inc."

MediaRecorder.isTypeSupported("video/webm;codecs=h264")
true

MediaRecorder.isTypeSupported("video/webm;codecs=avc1")
true

(In reply to Jean-Yves Avenard [:jya] from comment #3)

Google are the ones who set the spec on what is allowed in a webm. So that they could generate something against their own spec is really baffling.

Seeing that webm support isn't universal across browsers, the path you describe will be needed regardless.

What about raising a bug with chrome so that their recorder generate valid files instead, compliant with the spec they have written?

plnkr https://plnkr.co/edit/nZCNmv?p=catalogue

$ mediainfo h264.webm
General
Complete name : h264.webm
Format : Matroska
Format version : Version 4 / Version 2
File size : 3.13 MiB
Writing application : Chrome
Writing library : Chrome
IsTruncated : Yes
FileExtension_Invalid : mkv mk3d mka mks

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Codec ID : V_MPEG4/ISO/AVC
Width : 500 pixels
Height : 500 pixels
Display aspect ratio : 1.000
Frame rate mode : Variable
Language : English
Default : Yes
Forced : No

(In reply to Jeroen Vreeken from comment #2)

Indeed technically the file is a matroska file.
But I would really like you to reconsider the plan not to support any mkv.

First the requested codec (h.264) is already supported by firefox: so no change here.
Second firefox should be liberal in the files it support.
Especially since third: All the puzzle pieces are already present. The attached patch is less than 30 lines of code. It is barely a new feature.

And on a personal note: it would really help me to make live streaming using a media extensions possible.
I prefer VP8/9 myself as well, but some of my streams are generated using hardware encoders producing h.264 unfortunately.
This 30 line patch made everything possible, the alternative is producing a unique bytestream per client to get it through the mp4 demuxer. Which makes the decentralized streaming project I am working on a lot harder.

Related https://bugzilla.mozilla.org/show_bug.cgi?id=1563140

(In reply to guest271314 from comment #4)

Where in the WebM specification does the language state that WebM can ONLY support certain codecs?

Please refer to webmproject.org, what their first paragraph contains and what the FAQ state
https://www.webmproject.org/about/faq/

"What is WebM?

WebM is an open media file format designed for the web. WebM files consist of video streams compressed with the VP8 or VP9 video codec, audio streams compressed with the Vorbis or Opus audio codecs, and WebVTT text tracks. The WebM file structure is based on the Matroska media container.
"

While webm is based on matroska, it is not matroska. It was defined to contain a very narrow set of open source, royalty free codecs. AV1 was recently added to the list of support. H264 doesn't fit in that category.
There's no room for interpretation in what webm can contain.

The original WebM specification supported VP8 and Vorbis, correct? Now MediaRecorder does not support Vorbis, but rather supports Opus, and also supports VP9.

Chromium/Chrome currently supports video/x-matroska;codecs=h264, video/x-matroska;codecs=avc1 and video/webm;codecs=h264.

This last mimetype is nonsensical, webm shouldn't be used for h264.
Should Chrome decide to support matroska that is their prerogative. However, MediaRecorder supports webm (or mp4). If it generates a webm then it shouldn't generate one with h264 as webm doesn't support h264.

That Chrome reports true for "MediaRecorder.isTypeSupported("video/webm;codecs=h264")" is a bug IMHO and generate invalid content.

(In reply to Jean-Yves Avenard [:jya] from comment #9)

(In reply to guest271314 from comment #4)

Where in the WebM specification does the language state that WebM can ONLY support certain codecs?

Please refer to webmproject.org, what their first paragraph contains and what the FAQ state
https://www.webmproject.org/about/faq/

"What is WebM?

WebM is an open media file format designed for the web. WebM files consist of video streams compressed with the VP8 or VP9 video codec, audio streams compressed with the Vorbis or Opus audio codecs, and WebVTT text tracks. The WebM file structure is based on the Matroska media container.
"

While webm is based on matroska, it is not matroska. It was defined to contain a very narrow set of open source, royalty free codecs. AV1 was recently added to the list of support. H264 doesn't fit in that category.
There's no room for interpretation in what webm can contain.

The original WebM specification supported VP8 and Vorbis, correct? Now MediaRecorder does not support Vorbis, but rather supports Opus, and also supports VP9.

Chromium/Chrome currently supports video/x-matroska;codecs=h264, video/x-matroska;codecs=avc1 and video/webm;codecs=h264.

This last mimetype is nonsensical, webm shouldn't be used for h264.
Should Chrome decide to support matroska that is their prerogative. However, MediaRecorder supports webm (or mp4). If it generates a webm then it shouldn't generate one with h264 as webm doesn't support h264.

That Chrome reports true for "MediaRecorder.isTypeSupported("video/webm;codecs=h264")" is a bug IMHO and generate invalid content.

The language SHOULD in the WebM specification does not prohibit codecs other than VP8 or VP9.

https://www.webmproject.org/docs/container/

Video codec SHOULD be VP8 or VP9

See Does the WebM specification clearly prohibit video/webm;codecs=h264 and video/webm;codecs=avc1 even when the media is ultimately encoded as a Matroska container?

https://groups.google.com/a/webmproject.org/forum/#!searchin/webm-discuss/avc1%7Csort:date/webm-discuss/nJ39yu-CuKI/0xEM9lgPAwAJ

Answer

https://groups.google.com/a/webmproject.org/d/msg/webm-discuss/nJ39yu-CuKI/5uGXrZ4UAwAJ

The codecs are marked as SHOULD [1]. Parsers may handle the format
differently, but it can always be parsed as an mkv file.

The content is not "invalid" when encoded using avc1. At least in the sense that the file can be parsed as an MKV file where the quality of the resulting media output at Chromium when is exceeding and the file size is generally less when compared to VP8 or VP9.

"video/webm;codecs=h264" should perhaps be only "video/x-matroksa;codecs=avc1" as that is what mediainfo outputs.

Am not sure if you are trying to state that Matroksa does not support h264?

That Chrome reports true for "MediaRecorder.isTypeSupported("video/webm;codecs=h264")" is a bug IMHO

File the bug https://bugs.chromium.org/p/chromium/issues/list

Please read the specs more carefully:

https://www.webmproject.org/docs/container/
"Muxers should treat all guidelines marked SHOULD in this section as MUST. This will foster consistency across WebM files in the real world. "

So that sentence reads as "Video codec MUST be VP8 or VP9".

You are arguing semantics in a non-constructive fashion. Again, while WebM is based on matroska, that is irrelevant. WebM gives guarantees on what it will contains: open source codecs.

When you do:
MediaSource.isTypeSupported("video/webm") we know for certain that we can play this file, it doesn't rely on any external 3rd party framework: the answer is YES.
If you could now also supports h264, you're opening a can of worm. There's no longer any certainty as the ability to play. Firefox/gecko does not ship an H264 decoder. Not every platform supports H264, not all Windows installation ships with a H264 (or AAC) decoder (like in Europe or Korea), a 3rd party package must be installed first.

If webm could contain any codecs found in a mkv, there's no point for webm in the first place.

So that sentence reads as "Video codec MUST be VP8 or VP9".

No, that is not what the current specification actually states.

From reading the specification here SHOULD means should where printed. If your reading is that the language SHOULD be be changed to MUST, then directly ask for those changes to the specification.

Firefox/gecko does not ship an H264 decoder.

Does Firefox not support openh264 (https://wiki.mozilla.org/Media/openh264; https://en.wikipedia.org/wiki/OpenH264) encoding and decoding?

See also MPEG LA’s AVC License Will Not Charge Royalties for Internet Video that is Free to End Users through Life of License https://www.mpegla.com/wp-content/uploads/n-10-08-26.pdf

(DENVER, CO, US – 26 August 2010) – MPEG LA announced today that its AVC Patent Portfolio
License will continue not to charge royalties for Internet Video that is free to end users (known as
“Internet Broadcast AVC Video”) during the entire life of this License. MPEG LA previously announced
it would not charge royalties for such video through December 31, 2015 (see
http://www.mpegla.com/Lists/MPEG%20LA%20News%20List/Attachments/226/n-10-02-02.pdf), and
today’s announcement makes clear that royalties will continue not to be charged for such video beyond
that time. Products and services other than Internet Broadcast AVC Video continue to be royalty-bearing.

If webm could contain any codecs found in a mkv, there's no point for webm in the first place.

That is a valid point.

(In reply to guest271314 from comment #13)

So that sentence reads as "Video codec MUST be VP8 or VP9".

No, that is not what the current specification actually states.

From reading the specification here SHOULD means should where printed. If your reading is that the language SHOULD be be changed to MUST, then directly ask for those changes to the specification.

It's exactly what the specs states. SHOULD should be changed to MUST when it comes to a muxer. "Muxers should treat all guidelines marked SHOULD in this section as MUST" That's how it reads. I can't make it any clearer.

Does Firefox not support openh264 (https://wiki.mozilla.org/Media/openh264; https://en.wikipedia.org/wiki/OpenH264) encoding and decoding?

OpenH264 isn't yet fully suitable for media playback unfortunately. It was designed for WebRTC and low-latency. Support for anything other than the main profile is very recent too.
We also can't ship OpenH264, it must be available from Cisco in binary form for royalty fees calculation.

See also MPEG LA’s AVC License Will Not Charge Royalties for Internet Video that is Free to End Users through Life of License https://www.mpegla.com/wp-content/uploads/n-10-08-26.pdf

If only things were that black and white in real life. There's more than just MPEG-LA when it comes to H264/H265 patent pools.

The muxer guidelines indeed mention swapping should with must.
However:
-That is the muxer guidelines section, it does not apply to demuxers
-The patch I made is for playing files: hence demuxing is involved
-Files with h.264 exist and are supperior for streaming compared with mp4 bytestreams

In my opinion the muxing/demuxing debate comes down to: "Be conservative in what you send, be liberal in what you accept"

Attached patch webm_mp4.diff (obsolete) — — Splinter Review

I made a small change to the patch.
It now also accepts the "video/x-matroska" mime type.
This way files don't have to be labeled as 'webm', labeling them as matroska will be good enough. I think we can all agree that is a suiting name, even if we don't agree on whether it can be called 'webm'.
It even gives a bit more credit to the matroska creators...

Attachment #9075339 - Attachment is obsolete: true
Attached patch webm_mp4.diff (obsolete) — — Splinter Review

I updated the patch to only allow h.264 in a container reported to be a matroska one.

Attachment #9075600 - Attachment is obsolete: true

(In reply to Jean-Yves Avenard [:jya] from comment #14)

(In reply to guest271314 from comment #13)

So that sentence reads as "Video codec MUST be VP8 or VP9".

No, that is not what the current specification actually states.

From reading the specification here SHOULD means should where printed. If your reading is that the language SHOULD be be changed to MUST, then directly ask for those changes to the specification.

It's exactly what the specs states. SHOULD should be changed to MUST when it comes to a muxer. "Muxers should treat all guidelines marked SHOULD in this section as MUST" That's how it reads. I can't make it any clearer.

Does that also apply to Cues and SeekHead muxer guidelines; creating seekable WebM files that have a duration that is less than Infinity)https://bugs.chromium.org/p/chromium/issues/detail?id=642012)?

Attached patch webm_mp4.diff — — Splinter Review

Extend patch with a similar approach to AAC audio.
Note: both palemoon and chromium can play such files, palemoon even supports it for playing via the mediasource extensions. This brings firefox on par.

Attachment #9077358 - Attachment is obsolete: true

Re https://bugzilla.mozilla.org/show_bug.cgi?id=1562862#c11

and the file size is generally less when compared to VP8 or VP9

https://plnkr.co/edit/MDFCT5?p=preview

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: