1437003 - Incompatibility with H.264/AVC Level 5.2 when using MSE API

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Tomas Bucher]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36

Steps to reproduce:

It appears that Firefox might have some problem with decoding H.264 stream with AVC level 5.2 when processed by MSE with or without GPU acceleration enabled.

We have run into an issue where our 3840x2160p@50 HLS stream (declaring itself based on its properties as AVC level 5.2 and codec avc1.4d4034) is not being decoded by Firefox (tested versions 52, 56, 58 on various OSs with various GPUs). The browser always complains about unsupported codec.

If we halve the fps to 25 and thus decrease the AVC level to 5.1, the HLS stream can be then decoded properly.

This website confirms the lack of support for 5.2 when opened in Firefox https://cconcolato.github.io/media-mime-support/#video/mp4;%20codecs=%22avc1.4d4034%22 and not just Main profile, but Baseline, too.
However, the same content from the stream muxed without re-encoding with the same properties as regular mp4 container used within a standard <video> tag can be decoded without any issue on platforms that support it (only Win 7 without GPU acceleration couldn't do it) which makes me think that the underlying issue is likely somewhere within MSE.

All other major desktop browsers are fine in this regard, they can process the stream and their media mime support test results on the aforementioned website are ok, too.

I hope that this is nothing more than just some oversight in Firefox's MSE implementation that could be easily fixed. Unfortunately, I am not sure if we will be able to provide you with some sample stream for debugging, but that media mime support testing website should be a good start as the stream is seemingly refused just based on its codec info right away without an attempt to be decoded at all.


Actual results:

H.264/AVC Level 5.2 unplayable when using MSE API.


Expected results:

H.264/AVC Level 5.2 playable when using MSE API.

Comment 1

[Tomas Bucher]

Just to confirm the theory, if I replace codec descriptor avc1.4d4034 aka Main profile Level 5.2 in generated manifest with codec avc1.4d4033 aka Main profile Level 5.1, Firefox is happy to load the stream and play it even though the bitstream is still Level 5.2.

So the issue is truly not with the content itself, but rather with Level 5.2 missing in Firefox's supported codecs enumerator for MSE as the https://cconcolato.github.io/media-mime-support/#video/mp4;%20codecs=%22avc1.4d4034%22 website seems to suggest.

Comment 2

[Ralph Giles (:rillian)]

Chris, do you think we should extend our AVC profile guesstimate to include 5.2?

Comment 3

[Chris Pearce [:cpearce (Not reading bugmail)]]

It's may be as simple as changing IsWhitelistedH264Codec() in MP4Decoder.cpp. I'll defer this one to jya however, he's got more knowledge of our low level codec support these days.

Comment 4

[Tomas Bucher]

All right, I can now see the reasoning behind the decision. There's a comment in MP4Decoder.cpp that basically states that AVC is limited to level 5.1 due to Microsoft being the only one providing some form of documentation for decoder support. This limit for Windows is then used in Firefox on other platforms as well.

While referenced documentation https://msdn.microsoft.com/en-us/library/windows/desktop/dd797815%28v=vs.85%29.aspx stipulates 5.1 as limit, there is another document https://msdn.microsoft.com/en-us/library/windows/desktop/dd318775%28v=vs.85%29.aspx that says 5.2 has been supported since Windows 8.1, although that document is a bit ambiguous about whether it relates to decoding, too.

As opposed to Win 7, our tests in Firefox with Level 5.2 content and fake codec descriptor on Win 10 with the same hardware were 100% successful so there is some merit to the info that modern Windows starting with Windows 8.1 have better support for UHD.
We have also tested 5.2 content on High Sierra successfully, both with hardware acceleration enabled and disabled. Unfortunately, we have no access to Linux machines with GPUs, so tests were done successfully on Ubuntu 16.04 with software rendering only.

Generally, as I understand it, there is no obstacle in adding support for Level 5.2 to Firefox, it should be then matter of utilizing GPUs on platforms that can leverage it unless that particular GPU is blacklisted. Everything else could be otherwise done by CPU as a best effort the way it is done with lower levels. Platforms that simply do not support 5.2 should not be any different from what is happening now, browser will likely throw MEDIA_ERR_SRC_NOT_SUPPORTED or player will just load the audio track from the stream - it would not cause any detrimental effect when compared to current version of Firefox.

Comment 5

[Jean-Yves Avenard [:jya]]

I think we should just allow it, if there's a decoding error later that will be handled by the MFR and the decoder

Comment 6

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1437003 - Allow H264 level up to 5.2 inclusive.

Review commit: https://reviewboard.mozilla.org/r/224222/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/224222/

Comment 7

[Paul Adenot (:padenot)]

Comment on attachment 8955045 [details]
Bug 1437003 - Allow H264 level up to 5.2 inclusive.

https://reviewboard.mozilla.org/r/224222/#review230204

Not too qualified for this one, but reading the various messages in the bug, this makes sense.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8e31ba435fa8
Allow H264 level up to 5.2 inclusive. r=padenot

Comment 9

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/8e31ba435fa8