Incompatibility with H.264/AVC Level 5.2 when using MSE API

RESOLVED FIXED in Firefox 60

Status

()

defect
P2
normal
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: bucher, Assigned: jya)

Tracking

58 Branch
mozilla60
Points:
---

Firefox Tracking Flags

(firefox60 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

a year ago
User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36

Steps to reproduce:

It appears that Firefox might have some problem with decoding H.264 stream with AVC level 5.2 when processed by MSE with or without GPU acceleration enabled.

We have run into an issue where our 3840x2160p@50 HLS stream (declaring itself based on its properties as AVC level 5.2 and codec avc1.4d4034) is not being decoded by Firefox (tested versions 52, 56, 58 on various OSs with various GPUs). The browser always complains about unsupported codec.

If we halve the fps to 25 and thus decrease the AVC level to 5.1, the HLS stream can be then decoded properly.

This website confirms the lack of support for 5.2 when opened in Firefox https://cconcolato.github.io/media-mime-support/#video/mp4;%20codecs=%22avc1.4d4034%22 and not just Main profile, but Baseline, too.
However, the same content from the stream muxed without re-encoding with the same properties as regular mp4 container used within a standard <video> tag can be decoded without any issue on platforms that support it (only Win 7 without GPU acceleration couldn't do it) which makes me think that the underlying issue is likely somewhere within MSE.

All other major desktop browsers are fine in this regard, they can process the stream and their media mime support test results on the aforementioned website are ok, too.

I hope that this is nothing more than just some oversight in Firefox's MSE implementation that could be easily fixed. Unfortunately, I am not sure if we will be able to provide you with some sample stream for debugging, but that media mime support testing website should be a good start as the stream is seemingly refused just based on its codec info right away without an attempt to be decoded at all.


Actual results:

H.264/AVC Level 5.2 unplayable when using MSE API.


Expected results:

H.264/AVC Level 5.2 playable when using MSE API.
(Reporter)

Comment 1

a year ago
Just to confirm the theory, if I replace codec descriptor avc1.4d4034 aka Main profile Level 5.2 in generated manifest with codec avc1.4d4033 aka Main profile Level 5.1, Firefox is happy to load the stream and play it even though the bitstream is still Level 5.2.

So the issue is truly not with the content itself, but rather with Level 5.2 missing in Firefox's supported codecs enumerator for MSE as the https://cconcolato.github.io/media-mime-support/#video/mp4;%20codecs=%22avc1.4d4034%22 website seems to suggest.

Updated

a year ago
Component: Untriaged → Audio/Video
Product: Firefox → Core
Chris, do you think we should extend our AVC profile guesstimate to include 5.2?
Component: Audio/Video → Audio/Video: Playback
Flags: needinfo?(cpearce)
Priority: -- → P2
It's may be as simple as changing IsWhitelistedH264Codec() in MP4Decoder.cpp. I'll defer this one to jya however, he's got more knowledge of our low level codec support these days.
Flags: needinfo?(cpearce) → needinfo?(jyavenard)
(Reporter)

Comment 4

a year ago
All right, I can now see the reasoning behind the decision. There's a comment in MP4Decoder.cpp that basically states that AVC is limited to level 5.1 due to Microsoft being the only one providing some form of documentation for decoder support. This limit for Windows is then used in Firefox on other platforms as well.

While referenced documentation https://msdn.microsoft.com/en-us/library/windows/desktop/dd797815%28v=vs.85%29.aspx stipulates 5.1 as limit, there is another document https://msdn.microsoft.com/en-us/library/windows/desktop/dd318775%28v=vs.85%29.aspx that says 5.2 has been supported since Windows 8.1, although that document is a bit ambiguous about whether it relates to decoding, too.

As opposed to Win 7, our tests in Firefox with Level 5.2 content and fake codec descriptor on Win 10 with the same hardware were 100% successful so there is some merit to the info that modern Windows starting with Windows 8.1 have better support for UHD.
We have also tested 5.2 content on High Sierra successfully, both with hardware acceleration enabled and disabled. Unfortunately, we have no access to Linux machines with GPUs, so tests were done successfully on Ubuntu 16.04 with software rendering only.

Generally, as I understand it, there is no obstacle in adding support for Level 5.2 to Firefox, it should be then matter of utilizing GPUs on platforms that can leverage it unless that particular GPU is blacklisted. Everything else could be otherwise done by CPU as a best effort the way it is done with lower levels. Platforms that simply do not support 5.2 should not be any different from what is happening now, browser will likely throw MEDIA_ERR_SRC_NOT_SUPPORTED or player will just load the audio track from the stream - it would not cause any detrimental effect when compared to current version of Firefox.
(Assignee)

Comment 5

a year ago
I think we should just allow it, if there's a decoding error later that will be handled by the MFR and the decoder
Flags: needinfo?(jyavenard)
(Assignee)

Updated

a year ago
Assignee: nobody → jyavenard
Comment hidden (mozreview-request)

Comment 7

a year ago
mozreview-review
Comment on attachment 8955045 [details]
Bug 1437003 - Allow H264 level up to 5.2 inclusive.

https://reviewboard.mozilla.org/r/224222/#review230204

Not too qualified for this one, but reading the various messages in the bug, this makes sense.
Attachment #8955045 - Flags: review?(padenot) → review+

Comment 8

a year ago
Pushed by jyavenard@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8e31ba435fa8
Allow H264 level up to 5.2 inclusive. r=padenot

Comment 9

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/8e31ba435fa8
Status: UNCONFIRMED → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
You need to log in before you can comment on or make changes to this bug.