1757861 - Add AV1 codecs parameter string parsing for media support queries.

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Zaggy1024]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0

Actual results:

AV1 codec strings (i.e. "av01.0.09M.08") passed to HTMLVideoElement.canPlayType, MediaSource.isTypeSupported() and MediaCapabilities.decodingInfo() are not parsed for the parameters that are used by the decoders.

Currently, this only seriously affects MediaCapabilities, as the response from decodingInfo will always use default decoder settings, and not be properly guaranteeing that a specific profile, bit depth, etc. is supported. This may cause some hardware decoders to fail after a site attempts to verify support before playback.

Expected results:

AV1 codec strings should be parsed and validated, then passed to decoders via extradata fields. By doing this, hardware support (or lack thereof) in MediaCapabilities can be detected.

This will also allow invalid codec strings to be rejected, as Chrome currently does.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Zaggy1024]

Attachment: Bug 1757861 - Part 1 - Transition VideoColorSpace to use gfx and gfx::CICP color space enums. r=alwu

AV1 codec strings passed to MediaCapabilities, HTMLVideoElement.canPlayType and MediaSource.isTypeSupported will now be parsed fully and passed to decoders as av1C data. Before and during playback, MediaChangeMonitor is used to parse AV1 samples and detect when new decoders are needed.

This is the first step to allow detection of the hardware's capability to play specific AV1 configurations.

Depends on D140695

Comment 3

[Zaggy1024]

Attachment: Bug 1757861 - Part 2 - Add AV1 codec string parsing for MediaCapabilities, etc. decoding support checks. r=alwu

This adds both av1C and AV1 sequence header OBU parsing, which is used by MediaChangeMonitor to determine whether a new decoder is needed when the codec configuration changes during playback.

AV1 codec strings from MediaCapabilities, HTMLVideoElement.canPlayType and MediaSource.isTypeSupported will be converted to av1C format with a sequence header attached and passed to decoders to initialize them, allowing better detection of codec support in software or hardware to be implemented.

Depends on D141072

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/08bf7c943aa2
Part 1 - Transition VideoColorSpace to use gfx and gfx::CICP color space enums. r=alwu
https://hg.mozilla.org/integration/autoland/rev/6f97064ebae7
Part 2 - Add AV1 codec string parsing for MediaCapabilities, etc. decoding support checks. r=alwu

Comment 5

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/08bf7c943aa2
https://hg.mozilla.org/mozilla-central/rev/6f97064ebae7

Comment 6

[Alastor Wu [:alwu]]

Starting from Fx101, AV1 codec string (i.e. "av01.0.09M.08") sent to HTMLVideoElement.canPlayType(), MediaSource.isTypeSupported() and MediaCapabilities.decodingInfo() would be parsed correctly.

Comment 7

[Zaggy1024]

Attachment: Video support test.html

Attaching an HTML test doc with many different examples on it in case it is useful.

Comment 8

[Hamish Willee]

FYI FF101 docs work for this can be tracked here: https://github.com/mdn/content/issues/15463

What docs work in MDN do you think needs to be done for this?

My understanding is that
HTMLMediaElement.canPlayType(), MediaCapabilities.decodingInfo() and MediaSource.isTypeSupported() all take (either directly or as some configuration) a type. This is in most cases referred to as a mime type but in the examples in the docs also includes codec types - e.g. var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';

So previously this codec string was ignored for AV1 codec strings. My understanding is that now the string will be parsed so the user will get a more accurate answer as to whether the file can be played.

• Is that right?
• Is it just AV1 strings that are now parsed? Are strings for other codecs still ignored?

From a docs point of view we want to make it clear anything that a developer has to/can now do differently. Looking at those docs it looks like we say that the codec string can be used. So what actually do web developers need to know about this change for the docs. We could perhaps make the existence of the codec strings a bit more obvious in the parameters section.

Is there a required browser compatibility change? Normally you'd add an entry if two browsers do something markedly different when passed the same inputs. In this case I'm not sure - yes we'll be parsing the strings a bit better, but to end users that will mostly mean they'll just get a better indication of support, but still not "guarantees".

Comment 9

[Zaggy1024]

(In reply to Hamish Willee from comment #8)

What docs work in MDN do you think needs to be done for this?

My understanding is that
HTMLMediaElement.canPlayType(), MediaCapabilities.decodingInfo() and MediaSource.isTypeSupported() all take (either directly or as some configuration) a type. This is in most cases referred to as a mime type but in the examples in the docs also includes codec types - e.g. var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';

So previously this codec string was ignored for AV1 codec strings. My understanding is that now the string will be parsed so the user will get a more accurate answer as to whether the file can be played.

• Is that right?

That is largely right. For Firefox, previously, it would just treat all codec strings starting with "av01" as av1, ignoring all codec configuration options that could be passed that may affect support for playback, and will in most systems affect hardware decoding support, which is returned by MediaCapabilities.decodingInfo() as the 'powerEfficient' boolean. In most cases, simply the profile specified by the codec string will determine whether hardware decoding is available, but the whole codecs parameter string must follow the spec provided here: https://aomediacodec.github.io/av1-isobmff/#codecsparam

(In reply to Hamish Willee from comment #8)

• Is it just AV1 strings that are now parsed? Are strings for other codecs still ignored?

Three codecs are now fully parsed according to spec:

• AVC/H264
• VP9
• AV1

This page seems to have a fairly accurate view of how those ought to be formatted: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter

Only correction I would suggest to make there is for AV1, change the format string from "av01.P.LLT.DD[.M[.CCC[.cp[.tc[.mc[.F]]]]]]" to "av01.P.LLT.DD[.M.CCC.cp.tc.mc.F]". Chrome is not consistent with spec on this and accepts any number of parameters after ensuring that parameters up to bit depth are included, but as stated in the AV1 ISO-BMFF spec, all the optional fields should be treated as all or none:

The parameters sample entry 4CC, profile, level, tier, and bitDepth are all mandatory fields. If any of these fields are empty, or not within their allowed range, the processing device SHOULD treat it as an error.
All the other fields (including their leading '.') are optional, mutually inclusive (all or none) fields. If not specified then the values listed in the table below are assumed.

(In reply to Hamish Willee from comment #8)

From a docs point of view we want to make it clear anything that a developer has to/can now do differently. Looking at those docs it looks like we say that the codec string can be used. So what actually do web developers need to know about this change for the docs. We could perhaps make the existence of the codec strings a bit more obvious in the parameters section.

The difference for people trying to support Firefox should only be that the browser will now accurately return values for support for both playback and efficient decoding of AV1 videos, based on the codec parameters provided. Previously, it would not even return whether hardware decoding was supported, but now it will.

However, it would be good to ensure that all the pages involving codecs parameters (HTMLVideoElement "type" attribute, and the functions you mentioned).

(In reply to Hamish Willee from comment #8)

Is there a required browser compatibility change? Normally you'd add an entry if two browsers do something markedly different when passed the same inputs. In this case I'm not sure - yes we'll be parsing the strings a bit better, but to end users that will mostly mean they'll just get a better indication of support, but still not "guarantees".

After the change in this bug, the only notable difference between all the Chromium-based browsers and Gecko's handling of AV1 codec strings now is that Gecko requires all optional parameters to be specified if any one optional parameter is included, where Chrome will accept any number of optional parameters.

Another (much less important) thing that Chromium-based browsers ignore is that the AV1 spec allows for unexpected characters in the codec string, as long as all the required parameters are specified (up to bit depth, or the full set of optional parameters). When Chromium encounters those unexpected characters, parsing will fail rather than ignoring the characters. The quote from the AV1 spec:

If any character that is not '.', digits, part of the AV1 4CC, or a tier value is encountered, the string SHALL be interpreted ignoring all the characters starting from that character.

Finally, feel free to reference the attached test document to see how this behavior compares between browsers, I've put quite a few different examples of configurations in there that should help to visualize the behaviors and differences between them. Note in that document that some codec strings are marked as "Invalid", in the cases where browsers accept those, that is likely because it doesn't check for parameters that break spec or contradict other parameters (for example profiles with subsampling modes that are unsupported by those profiles). Those cases should be considered bugs, but don't affect the developer experience usually, since they can and should create codecs strings from the parameters they pass to the encoder they are using, which will guarantee that they are correct.

Let me know if you have any more questions or clarifications, hopefully that explains a good amount of this, it honestly gets quite deep into each codec's specification so it's a bit dense.

Comment 10

[Hamish Willee]

Hi Zaggy,
Thanks so much for that help. FYI only:

• I'm still trying to work out the right way to do compatibility data.
• The docs are done in https://github.com/mdn/content/pull/15813. Basically fixed up https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter as you suggested and cross linked it from the places where the parameter can be set. As an aside, I also integrated all the dictionaries, which we we no longer have "on there own" in MDN docs. So this area is a bit tidier now.