Bugzilla

(In reply to Hamish Willee from comment #8)
> What docs work in MDN do you think needs to be done for this? 
> 
> My understanding is that 
> [`HTMLMediaElement.canPlayType()`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/canPlayType), [`MediaCapabilities.decodingInfo()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaCapabilities/decodingInfo) and [`MediaSource.isTypeSupported()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaSource/isTypeSupported) all take (either directly or as some configuration) a type. This is in most cases referred to as a mime type but in the examples in the docs also includes codec types - e.g. `var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';`
> 
> So previously this codec string was ignored for AV1 codec strings. My understanding is that now the string will be parsed so the user will get a more accurate answer as to whether the file can be played. 
> - Is that right?

That is largely right. For Firefox, previously, it would just treat all codec strings starting with "av01" as av1, ignoring all codec configuration options that could be passed that _may_ affect support for playback, and _will_ in most systems affect hardware decoding support, which is returned by MediaCapabilities.decodingInfo() as the 'powerEfficient' boolean. In most cases, simply the profile specified by the codec string will determine whether hardware decoding is available, but the whole codecs parameter string must follow the spec provided here: https://aomediacodec.github.io/av1-isobmff/#codecsparam

> - Is it just AV1 strings that are now parsed? Are strings for other codecs still ignored?

Three codecs are now fully parsed according to spec:
- AVC/H264
- VP9
- AV1

This page seems to have a fairly accurate view of how those ought to be formatted: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter

Only correction I would suggest to make there is for AV1, change the format string from "av01.P.LLT.DD[.M[.CCC[.cp[.tc[.mc[.F]]]]]]" to "av01.P.LLT.DD[.M.CCC.cp.tc.mc.F]". Chrome is not consistent with spec on this and accepts any number of parameters after ensuring that parameters up to bit depth are included, but as stated in the AV1 ISO-BMFF spec, all the optional fields should be treated as all or none:

>The parameters sample entry 4CC, profile, level, tier, and bitDepth are all mandatory fields. If any of these fields are empty, or not within their allowed range, the processing device SHOULD treat it as an error.
>All the other fields (including their leading '.') are optional, mutually inclusive (all or none) fields. If not specified then the values listed in the table below are assumed.

> From a docs point of view we want to make it clear anything that a developer has to/can now do differently. Looking at those docs it looks like we say that the codec string can be used. So what actually do web developers need to know about this change for the docs. We could perhaps make the existence of the codec strings a bit more obvious in the parameters section. 

The difference for people trying to support Firefox should only be that the browser will now accurately return values for support for both playback and efficient decoding of AV1 videos, based on the codec parameters provided. Previously, it would not even return whether hardware decoding was supported, but now it will.

However, it would be good to ensure that all the pages involving codecs parameters (HTMLVideoElement "type" attribute, and the functions you mentioned).

> Is there a required [browser compatibility change](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/canPlayType#browser_compatibility)? Normally you'd add an entry if two browsers do something markedly different when passed the same inputs.  In this case I'm not sure - yes we'll be parsing the strings a bit better, but to end users that will mostly mean they'll just get a better indication of support, but still not "guarantees".

After the change in this bug, the only notable difference between all the Chromium-based browsers and Gecko's handling of AV1 codec strings now is that Gecko requires all optional parameters to be specified if any one optional parameter is included, where Chrome will accept any number of optional parameters.

Another (much less important) thing that Chromium-based browsers ignore is that the AV1 spec allows for unexpected characters in the codec string, as long as all the required parameters are specified (up to bit depth, or the full set of optional parameters). When Chromium encounters those unexpected characters, parsing will fail rather than ignoring the characters. The quote from the AV1 spec:

> If any character that is not '.', digits, part of the AV1 4CC, or a tier value is encountered, the string SHALL be interpreted ignoring all the characters starting from that character.

Finally, feel free to reference the attached [test document](https://bug1757861.bmoattachments.org/attachment.cgi?id=9271759) to see how this behavior compares between browsers, I've put quite a few different examples of configurations in there that should help to visualize the behaviors and differences between them. Note in that document that some codec strings are marked as "Invalid", in the cases where browsers accept those, that is likely because it doesn't check for parameters that break spec or contradict other parameters (for example profiles with subsampling modes that are unsupported by those profiles). Those cases should be considered bugs, but don't affect the developer experience usually, since they can and should create codecs strings from the parameters they pass to the encoder they are using, which will guarantee that they are correct.

Let me know if you have any more questions or clarifications, hopefully that explains a good amount of this, it honestly gets quite deep into each codec's specification so it's a bit dense.

(In reply to Hamish Willee from comment #8)
> What docs work in MDN do you think needs to be done for this? 
> 
> My understanding is that 
> [`HTMLMediaElement.canPlayType()`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/canPlayType), [`MediaCapabilities.decodingInfo()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaCapabilities/decodingInfo) and [`MediaSource.isTypeSupported()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaSource/isTypeSupported) all take (either directly or as some configuration) a type. This is in most cases referred to as a mime type but in the examples in the docs also includes codec types - e.g. `var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';`
> 
> So previously this codec string was ignored for AV1 codec strings. My understanding is that now the string will be parsed so the user will get a more accurate answer as to whether the file can be played. 
> - Is that right?

That is largely right. For Firefox, previously, it would just treat all codec strings starting with "av01" as av1, ignoring all codec configuration options that could be passed that _may_ affect support for playback, and _will_ in most systems affect hardware decoding support, which is returned by MediaCapabilities.decodingInfo() as the 'powerEfficient' boolean. In most cases, simply the profile specified by the codec string will determine whether hardware decoding is available, but the whole codecs parameter string must follow the spec provided here: https://aomediacodec.github.io/av1-isobmff/#codecsparam

(In reply to Hamish Willee from comment #8)
> - Is it just AV1 strings that are now parsed? Are strings for other codecs still ignored?

Three codecs are now fully parsed according to spec:
- AVC/H264
- VP9
- AV1

This page seems to have a fairly accurate view of how those ought to be formatted: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter

Only correction I would suggest to make there is for AV1, change the format string from "av01.P.LLT.DD[.M[.CCC[.cp[.tc[.mc[.F]]]]]]" to "av01.P.LLT.DD[.M.CCC.cp.tc.mc.F]". Chrome is not consistent with spec on this and accepts any number of parameters after ensuring that parameters up to bit depth are included, but as stated in the AV1 ISO-BMFF spec, all the optional fields should be treated as all or none:

>The parameters sample entry 4CC, profile, level, tier, and bitDepth are all mandatory fields. If any of these fields are empty, or not within their allowed range, the processing device SHOULD treat it as an error.
>All the other fields (including their leading '.') are optional, mutually inclusive (all or none) fields. If not specified then the values listed in the table below are assumed.

(In reply to Hamish Willee from comment #8)
> From a docs point of view we want to make it clear anything that a developer has to/can now do differently. Looking at those docs it looks like we say that the codec string can be used. So what actually do web developers need to know about this change for the docs. We could perhaps make the existence of the codec strings a bit more obvious in the parameters section. 

The difference for people trying to support Firefox should only be that the browser will now accurately return values for support for both playback and efficient decoding of AV1 videos, based on the codec parameters provided. Previously, it would not even return whether hardware decoding was supported, but now it will.

However, it would be good to ensure that all the pages involving codecs parameters (HTMLVideoElement "type" attribute, and the functions you mentioned).

(In reply to Hamish Willee from comment #8)
> Is there a required [browser compatibility change](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/canPlayType#browser_compatibility)? Normally you'd add an entry if two browsers do something markedly different when passed the same inputs.  In this case I'm not sure - yes we'll be parsing the strings a bit better, but to end users that will mostly mean they'll just get a better indication of support, but still not "guarantees".

After the change in this bug, the only notable difference between all the Chromium-based browsers and Gecko's handling of AV1 codec strings now is that Gecko requires all optional parameters to be specified if any one optional parameter is included, where Chrome will accept any number of optional parameters.

Another (much less important) thing that Chromium-based browsers ignore is that the AV1 spec allows for unexpected characters in the codec string, as long as all the required parameters are specified (up to bit depth, or the full set of optional parameters). When Chromium encounters those unexpected characters, parsing will fail rather than ignoring the characters. The quote from the AV1 spec:

> If any character that is not '.', digits, part of the AV1 4CC, or a tier value is encountered, the string SHALL be interpreted ignoring all the characters starting from that character.

Finally, feel free to reference the attached [test document](https://bug1757861.bmoattachments.org/attachment.cgi?id=9271759) to see how this behavior compares between browsers, I've put quite a few different examples of configurations in there that should help to visualize the behaviors and differences between them. Note in that document that some codec strings are marked as "Invalid", in the cases where browsers accept those, that is likely because it doesn't check for parameters that break spec or contradict other parameters (for example profiles with subsampling modes that are unsupported by those profiles). Those cases should be considered bugs, but don't affect the developer experience usually, since they can and should create codecs strings from the parameters they pass to the encoder they are using, which will guarantee that they are correct.

Let me know if you have any more questions or clarifications, hopefully that explains a good amount of this, it honestly gets quite deep into each codec's specification so it's a bit dense.