1227017 - Add MediaSource.isTypeSupported() support for video width, height, and bitrate parameters

Status: NEW

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Jean-Yves Avenard [:jya]]

This would be particularly useful with youtube which uses MediaSource.IsTypeSupported to query if a particular codec is supported.

Example:
IsTypeSupported(aType=video/webm; codecs="vp9")OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=640)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=99999)[not supported] 
IsTypeSupported(aType=video/webm; codecs="vp9")OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; height=360)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; height=99999)[not supported] 
IsTypeSupported(aType=video/webm; codecs="vp9")OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; framerate=30)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; framerate=9999)OK 
IsTypeSupported(aType=video/webm; codecs="vp9")OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; bitrate=300000)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; bitrate=2000000000)OK 
IsTypeSupported(aType=video/mp4; codecs="avc1.4d4015"; width=426; height=240)OK 
IsTypeSupported(aType=video/mp4; codecs="avc1.4d401e"; width=640; height=360)OK 
IsTypeSupported(aType=video/mp4; codecs="avc1.4d401e"; width=854; height=480)OK 
IsTypeSupported(aType=video/mp4; codecs="avc1.4d401f"; width=1280; height=720)OK 
IsTypeSupported(aType=video/mp4; codecs="avc1.640028"; width=1920; height=1080)OK 
IsTypeSupported(aType=audio/mp4; codecs="mp4a.40.2")OK 
IsTypeSupported(aType=video/mp4; codecs="avc1.4d400c"; width=256; height=144)OK 
IsTypeSupported(aType=audio/webm; codecs="vorbis")OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=426; height=240)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=640; height=360)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=854; height=480)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=1280; height=720)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=1920; height=1080)OK 
IsTypeSupported(aType=video/webm; codecs="vp9"; width=256; height=144)OK 

An extra parameter is provided: width or height . If we know a particular resolution can't be decoded properly or fast enough we should report "NO" there.

YouTube appears to use the decoder that supports the highest resolution supported for a particular video. E.g. if the highest resolution provided by a video is 1080p and webm reports as only supporting 720p while h264 reports supporing 1080p: h264 will be used.

Comment 1

[Chris Peterson [:cpeterson]]

The MSE spec just says isTypeSupported() should take a valid MIME type string, which may include some combination of media type, media subtype, and codecs. It doesn't describe the width, height, and bitrate parameters that YouTube is using, but these parameters would be very useful because the VPx codec strings don't provide profile information. We currently have no way to tell the website that a user's laptop can play 1080p H.264 (in hardware) and 720p VP9 (in software), but can't keep up with 1080p VP9.

https://w3c.github.io/media-source/#widl-MediaSource-isTypeSupported-boolean-DOMString-type

Comment 2

[Jean-Yves Avenard [:jya]]

The MSE spec clearly states on what conditions we should return "NO"

"5. If the MediaSource does not support the specified combination of media type, media subtype, and codecs then return false."

it doesn't state that we can return "NO" based on any mimetype parameters.

The MSE spec must be amended if we want to support this (which would be very useful)

Comment 3

[Chris Peterson [:cpeterson]]

YouTube says these codec parameters were an extension for smart TVs and are not implemented in desktop Chrome. They are interested in standardizing something in this area to improve content support checks.

Comment 4

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

If we want to roll VP9 out to more users we could return 'no' based the following limits:

>=75 FPS => 720p60
>=40 FPS => 720p30

Comment 5

[Soledad Penades [:sole] [:spenades]]

need-infoing Eric for updating the MDN doc pages which are currently mentioning the old canRecord... method

Comment 6

[Eric Shepherd [:sheppy]]

(In reply to Soledad Penades [:sole] [:spenades] from comment #5)
> need-infoing Eric for updating the MDN doc pages which are currently
> mentioning the old canRecord... method

So... MediaSource.isTypeSupported() is already documented.

However, I do see MediaRecord.canRecordMimeType() instead, which does need to be fixed. I will slot that in as soon as I can.

Comment 7

[Jean-Yves Perrier [:teoli]]

Note that we should likely wait until this lands before documenting it.

Comment 8

[Eric Shepherd [:sheppy]]

(In reply to Jean-Yves Perrier [:teoli] from comment #7)
> Note that we should likely wait until this lands before documenting it.

Yes, documenting these parameters should wait. However, :sole is correct that the canRecordMimeType method no longer exists on MediaRecord; that's already implemented, so this should be updated on MDN, here: 

https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/canRecordMimeType

and

https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder should be updated to link to a new page https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/isTypeSupported which documents that method.

Comment 9

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Mass change P2 -> P3

Comment 10

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Bug 1176218 implements the support for YouTube parameters (and uses them to disallow resolutions that we know are too high for different versions of Windows.)

I think we can keep this bug open to work on documentation, and standardization (or not) of these parameters.

Comment 11

[Andrey Bienkowski]

https://bugzilla.mozilla.org/show_bug.cgi?id=1548706