1379050 - Remove support for moz*Frames attributes from the video element

Status: REOPENED

(Core :: Audio/Video: Playback, task, P3)

Description

[Tim Huang[:timhuang]]

Per Bug 1369309 comment 19, we should remove moz*Frames support including ...
  * HTMLVideoElement.mozParsedFrames
  * HTMLVideoElement.mozDecodedFrames
  * HTMLVideoElement.mozPresentedFrames
  * HTMLVideoElement.mozPaintedFrames
  * HTMLVideoElement.mozFrameDelay

For two reasons, first, they are not official spec and no one implements them except us. Second, these APIs have a potential risk of browser fingerprinting.
For these reasons, we should remove them.

Comment 1

[Tim Huang[:timhuang]]

According to Jet's opinion and I have found that some addons are still using this API. So, we should remain this API around.

Comment 2

[Jean-Yves Avenard [:jya]]

It would be better to inform those addons to use the official VideoPlaybackQuality interface instead

https://wicg.github.io/media-playback-quality/

very unfortunate to not drop those redundant attributes that aren't per spec.

We've made addons totally incompatible in nightly recently and with much harder change into the API.

The change to use VideoPlaybackQuality is simple enough.

I believe we should reconsider this decision in not dropping it.

Comment 3

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to Tim Huang[:timhuang] from comment #1)
> According to Jet's opinion and I have found that some addons are still using
> this API. So, we should remain this API around.

These values should just be removed. They've been meaningless since we made the media stack async in 2014.

Comment 4

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Looks like we'll need to migrate WebRTC over to using https://developer.mozilla.org/en-US/docs/Web/API/VideoPlaybackQuality first.

Comment 5

[Nils Ohlmeier [:drno]]

I just commented out waitForPixelColor() from here http://searchfox.org/mozilla-central/rev/b7e531410ebc1c7d74a78e86a894e69608db318c/dom/canvas/test/captureStream_common.js#146 which appears to be one function still using mozPaintedFrames.
And with waitForPixelColor() not being available webrtc tests depending on video playback verification still pass. So I think our webrtc test don't depend on any of the above listed moz*Frames.

That being said it makes sense to add some telemetry to find out if there are WebRTC services out there using/depending on any of these.

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

waitForPixelColor() causes timeouts if mozPaintedFrames does not advance, so commenting it out disables the tests.

I like the direction, but see at least one obstacle:

In https://jsfiddle.net/jib1/cfcoqdwz/6/ v.getVideoPlaybackQuality().totalVideoFrames returns 0 in both release and Nightly.

Compare to the base version https://jsfiddle.net/jib1/cfcoqdwz using v.mozPaintedFrames || v.webkitDecodedFrameCount which works.

Do we have a bug for that? We'd need totalVideoFrames fixed before we can migrate WebRTC tests away from mozPaintedFrames.

Also note that in Chrome and Canary I get TypeError: v.getVideoPlaybackQuality is not a function...

I'd feel better about removing these once a stable web compatible alternative exists.

Comment 7

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #6)
> I'd feel better about removing these once a stable web compatible
> alternative exists.

Edge and Firefox already implement it. It's only busted on Chrome and Safari.

Comment 8

[Jean-Yves Avenard [:jya]]

totalVideoFrames is managed by the MediaFormatReader, which doesn't get involved with webRTC...

This is more a webRTC bug really... 

I'm fairly certain it's supported by Chrome, they were the one pushing for this API after all (along with Microsoft)

Comment 9

[Jean-Yves Avenard [:jya]]

if webRTC supports moz*Frames, adapting it to instead support the video playback quality should be fairly straight forward.

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

VideoPlaybackQuality is apparently behind a pref in Chrome https://bugs.chromium.org/p/chromium/issues/detail?id=644731

Unprefing appears blocked on this spec issue, which suggests VideoPlaybackQuality is legacy (?)
https://github.com/WICG/media-playback-quality/issues/7

Comment 11

[Tim Huang[:timhuang]]

Unsigned myself since I am no longer working on this.

Comment 12

[Jean-Yves Avenard [:jya]]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #10)

VideoPlaybackQuality is apparently behind a pref in Chrome
https://bugs.chromium.org/p/chromium/issues/detail?id=644731

Unprefing appears blocked on this spec issue, which suggests
VideoPlaybackQuality is legacy (?)
https://github.com/WICG/media-playback-quality/issues/7

it has.