1586505 - getVideoPlaybackQuality does not work with video from a webcam

Status: NEW

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Joe]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36

Steps to reproduce:

Run video.getVideoPlaybackQuality() when the video is showing a local webcam.

Actual results:

All the frame counts are 0.

Expected results:

The frame counts should be increasing over time is the same way as if playing a video file or accessing video.mozPaintedFrames.

Comment 1

[Joe]

I created a codepen to demonstrate the issue:

https://codepen.io/jproov/pen/qBBBRVN

Comment 2

[Alin Ilea, Desktop Test Engineering]

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Hi,

I have managed to reproduce this issue on latest FF release 69.0.2, latest Beta 70.0b13 and latest Nightly build 71.0a1 (2019-10-08) using Mac OS X 10.14 and Windows 10.
Further, I will move this over to a component so developers can take a look over it. If this is not the correct component please feel free to change it to an appropriate one.

Thanks for the report.

Comment 3

[Andreas Pehrson [:pehrsons]]

This method was originally speced only for MSE. That's why there's no support for MediaStreams. I filed a spec bug to clarify what this should report when playing a MediaStream with live video tracks.

Comment 4

[Paul Adenot (:padenot)]

Blocked on spec, P3.

Comment 5

[Joe]

I would say the most important stat for live video is the total frame count (which is currently only accessible with mozPaintedFrames). If the dropped and corrupted frame counts do not make sense in this case, I don't see an issue in having them report 0 (as they do now). It would be good to at least get the total frame count working with the new API.

Comment 6

[Joe]

I have just checked and Chrome does supported droppedVideoFrames as I see it as non 0 values on this test (you have to enable chrome://flags/#enable-experimental-web-platform-features):

https://codepen.io/jproov/pen/qBBBRVN

Is it worth looking at how Chrome calculate droppedVideoFrames and corruptedVideoFrames? The spec seems clear enough on the dropped count calculation and the corrupted count may simply not be relevant for live video:

https://w3c.github.io/media-playback-quality/#dfn-dropped-video-frame-count

Comment 7

[Andreas Pehrson [:pehrsons]]

(In reply to Joe from comment #6)

I have just checked and Chrome does supported droppedVideoFrames as I see it as non 0 values on this test (you have to enable chrome://flags/#enable-experimental-web-platform-features):

https://codepen.io/jproov/pen/qBBBRVN

Is it worth looking at how Chrome calculate droppedVideoFrames and corruptedVideoFrames? The spec seems clear enough on the dropped count calculation and the corrupted count may simply not be relevant for live video:

https://w3c.github.io/media-playback-quality/#dfn-dropped-video-frame-count

On the dropped count:
There's no concept of "predecode" for a MediaStreamTrack as the HTMLVideoElement is not handling decoding (if the source of the track happens to be an RTCPeerConnection, otherwise no decoding is involved at all). The spec needs to make clear how this should work.

The corrupt count is deprecated, so that seems irrelevant.

Comment 8

[Joe]

My understanding of predecode for live video is when the camera has supplied a frame. If this frame does not get displayed in the video element (because it missed the display deadline for example), I would consider it dropped.

In any case, it would be very useful to have totalVideoFrames working as this should be easy to count.

Comment 9

[Andreas Pehrson [:pehrsons]]

(In reply to Joe from comment #8)

My understanding of predecode for live video is when the camera has supplied a frame.

Where is this defined? What about a canvas? Another video element?

Comment 10

[Joe]

To me, predecode is the stage before being rendered for display. In the case of live video, this stage is the output from the camera.

Is it possible to have the source of a video element be a canvas or another video element?

Comment 11

[Joe]

There is an ongoing discussion for shipping this feature in Chrome:

https://bugs.chromium.org/p/chromium/issues/detail?id=644731

Maybe understanding their interpretation of totalVideoFrames for live video would be helpful for suggesting improvements to the spec?

Comment 12

[aw.vershinin]

Hi, I'm testing WebRTC application with playwright.dev and Firefox Nightly and I also expected it could be possible to check video playing by getVideoPlaybackQuality() and totalFrames amount.

Here is my test that is passed when using Chromium and failed when using Firefox:
https://github.com/viraxslot/firefox-fake-video-test

Hope this bug will be resolved.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I agree with comment 5 and comment 12 that totalVideoFrames is the useful part. E.g. instead of mochitests, we like writing WPT, but can't use:

const frames = v => v.mozPaintedFrames || v.webkitDecodedFrameCount;

We'd need to write WPTs using standard APIs, which would be:

const frames = v => v.getVideoPlaybackQuality().totalVideoFrames;

The "blocked on spec" comment 4 appears to only be about droppedVideoFrames, where returning 0 seems reasonable for lack of other info.

Chrome shipped this shortly after comment 11 and totalVideoFrames from MediaStreams works. Safari appears to return 0 like Firefox, but that seems for them to fix, and shouldn't block our progress or us writing WPT tests.

Comment 14

[beaufort.francois]

When fixing Chrome web platform tests at https://github.com/web-platform-tests/wpt/commit/21b6094665a8eb23769dac57f980848dc836ec2a, I realized I just realized tests are not broken on firefox.

I applied this diff which I guess should be changed (again) sadly to make it work on all browsers. Returning zero for totalVideoFrames is actually okay in Chromium.

-  const frames = () => v.getVideoPlaybackQuality()?.totalVideoFrames || v.mozPaintedFrames;
+  const frames = () => v.getVideoPlaybackQuality()?.totalVideoFrames ?? v.mozPaintedFrames;

Shall we use something like this instead?

const frames = () => v.mozPaintedFrames || v.getVideoPlaybackQuality().totalVideoFrames;

Comment 15

[beaufort.francois]

I've eventually fixed both Firefox and Chromium with https://github.com/web-platform-tests/wpt/commit/4047d18232b73b05a6f15667316398c66ed421b5
Let me know if that works for you.

Comment 16

[Paul Adenot (:padenot)]

Thanks François, this works for us, as in it allows testing what this tests wants to test, on both implementations. We'll eventually solve https://github.com/w3c/media-playback-quality/issues/19, but that doesn't need to block this.

Maybe Chromium folks can help answer the question we're asking in this issues, so that we can align all this sooner than later?

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

SGTM

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

Clear a needinfo that is pending on an inactive user.

Inactive users most likely will not respond; if the missing information is essential and cannot be collected another way, the bug maybe should be closed as INCOMPLETE.

For more information, please visit BugBot documentation.

Comment 19

[Paul Adenot (:padenot)]

We're keeping this open because we haven't solved the problem. It's still blocked on spec and there's been new development in the area, around stats.

Comment 20

[Paul Adenot (:padenot)]

302 jib, it's related to the stats work that is ongoing, but this particular bit might be stable in the spec.

Comment 21

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

It's somewhat related to track.stats (MediaStreamTrackVideoStats and MediaStreamTrackAudioStats w/some members lacking consensus), except right now stats are limited to "microphone audio tracks" and "camera and screenshare video tracks"... who knows if that will change, but yes it's at least a competing surface already for those sources.

But getVideoPlaybackQuality() is presumably still useful for e.g. remote peer connection tracks, not covered by track stats (at least not currently).

And at least when it comes to replacing mozPaintedFrames (comment 13) I think there's value here in comming to some agreement.

I haven't read super-carefully but naively I'd expect getVideoPlaybackQuality().totalVideoFrames to start counting from zero after about here:

video.srcObject = new MediaStream([track]);
await new Promise(r => video.onloadedmetadata = r);

vs. track.stats which count from once capture begins, e.g. after here:

await new Promise(r => transceiver.receiver.track.onunmute = r);

I.e. these counters will be different, either by some fixed offset in the simple case, or even wildly different depending how the application is passing the track around maybe reusing it in different video elements.

So maybe there is room for both?

Comment 22

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Uh my last comment assumed track.stats would be added to remote peer connection tracks... but imagine a similar example with local playback of a gUM track for an example that works today.