1545087 - [meta] Status of dropped frames for Playback Performance Tests (2019) on FireTV stick 4k

Status: RESOLVED INACTIVE

(Testing :: Performance, task, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Reference test results for Firefox (WebView)

This bug will be used to share the current drop frame rate of GV on FireTV stick 4k as long as we cannot report the status via Raptor to perfherder.

I will try to regularly run those tests as best on a daily basis so that we can identify major bugs, and regressions as early as possible. Those I will add as dependencies to this bug.

As already attached you can find the reference test results for Firefox as based on WV. It's basically all passing.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Note that the first two measurements will be done against the site as given by Google which is:

https://ytlr-cert.appspot.com/2019/main.html?test_type=playbackperf-test

But given that Tarek created a mirror of those tests (bug 1542688) for our own infrastructure I might continue with:

http://yttest.dev.mozaws.net/2019/main.html?test_type=playbackperf-test

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: 20190417_GV_nightly

Status update for GV example browser from 2019-04-17. 1x playback tests failed due to dropped frames are:

• VP9.2160p30@1X: 2/455 => 0.4%
• VP9.*p60@1X: >40/900 => ~4.4%

Also when I ran the tests unattended the geckoview_example browser crashed a couple of times. I will try to check this tomorrow.

Comment 3

[Vicky Chin [:vchin]]

FYI cpearce. Does this align with what you'd expect. This is just from one run as I understand it. How would you like the dropped frames reported here handled? Would you like bugs to be opened?

Comment 4

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Vicky Chin [:vchin] from comment #3)

FYI cpearce. Does this align with what you'd expect. This is just from one run as I understand it. How would you like the dropped frames reported here handled? Would you like bugs to be opened?

I have been seeing playback fail randomly (bug 1542956), and we have bugs covering slow paints (bug 1543625, bug 1545031), so I was going to wait to see if fixing those makes us perform better on YouTube's certification suite.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: 20190424_GV_nightly

As it looks like we are pretty stable for the 1x playback rate and at maximum 30fps. For those tests we are below the 4% drop frame rate for Q2, even with some slight variations in results.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The first results from running the tests locally via Raptor but muted can be found here:

https://gist.github.com/whimboo/3b3295d2de1105b20c70f10180dd78f4

There is a suite value of 24.51, which says that we have a mean of 24.5 dropped frames per test.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The interesting subtest results for the 2160p30@1x video are:

   {

      "lowerIsBetter": true,
      "replicates": [
        0
      ],
      "value": 0.0,
      "name": "VP9.2160p30@1X_dropped_frames",
      "unit": "score"
    },
    {
      "lowerIsBetter": false,
      "replicates": [
        453
      ],
      "value": 453.0,
      "name": "VP9.2160p30@1X_decoded_frames",
      "unit": "score"
    },
    {
      "lowerIsBetter": true,
      "replicates": [
        0.0
      ],
      "value": 0.0,
      "name": "VP9.2160p30@1X_%_dropped_frames",
      "unit": "score"
    },

Means there are zero frame drops.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Eric, not sure if this bug is still needed or not, or if it provides any value for you. Now that the performance tests can be run locally, everyone could run it on their own. CI support is still blocked by no available testing hardware.

Shall we close this bug?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No reply from anyone, so I assume this bug isn't that important anymore.

Given that I move back to my team next week, I will put this back into the general bucket, and will leave it open for Dave, Rob, or anyone else to pick-up on it.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

There's been no further work on FireTV performance. Closing as incomplete.