1566773 - Fail YouTube playback performance tests that report 0 decoded frames

Status: RESOLVED DUPLICATE of bug 1625965

(Testing :: Raptor, enhancement, P3)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Currently we have some YouTube playback performance tests that are reporting 0 decoded frames and 0 dropped frames. As we're visualising the number of dropped frames, this is very misleading when no frames are decoded. If a video playback fails and there are 0 decoded frames then we should fail the test. We should first resolve bug 1558456, which disables tests that are known to fail on the Moto G5.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Interesting given that I was told that for Raptor tests we won't have failed tests, given that we measure performance data. Has something changed here, or does it mean we mark the full test job as failure on Treeherder? How would Perfherder handle such a situation?

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] (partly away July 22th - July 26th) from comment #1)

Interesting given that I was told that for Raptor tests we won't have failed tests, given that we measure performance data. Has something changed here, or does it mean we mark the full test job as failure on Treeherder? How would Perfherder handle such a situation?

This wouldn't be failing due to a performance issue, it would fail the entire job and not report results to Perfherder. Alternatively, we'd need our dashboards to only plot dropped frames when decoded frames > 0. My concern here is that decoded frames should never be 0, and that this currently goes unnoticed.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, that makes sense then. Thanks.

Comment 4

[Marian Raiciof [:marauder]]

I have created a pull request for some javascript changes, so we can stop the test when we first detect those 0 decoded frames:
https://github.com/mozilla/perf-youtube-playback/pull/5/files

Then we can fail the test from raptor.
Here i have two solutions:

1. Fail the test earlier in the process, when we receive the webext_results:
   https://hg.mozilla.org/try/rev/7e223de0895a2c8a570c7d3f43da753e0e34804c
   https://treeherder.mozilla.org/#/jobs?repo=try&revision=c40d9fd2a67c205beb728c7f24e55f783c7bd561

2. Or fail the test on output.py when we calculate the percent_dropped:
   https://hg.mozilla.org/try/rev/9c01e8cbe478829357c83aa5538f3dd9f03c2af5
   https://treeherder.mozilla.org/#/jobs?repo=try&revision=ca14370d82007e214b0e53e65e9f3c63cbe67b09

Comment 5

[Marian Raiciof [:marauder]]

Attachment: Bug 1566773 - Fail YouTube playback performance tests that report 0 decoded frames.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Marian, is your work here completely blocked on bug 1609373? If yes then please unassign and reset the priority to P3.

Comment 7

[Marian Raiciof [:marauder]]

Yes, the bug 1609373 will take care of zero decoded frames.
I almost finished the changes there.

So marking this as P3.

Thanks!

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Is this still needed?

Comment 9

[Marian Raiciof [:marauder]]

Hey Dave, Florin’s patch for Bug 1625965 , https://phabricator.services.mozilla.com/D69510 -> output.py line 638
has a code that will check for the "fail" flag in the results and the related commit on Github has been merged:
https://github.com/mozilla/perf-youtube-playback/commit/3041b1900573a964424e526c4e8a048a2e501d99

I think we should use Florin's patch to avoid duplicate solutions.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Marian Raiciof [:marauder] from comment #9)

I think we should use Florin's patch to avoid duplicate solutions.

So what does it mean for this bug? Obsolete or duplicate of bug 1625965?

Comment 11

[Marian Raiciof [:marauder]]

I'll mark this as duplicate of 1625965