1972093 - Some shopify applink video tests incorrectly given very fast times

Status: ASSIGNED

(Testing :: mozperftest, defect)

Description

[Andrew Creskey [:acreskey]]

Attachment: vid56_fenix.mp4

Running the shopify applink startup test locally, i.e. cvne-shopify.sh, I'm seeing a handful of fast outliers with incorrect timings.

Plotted here
https://zingy-sorbet-48e9a5.netlify.app/compare.html

When I look at the associated videos, they do not match the reported timings.
e.g. for the video attached, the reported time to completion was 1465.42222222222ms, from the baseline_h2 test run.

Seeing this for 4 videos of the 200 iterations (increasing the number of iterations to get statistically significant results)

Comment 1

[Andrew Creskey [:acreskey]]

This is almost certainly a dupe of bug 1970614

Comment 2

[Andrej (:aglavic)]

Attachment: Bug 1972093 - Fix video startup script inaccurate timings. r?#perftest