747095 - Run multiple iterations of tests and average results

Status: RESOLVED FIXED

(Testing Graveyard :: Peptest, defect)

Description

[Mark Côté [:mcote]]

We should be able to reduce the variance in results by running each test several times and taking the average, as Talos does. We can also drop the biggest outlier(s) as well.

This will require a change to peptest itself to take a configurable number of iterations (since we may need to play with the number), and to the results server to do the averaging. In any case, we want to make sure we save all the results, even if we only display the average, so that we can experiment with averaging techniques.

Comment 1

[Mark Côté [:mcote]]

Attachment: option for number of test iterations

This adds the --iterations option to repeat test execution. Also changed all headers to MPL 2.0.

This will probably cause a conflict just because I checked in the version bump separately, but it's a tiny change.

This is the same patch as pull request https://github.com/mozilla/peptest/pull/2. I'll merge the request in myself after getting an r+ here.

Comment 2

[cmtalbert]

Comment on attachment 616822 [details] [diff] [review]
option for number of test iterations

reviewed on github as part of the pull request. R+'d and merged.

Comment 3

[cmtalbert]

Merged into peptest repo: https://github.com/mozilla/peptest/commit/ee880973fbf6551d17055fbbd08489e463700487

Comment 4

[Mark Côté [:mcote]]

Actually I want to keep this open until we have buildbot running multiple iterations. I'll file separate blocking bugs for that.

Comment 5

[Mark Côté [:mcote]]

Code to average results before displaying is done and deployed (https://github.com/markrcote/peptest-metrics/commit/32b578592a29116c6326b739e89150e5cf4467ad)

Comment 6

[Mark Côté [:mcote]]

Peptests are now being run 10 times each with every push to try, m-i, and m-c. Still may need to tweak the averaging algorithm, but this is essentially done.