Run multiple iterations of tests and average results

RESOLVED FIXED

Status

Testing Graveyard
Peptest
RESOLVED FIXED
6 years ago
28 days ago

People

(Reporter: mcote, Assigned: mcote)

Tracking

Details

Attachments

(1 attachment)

(Assignee)

Description

6 years ago
We should be able to reduce the variance in results by running each test several times and taking the average, as Talos does. We can also drop the biggest outlier(s) as well.

This will require a change to peptest itself to take a configurable number of iterations (since we may need to play with the number), and to the results server to do the averaging. In any case, we want to make sure we save all the results, even if we only display the average, so that we can experiment with averaging techniques.
(Assignee)

Comment 1

6 years ago
Created attachment 616822 [details] [diff] [review]
option for number of test iterations

This adds the --iterations option to repeat test execution. Also changed all headers to MPL 2.0.

This will probably cause a conflict just because I checked in the version bump separately, but it's a tiny change.

This is the same patch as pull request https://github.com/mozilla/peptest/pull/2. I'll merge the request in myself after getting an r+ here.
Attachment #616822 - Flags: review?(halbersa)
(Assignee)

Updated

6 years ago
Attachment #616822 - Flags: review?(halbersa) → review?(ctalbert)

Comment 2

6 years ago
Comment on attachment 616822 [details] [diff] [review]
option for number of test iterations

reviewed on github as part of the pull request. R+'d and merged.
Attachment #616822 - Flags: review?(ctalbert) → review+

Comment 3

6 years ago
Merged into peptest repo: https://github.com/mozilla/peptest/commit/ee880973fbf6551d17055fbbd08489e463700487
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Assignee)

Comment 4

6 years ago
Actually I want to keep this open until we have buildbot running multiple iterations. I'll file separate blocking bugs for that.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Updated

6 years ago
Depends on: 749275
(Assignee)

Comment 5

6 years ago
Code to average results before displaying is done and deployed (https://github.com/markrcote/peptest-metrics/commit/32b578592a29116c6326b739e89150e5cf4467ad)
(Assignee)

Comment 6

6 years ago
Peptests are now being run 10 times each with every push to try, m-i, and m-c. Still may need to tweak the averaging algorithm, but this is essentially done.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED

Updated

28 days ago
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.