1163088 - [raptor] run gaia raptor suite 6x right off, and use median results

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Robert Wood [:rwood]]

When the raptor emulator tests are running and reporting to treeherder, we want to be sure a flagged regression is a valid regression. Running the test once on the reference build and once on the tester build, comparing the results (and having an apparent regression appear) is not always a definitive result due to environmental factors, etc.

In order to make the results more reliable, implement raptor suite repeats. Run the suite once (launch test of 30 runs, on reference build and on tester build). If the results indicate no regression, finish and mark the result task green.

If the first suite results indicate a possible regression, then repeat the suite 6 more times concurrently (on the same builds). Then calculate the median results for the reference build and test build from all six runs, and compare. If this result indicates a regression, flag the result task as red, otherwise green. This way an indicated regression is well proven before being flagged (thanks for the input :jmaher).

Comment 1

[Robert Wood [:rwood]]

As discussed with James, instead of waiting for the first suite to finish and then kicking off more runs if a regression is detected, just launch the suite 6x right from the start. Use the median results to check for a performance regression. This way, since each suite runs concurrently, the testing time won't be extended, but yet we will be confident the regression is valid if flagged.

Comment 2

[Robert Wood [:rwood]]

Attachment: https://github.com/rwood-moz/raptor-results/pull/2

Update raptor-results docker image code to:
- receive results for a variable number of suite runs (variable number of base and patch taskIds
- use the median result in each set (base and patch results) to check for the performance regression

Comment 3

[:Eli Perelman]

Comment on attachment 8607728 [details] [review]
https://github.com/rwood-moz/raptor-results/pull/2

r+ with comments addressed.

Comment 4

[Robert Wood [:rwood]]

Thanks Eli. The raptor-results docker image portion of this bug has been merged:

https://github.com/rwood-moz/raptor-results/commit/e8191e782cc24160e1456267b535177285280403

Comment 5

[GitHub Autolander Bot]

Attachment: [gaia] rwood-moz:bug1163088 > mozilla-b2g:master

Comment 6

[Robert Wood [:rwood]]

Raptor-results image, didn't commit the post-review updates; did that now:

https://github.com/rwood-moz/raptor-results/commit/12a286f039c85417368e7df226269ecc812c5ca7

Comment 7

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/gaia/pull/30179

Taskcluter raptor decision task code, update to generate appropriate raptor tasks for N number of suite iterations

Comment 8

[Robert Wood [:rwood]]

Landed myself, don't know why autolander doesn't seem to work for my raptor patches.

https://github.com/mozilla-b2g/gaia/commit/ea07a22e99979fbce2df447df73caf76c351909e