1176237 - [raptor] reduce variance to help eliminate falsely-detected regressions on gaia-ci

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Robert Wood [:rwood]]

The raptor launch test is running on all apps on gaia-ci, but not currently posting to treeherder due to variance and false alarms. Investigate and try ways to reduce variance and ideally have zero falsely-detected performance regressions.

The variance between aws instances, even though the same instance type, is the core cause. Confirmed by :ekyle by plotting raw raptor data.

Try running the base and base+patch tasks on the same instance instead of separate instances for each, that will help. Also when taskcluster support for ramdisks is available, running the emulator on a ramdisk will help also.

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] rwood-moz:bug1176237 > mozilla-b2g:master

Comment 2

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/gaia/pull/30882

Change the raptor graph decision code to create a single raptor task per app. Use the new raptor-gaia docker image. Once this is merged, I will be requesting the treeherder folks to hide the newly-named raptor tasks until the tweaks are completed.

Comment 3

[Robert Wood [:rwood]]

Comment on attachment 8631232 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/30882

Greg is on PTO, John would you have some time for a review please? This is step 1, changing to a single task per app for the raptor launch test. Once this is merged I'll be hiding the tasks on treeherder until it is finished up.

Comment 4

[Robert Wood [:rwood]]

Comment on attachment 8631232 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/30882

Back to Greg as he's back and has done the Raptor-related reviews in the past, thanks

Comment 5

[Greg Arndt [:garndt]]

Comment on attachment 8631232 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/30882

looks good, if this works for you then sheeep it

side note: noticed that in your new raptor docker image there is an entrypoint specified that doesn't appear to exist unless I'm overlooking it.

Comment 6

[Robert Wood [:rwood]]

Thanks Greg. Landed: https://github.com/mozilla-b2g/gaia/commit/c2c798dd2faadc49c5440abaa60fb2ea3f57113f

Comment 7

[Robert Wood [:rwood]]

Update: In the last 50 Raptor suite runs on gaia and gaia-master, there have been zero falsely-flagged regressions. Also, the test of introducing an intentional app launch regression and having Raptor catch it, was a success (see Bug 1171167). So far these results look good, a definite improvement. Awaiting feedback from :ekyle on the data also, and will keep an eye on the trees.

Comment 8

[Robert Wood [:rwood]]

Received detailed feedback from :ekyle on some analysis he did on some of the Raptor gaia-ci data. Taking that feedback into consideration, and looking at the Raptor suite runs on gaia-ci thus far, I believe the refactored automation is looking good.

Due to the nature of running performance tests on the emulator, in docker, on aws taskcluster workers, etc, there will always be some variance and therefore we are limited to the sensitivity of regression that can be detected. With the current setup, number of app launches, and automatic retries, it looks like detecting a 15% or higher regression on gaia-ci is valid. Regressions smaller than that should be detected on the Raptor tests running on-device post-commit.

There will always be room for improvement and we can make tweaks down the road, for instance when ramdisk support is added to taskcluster workers. The new Raptor tasks are now visible on treeherder (gaia and gaia-master) by default.