1171604 - [raptor] decide which apps to run the launch test against based on code changes

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Robert Wood [:rwood]]

Build some intelligence into the raptor decision code: decide which apps to run the launch test against, based upon the code changes.

We could run the test on all apps by default (current behaviour) except if a push only touches one (or more apps) then only run the launch test on the effected apps.

Or we could not run the launch test by default and only run it when app code is changed, and only on the apps whose code was touched.

This would reduce the number of raptor tasks on each push; however potentially miss some regressions where non-app specific code changes inadvertently increase launch time.

Eli, James, what do you think?

Comment 1

[:Eli Perelman]

If we are speaking of Gaia PRs:

- I think we should always run *all* the launch tests if the change occurs in /apps/verticalhome, /apps/system, /shared, or any other directory that could have far-reaching consequences, e.g. maybe settings, search.
- For meta-only-changes like Marionette, Raptor, tests, etc., maybe we don't run any launch tests.
- For app-specific changes, we only run the launch test for the apps in question, e.g. changes only in /apps/clock run only the launch test for Clock.

Thoughts?

Comment 2

[James Lal [:lightsofapollo]]

I am +1 on having some basic way to limit the number of tests on PRs with two exceptions:

 - we should have it configurable in tree
 - there should be a documented way to override the reduced set to run them all

Also we should always run full set on master/branch/etc..

Comment 3

[Robert Wood [:rwood]]

With the Raptor on gaia-ci refactoring done in Bug 1176237 there is only one Raptor task per app now by default, greatly reducing the number of tasks we were running originally. Therefore I don't believe the amount of work that would be required for this particular change, would be worth the value.