[raptor] decide which apps to run the launch test against based on code changes

RESOLVED WONTFIX

Status

Firefox OS
Gaia::PerformanceTest
RESOLVED WONTFIX
3 years ago
3 years ago

People

(Reporter: rwood, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
Build some intelligence into the raptor decision code: decide which apps to run the launch test against, based upon the code changes.

We could run the test on all apps by default (current behaviour) except if a push only touches one (or more apps) then only run the launch test on the effected apps.

Or we could not run the launch test by default and only run it when app code is changed, and only on the apps whose code was touched.

This would reduce the number of raptor tasks on each push; however potentially miss some regressions where non-app specific code changes inadvertently increase launch time.

Eli, James, what do you think?
(Reporter)

Updated

3 years ago
Flags: needinfo?(eperelman)
(Reporter)

Updated

3 years ago
Flags: needinfo?(jlal)

Comment 1

3 years ago
If we are speaking of Gaia PRs:

- I think we should always run *all* the launch tests if the change occurs in /apps/verticalhome, /apps/system, /shared, or any other directory that could have far-reaching consequences, e.g. maybe settings, search.
- For meta-only-changes like Marionette, Raptor, tests, etc., maybe we don't run any launch tests.
- For app-specific changes, we only run the launch test for the apps in question, e.g. changes only in /apps/clock run only the launch test for Clock.

Thoughts?
Flags: needinfo?(eperelman)
I am +1 on having some basic way to limit the number of tests on PRs with two exceptions:

 - we should have it configurable in tree
 - there should be a documented way to override the reduced set to run them all

Also we should always run full set on master/branch/etc..
Flags: needinfo?(jlal)
(Reporter)

Comment 3

3 years ago
With the Raptor on gaia-ci refactoring done in Bug 1176237 there is only one Raptor task per app now by default, greatly reducing the number of tasks we were running originally. Therefore I don't believe the amount of work that would be required for this particular change, would be worth the value.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.