1042550 - [meta] Auto Regression Detection System

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Performance, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

Bugs to make sure that we don't have to bisect anymore. A few things:

1) We want to make sure we test every gaia commit on datazilla
2) When a Gecko commit occurs, auto bisect gecko to find the offending commit
3) Tune fxos-perf alerts so that they are less sensitive and more effective.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I really like this plan and goal, it makes a lot of sense and should reduce the work required.

A few concerns that could be concerns in the future but are not now:
* if we double the runtime of our performance tests
* if we increase the branches we require active testing on
* if we increase the platforms (flame will not be around forever) we require testing on
* if there is a regression or a problem with all the devices/network and it takes a couple days to manually get them flashed/online and running again

All of these scenarios are things we have seen on the android and desktop browsers and have somehow cropped up on a recurring basis for years.

The most dangerous pieces to increased load (backlog due to one of the 4 reasons above) is requiring testing for every commit.  It could take a week to play catch up, at that point you are better off doing a manual bisection.  My only advice would be that as these systems are built/tweaked we have knobs to adjust the requirements to account for 5x the current load per device.

Comment 2

[Mason Chang [Inactive] [:mchang]]

Hi Joel,

Good notes, thanks for the experience. Yeah I don't imagine that we will be able to keep up with every Gecko commit any time soon, but at least for now, this is the current game plan to test every gaia commit, "somewhat" auto-manually bisect gecko commits, and strictly on master for now. I want to limit the scope to what we're currently doing, then we can expand more later if devices allow.

Comment 4

[Mason Chang [Inactive] [:mchang]]

Reopen if need be.