[meta] Auto Regression Detection System

RESOLVED WONTFIX

Status

Firefox OS
Performance
P1
normal
RESOLVED WONTFIX
4 years ago
4 years ago

People

(Reporter: mchang, Assigned: mchang)

Tracking

({perf})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [c=automation p=4 s= u=])

(Assignee)

Description

4 years ago
Bugs to make sure that we don't have to bisect anymore. A few things:

1) We want to make sure we test every gaia commit on datazilla
2) When a Gecko commit occurs, auto bisect gecko to find the offending commit
3) Tune fxos-perf alerts so that they are less sensitive and more effective.
(Assignee)

Updated

4 years ago
Blocks: 1042551
(Assignee)

Updated

4 years ago
No longer blocks: 1042551
Depends on: 1042551
(Assignee)

Updated

4 years ago
Depends on: 1042553
(Assignee)

Updated

4 years ago
Depends on: 1042555
I really like this plan and goal, it makes a lot of sense and should reduce the work required.

A few concerns that could be concerns in the future but are not now:
* if we double the runtime of our performance tests
* if we increase the branches we require active testing on
* if we increase the platforms (flame will not be around forever) we require testing on
* if there is a regression or a problem with all the devices/network and it takes a couple days to manually get them flashed/online and running again

All of these scenarios are things we have seen on the android and desktop browsers and have somehow cropped up on a recurring basis for years.

The most dangerous pieces to increased load (backlog due to one of the 4 reasons above) is requiring testing for every commit.  It could take a week to play catch up, at that point you are better off doing a manual bisection.  My only advice would be that as these systems are built/tweaked we have knobs to adjust the requirements to account for 5x the current load per device.
(Assignee)

Comment 2

4 years ago
Hi Joel,

Good notes, thanks for the experience. Yeah I don't imagine that we will be able to keep up with every Gecko commit any time soon, but at least for now, this is the current game plan to test every gaia commit, "somewhat" auto-manually bisect gecko commits, and strictly on master for now. I want to limit the scope to what we're currently doing, then we can expand more later if devices allow.
(Assignee)

Updated

4 years ago
Duplicate of this bug: 1011567
(Assignee)

Updated

4 years ago
Depends on: 1011570
(Assignee)

Updated

4 years ago
Assignee: nobody → mchang
Whiteboard: [c=automation p= s= u=] → [c=automation p=4 s= u=]
(Assignee)

Updated

4 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 4

4 years ago
Reopen if need be.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.