Open Bug 1637555 Opened 2 years ago Updated 2 years ago

Consider running smart scheduling on central and backstop pushes too (restricted to low value platforms)

Categories

(Firefox Build System :: Task Configuration, enhancement)

enhancement

Tracking

(Not tracked)

People

(Reporter: marco, Unassigned)

References

(Depends on 2 open bugs, Blocks 1 open bug)

Details

Me and :jmaher were discussing about platform-independent failures on Matrix and how to reduce the set of platforms on which we run platform-independent tests, especially on central and for low value platforms such as linux32.
Detecting what tests are platform-independent is not so easy, unless you are OK to accept a given rate of mistakes.
So we thought: since bugbug [smart scheduling] is handling these kind of things automatically using data / ML, what if we let it run on central for low value platforms?

On "low value" platforms, we can accept a failure detection rate which is not 100%, so we could enable smart scheduling on central only for those platforms.

The first step could be to run bugbug on lower value platforms on the autoland backstop and see what regressions we find on central. Then, if the rate is acceptable, we can move it to central and ride the trains.

I would go with a theory that linux is fastest, cheapest, and 80%+ of our tests will find regressions on linux, whereas <=20% of tests will have a good chance of finding regressions on other platforms (osx, windows 32, windows 64, android). One concern I have is that we find many regressions on a specific platform, but the fix is to edit/disable the test case. While it is important to know the test is mapped to source changes, it doesn't change the reality that these regressions are not resulting in changes to the product, the only reason we run tests.

I guess validating that theory with data we have would be useful. Applying it to autoland backstop and watching for lower value platform specific regressions on m-c (since m-c would run all tests, but regressions would show up ~24 hours later than normal) would give us data and confidence to move forward or adjust our assumptions about what is lower value.

Summary: Consider running smart scheduling on central (restricted to low value platforms) → Consider running smart scheduling on central and backstop pushes too (restricted to low value platforms)
Depends on: 1648723
Depends on: 1648724
Duplicate of this bug: 1635462
You need to log in before you can comment on or make changes to this bug.