1635921 - Compose |mach try auto| with other try selectors (e.g fuzzy)

Status: NEW

(Developer Infrastructure :: Try, enhancement, P3)

Description

[Andrew Halberstadt [:ahal]]

Currently the optimization algorithm that ./mach try auto uses chooses the platform configurations that the tests run on. Either via the ML algorithm, or via a more 'naive' method after the ML algorithm.

In certain cases, developers may only care about specific configurations. Either because they know it can't possibly affect anything else, or they are just focusing in on those configurations for now and will get to the others later.

Steve Fink had the idea of implementing ./mach try fuzzy --auto. This would work just like normal ./mach try fuzzy except:

1. We would remove chunks from the labels (so we only select configurations)
2. We would set the necessary parameters to trigger the ML optimizer in addition

There would likely be a fair amount of taskgraph work needed to support this. I'd probably be inclined to remove chunk numbers in the general case as well, so I think I'd like to depend on bug 1557848 first.

A similar approach could work for ./mach try chooser.

Comment 1

[Andrew Halberstadt [:ahal]]

My intuition is saying that we should also block on test path scheduling here. Though it's more of a blocker in the sense of the overall project rather than a hard blocker.

Priority wise it'll likely be finished first anyway, and once it is done the taskgraph portions of this bug will probably be easier to solve.

Comment 2

[Marco Castelluccio [:marco]]

This is very related to bug 1632571 (we might even consider this as an alternative to bug 1632571 and never fix bug 1632571).

Comment 3

[Jeff Muizelaar [:jrmuizel]]

I was going to filing a bug for this same thing and would find it useful. As a stop gap, is it possible to have ./mach try auto print the tests that it's going to run without running them?

Comment 4

[Andrew Halberstadt [:ahal]]

No, that's not (easily) possible. We don't run the ML until the decision task runs, so can't know what it will pick ahead of time. Even if we allowed uploads of arbitrary patches to the REST API, it would make your try pushes take 3-5 minutes. Solving this bug would be much easier.

Comment 5

[Matthew Gaudet (he/him) [:mgaudet]]

I have a slightly different use case for composition: I'd like to be able to do something like ./mach try fuzzy auto, where I get the fuzzy chooser, but then that's unioned with the set of tests mach try auto selects on push.

I'd like to ensure that a particular set of tests gets run, setting a lower bar, then letting auto find things that are useful that I might have missed.