1396175 - Try runs talos tests on too many configurations by default

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Kris Maglione [:kmag]]

I understand why we need to do talos runs for QR and Stylo runs on checkin, but when I push to try with `-t all`, 12 Linux tests run on 4 configurations. If I use `--rebuild-talos 5` as is generally recommended, that spawns 250 jobs just for Linux. The end result is that I often have to wait several hours for any of the tests to even start, due to the huge backlogs this generates.

And, since the changes that I'm usually testing aren't affected by QR or Stylo, most of those tests aren't actually useful. Sometimes I take the time to painstakingly cancel the unnecessary extra runs, but usually it's just not worth the time it takes.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

should we have try specify platform configs in finer detail:
try -b o -p linux64,linux64-stylo,linux64-qr -u none -t all

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

We could remove try from the run-on-projects for some of those platforms, but that starts to get fiddly, and I suspect the QR and Stylo folks do want to see those results for their try pushes.

Try fuzzy might be useful here -- you could --save a `./mach try fuzzy` query that selected only the platforms that provided you the most value.

Comment 3

[Kris Maglione [:kmag]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #1)
> should we have try specify platform configs in finer detail:
> try -b o -p linux64,linux64-stylo,linux64-qr -u none -t all

That sounds like a good solution to me.

Comment 4

[Kris Maglione [:kmag]]

Although, it's probably worth noting that this is an issue on Windows, too, albeit a smaller one, since those tests only run in two configurations. The proposed platform solution wouldn't work there, since those tests run on the same "platform", but in a different group (Ts-e10s vs. T-e10s).

It would be nice to have a way to select more specific talos sets there, too, since our Windows slaves tend to be even more resource constrained than the Linux ones.

Comment 5

[Kris Maglione [:kmag]]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> We could remove try from the run-on-projects for some of those platforms,
> but that starts to get fiddly, and I suspect the QR and Stylo folks do want
> to see those results for their try pushes.

To be clear, I'm not suggesting that we shouldn't *be able to* run QR/Stylo talos tests on try. That's definitely important for the people working on those features. I just don't think they should run by default, since we all wind up paying for the extra load that generates, QR/Stylo people included.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I believe |mach try fuzzy| would be the best resource for this, unfortunately it doesn't work with --rebuild X, so maybe we should get that supported first (bug 1414919)