1532254 - pushes to Try without changes don't run all the requested jobs (usually for measuring talos baseline performance)

Status: RESOLVED WONTFIX

(Firefox Build System :: Task Configuration, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Try pushes without changes to measure the baseline performance optimize away many of the requested jobs.

Example:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bbdd832ca5fb4c0af6b717265cf6cf7c5abf4659

Try syntax triggers the jobs if there are code changes:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0eecfe6b6a7f94ee915400a76688d684524e4ec9

As does a change to a code comment:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5b086c71372327ec9139304f3166a1b87f5dfba

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

in the above example there is a target-tasks.json which shows all the talos jobs:
https://taskcluster-artifacts.net/LGls6pg0Qe6kF7rQyRWR3w/0/public/target-tasks.json

between this and the raw log for the decision task, I do not see any evidence that we have problem during the decision task. I am curious what code would say "there are 0 lines of code changes, do not run anything"

Comment 2

[Mark Banner (:standard8)]

Extracted From a discussion in #ci last week:

Standard8: So I’ve just triggered two sets of try builds with the same command… the first only has Mac builds, the second has everything expected… I’m confused
Standard8: https://treeherder.mozilla.org/#/jobs?repo=…32ca5fb4c0af6b717265cf6cf7c5abf4659
Standard8: https://treeherder.mozilla.org/#/jobs?repo=…e6b6a7f94ee915400a76688d684524e4ec9
Standard8: unless it was the completely empty base changeset? but that would be weird...
catlee: Callek, tomprince: any ideas? looks like the target tasks are correct at first glance
tomprince: catlee: Still looking, but first guess is fallout from 'optimize_target_tasks` change.
catlee: yeah, that's what I'm wondering
tomprince: catlee: treeherder.mozilla.org/logviewer.ht…=230795087&repo=try&lineNumber=3023 suggests that is the case.
Aryx: the one with more jobs only removes 43 tasks, the first one 45
Callek: That's my guess too
catlee: boo

Comment 3

[:Gijs (he/him)]

(In reply to Mark Banner (:standard8) from comment #2)

tomprince: catlee: Still looking, but first guess is fallout from 'optimize_target_tasks` change.

This is going to mess with productivity for folks. Which change is that, and who could look at getting this fixed so explicitly requested talos jobs continue to run?

Comment 4

[Tom Prince [:tomprince]]

This was deliberate change, and I think the solution is going to be, use mach try fuzzy if you need more control over what runs.

:catlee is probably better to ask about this, as the change was made to reduce CI load.

Comment 5

[:Gijs (he/him)]

FWIW, it doesn't help that when you then "add new jobs" and/or retriggers to get the talos runs anyway, we end up triggering a build job for every talos job, cf. this push https://treeherder.mozilla.org/#/jobs?repo=try&revision=c0daea63a35c52f99728bf5436eadee7c5a7967a .

Comment 6

[Chris AtLee [:catlee]]

Yeah, I would recommend mach try fuzzy to get exactly what you want.

I think maybe ./mach try fuzzy --artifact -q 'win64talos | linux64talos | mactalos' --rebuild 5 will reproduce what you wanted from the try push above?

Comment 7

[Mark Banner (:standard8)]

TBH I find fuzzy unintuitive and difficult to use, for starters something like the command in comment 6 would assume that I know exactly how the various tests are named.

My preferred way going forward is probably going to be mach try chooser.

I can see other people potentially being hit by this - either it should work in all modes, or there should be warnings where it doesn't.

Comment 8

[:Gijs (he/him)]

(In reply to Chris AtLee [:catlee] from comment #6)

Yeah, I would recommend mach try fuzzy to get exactly what you want.

So does mach try chooser not work? Why is it different from fuzzy?

Comment 9

[Andrew Halberstadt [:ahal]]

(In reply to :Gijs (he/him) from comment #8)

So does mach try chooser not work? Why is it different from fuzzy?

mach try chooser should behave identically to fuzzy as far as this bug is concerned. If anyone is seeing differently, please let me know.

Also worth noting that this bug will be automatically fixed by bug 1400295, though not sure what the timeline for that bug is (I'm hoping sometime in 2019).

Comment 10

[Tom Prince [:tomprince]]

I think this is something that isn't going to be fixed with try syntax.