1648719 - Evaluate if we should disable "mach try syntax" or not

Status: NEW

(Developer Infrastructure :: Try, task)

Description

[Sylvestre Ledru [:Sylvestre]]

It used to be hard to run testsuites.
Now, thanks to 'mach try auto' or 'mach try fuzzy', we have smarter way to run testsuites testing what we changed.

As "mach try syntax" can quickly increase cost without great ROI, maybe we should just deprecate it.

Comment 1

[Mike Hommey [:glandium]]

Not everyone who pushes to try want tests, though, and while it's not easier to run test suites, mach try fuzzy is not as convenient as mach try syntax to trigger a set of builds. Although the latter is broken now... I guess some presets for builds-only could be useful.

Comment 2

[Nathan Froyd [:froydnj]]

I dunno, try -b o -p win -u none -t none used to be a lot more convenient than mach try fuzzy for build- !signing !rusttests !ship win.

Comment 3

[Nathan Froyd [:froydnj]]

Alternatively, do we have any data on what people's typical attempts with try syntax vs. fuzzy vs. auto are from a cost perspective? Maybe this is skewed by people doing -p all -u all -t none for try syntax, or similar, but data would at least be more concrete than hand-waving about it being easier to DTRT with mach try fuzzy.

Comment 4

[Andrew Halberstadt [:ahal]]

Bug 1400295 would get mach try syntax to parse the syntax client side and would reduce maintenance burden in the taskgraph module (provided we disallow hg push -m "try: ..."). If that bug were solved, I wouldn't mind keeping try syntax around for the sole purpose of keeping some developers happy, there would be very few downsides to keeping it (especially if we remove the all alias). Though at the same time I'm very confident that any use cases of people using mach try syntax could instead be handled with in-tree presets. Plus the number of developers using it is declining rapidly (<15% of pushes now), so figuring out why people use try syntax and replacing those use cases with presets might be easiest in the long run.

(In reply to Nathan Froyd [:froydnj] from comment #2)

I dunno, try -b o -p win -u none -t none used to be a lot more convenient than mach try fuzzy for build- !signing !rusttests !ship win.

This is basically why try syntax is broken :D, it doesn't know about half the tasks we have and the above use case is just a coincidence that what it knows about is what you'd like to schedule.

Comment 5

[Tom Prince [:tomprince]]

would reduce maintenance burden in the taskgraph module

I'm not convinced that having the code in taskcluster/taskgraph/try_syntax.py vs tools/tryselect/selectors/syntax.py would change the maintenance burden. I think the burden comes from the impedance mismatch between syntax and the way we define tasks.

In particular:

This is basically why try syntax is broken :D, it doesn't know about half the tasks we have

is still true.

Comment 6

[Andrew Halberstadt [:ahal]]

It's been fairly low maintenance burden since we've more or less just stopped servicing try syntax requests. Though yeah, I guess the burden in taskgraph has been pretty trivial as well, so it was a bad argument.

Fyi, I just posted https://phabricator.services.mozilla.com/D81352 over on bug 1648791. See the commit message for usage. It should at least address the concern about not having a better way than try syntax to select builds.

Comment 7

[Tom Prince [:tomprince]]

Though yeah, I guess the burden in taskgraph has been pretty trivial as well, so it was a bad argument.

The main burden has been how it interacts with ci-cost work, which would be the same either place.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Nathan Froyd [:froydnj] from comment #2)

I dunno, try -b o -p win -u none -t none used to be a lot more convenient than mach try fuzzy for build- !signing !rusttests !ship win.

Can new presets help with this?

I tried to make some improvements to build selection with try syntax recently, and with the vast variety of different build types we now have, translating the intend behind a few characters of try syntax into the correct set of builds is really tricky. This, combined with the relatively low try syntax usage (https://gist.github.com/ahal/ebfc230994799517166fc32a04da8884) really makes me think we should deprecate it.

Comment 9

[Mike Hommey [:glandium]]

(In reply to [e-mail] (:bhearsum) from comment #8)

(In reply to Nathan Froyd [:froydnj] from comment #2)

I dunno, try -b o -p win -u none -t none used to be a lot more convenient than mach try fuzzy for build- !signing !rusttests !ship win.

Can new presets help with this?

If tasks wouldn't keep disappearing from the task set, it would. Currently using the (non-landed) build preset doesn't give you opt builds.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Mike Hommey [:glandium] from comment #9)

(In reply to [e-mail] (:bhearsum) from comment #8)

(In reply to Nathan Froyd [:froydnj] from comment #2)

I dunno, try -b o -p win -u none -t none used to be a lot more convenient than mach try fuzzy for build- !signing !rusttests !ship win.

Can new presets help with this?

If tasks wouldn't keep disappearing from the task set, it would. Currently using the (non-landed) build preset doesn't give you opt builds.

OK. They should be much less likely to break or change after we finish all of our task selection tweaks on central & autoland, and the bug you eluded to is fixed (which should be Very Soon).

I feel your frustration with that - but some of this is going to happen as we make sweeping changes like we've been doing for the past few months.

Comment 11

[Mike Hommey [:glandium]]

Can we wait for things to be ironed out before removing try syntax?

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Mike Hommey [:glandium] from comment #11)

Can we wait for things to be ironed out before removing try syntax?

I don't have any issue with that. I wasn't trying to suggest it should die right this moment (sorry it came off that way) - I just don't think it's worth spending time fixing at this point.