1499822 - [tryselect] Implement |mach try chooser|

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, enhancement)

Description

[Andrew Halberstadt [:ahal]]

While |mach try fuzzy| is great for a lot of things, one area it is worse that try syntax is that you need to *know* what you are looking for. With try syntax, you can open up the trychooser web page and see at a glance what the options are, whereas there is no such luxury with fuzzy.

However, trychooser isn't great because it needs to be updated manually and is always horribly out of date. If you rely on it, you'll inevitably miss running tasks that you maybe should have been running.

To get the best of both worlds, I'm proposing a new selector |mach try chooser|. This tool would do a few things:

1. Build the task graph (like fuzzy does)

2. Dynamically compute the tasks that can be selected. We can do this by looking at various keys in the task graph like "kind", "suite", "tags", etc. Some hardcoding will be required to manage the different types of tasks here, but once a type is supported, new tasks will be added automatically added.

3. Run a flask app that displays a "trychooser" like page in the browser based on the computed tasks.

4. Provide a "Push" button which will schedule all selected options (no more need for copy/pasting syntax strings).


One thing to note, is that this "trychooser" page would not need to generate any try syntax on the backend. We could filter the actual taskgraph in real time as the user checks/unchecks items, we could even provide a preview of all the task labels the combination of checkboxes would schedule (a la try fuzzy).

Comment 1

[(not currently active) Ted Mielczarek]

This sounds like a really cool idea! The set of tasks we run has gotten so large that I think there's a lot of room for experimenting with new ways to visualize and filter them. For my usecases I often want to run either all types of builds for one platform, or builds of a certain type on all platforms. I'd love to have an interface similar to trychooser like:

Build Type
==========
[ ] opt [ ] debug [ ] pgo [ ] asan [ ] valgrind [ ] rusttest ...

Platform
========
[ ] win32 [ ] win64 [ ] linux [ ] linux64 [ ] mac [ ] android ...

Test Suites
===========
[ ] mochitest [ ] xpcshell [ ] reftest ...

Preferably with "Select All" / "Clear All" buttons for each section. I suspect people would probably want a way to filter test suites down by chunk as well. I'm not sure how you'd best present that. If we do chunking differently per-platform / build type it might also be nice to expose the "select tests by path" functionality that `mach try syntax` has. I can see a use for "I'd like to run test_whatever in debug builds on these platforms."

Comment 2

[Andrew Halberstadt [:ahal]]

I have a working prototype already :). Hoping to polish it up a bit more in time for the demo sessions at the all hands.

My current idea is to use (nicely styled) multiselects instead of checkboxes, so shift/ctrl+clicking will work. There's also a live preview window (similar to fuzzy), which updates with the task labels that would get selected as you click.

I haven't implemented chunking support yet, but my current idea is to have a text input inside the multiselect <option> tags. It would work similar to printer page setups (e.g "1-3,5"). The maximum number would be bounded by whatever platform has the most chunks for that suite.

I'm not sure if mixing build types and build variants is going to be possible using the way the prototype is currently set up, but I think we could add a new section:

Build Type
==========
[] opt [] debug

Build Variant
=============
[] asan [] pgo [] ccov ...

Platform
========
[] win32 [] win64 ...

Comment 3

[Andrew Halberstadt [:ahal]]

After chatting with Armen, I think it might be better to try and re-write this on react. However since I have things mostly working already, I'm tempted to get what I have landed first (without advertising) so people can start experimenting and providing feedback.

Worth noting that I've never used react before, so this might not be a small undertaking for me.

Comment 4

[(not currently active) Ted Mielczarek]

I agree that landing what you currently have as a first cut and making improvements from there makes sense. (Even if you wind up rewriting it in react.)

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1499822 - [taskgraph] Add a tag to help differentiate 'android-stuff' from builds, r?jmaher

This will be used by |mach try chooser| to help pull 'android-stuff' tasks
(which aren't really builds) apart from the main build tasks.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1499822 - [tryselect] Implement |mach try chooser|

Usage:

  $ ./mach try chooser

Will start a local flask server and server a "trychooser-like" page
that is dynamically generated from the taskgraph.

Depends on D14902

Comment 7

[Andrew Halberstadt [:ahal]]

Hey Armen, I know you are heading on your leave soon and this is a large complicated patch. So if you don't have time feel free to unassign yourself and I'll find someone else to review. That being said, I'm not expecting a super thorough review here.

We talked about using React for this at the all hands, and I do still agree that's the way to go. But since my old hacky way is already working, and I'm not sure how long a React re-write will take, I'd like to get what I have landed first.

I also plan on adding docs and tests before landing, but I won't have that done before you leave, so I'll do it in a new commit and get someone else to review that in the new year.

My apologies for making you look at this mess :)

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1499822 - [lint] Enable codespell on tools/tryselect, r?jmaher

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f4bd2ef9b729
[taskgraph] Add a tag to help differentiate 'android-stuff' from builds, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/9a330195eca6
[lint] Enable codespell on tools/tryselect, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/beb286aeae33
[tryselect] Implement |mach try chooser| r=sclements

Comment 10

[Andreea Pavel [:apavel]]

Backed out for breaking gecko decision task.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&searchStr=gecko%2Cdecision%2Ctask%2Copt%2Cgecko%2Cdecision%2Ctask%2C%28d%29&selectedJob=220690676

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=220690676&repo=autoland&lineNumber=3956

Backout: https://hg.mozilla.org/integration/autoland/rev/5ecacfcd6e65fe5a1dee88b40ef00ca20c0282e8

From IRC:

tomprince> apavel|sheriffduty: Should backout https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=9a330195eca6 for that

<apavel|sheriffduty> ok, however the gecko task after that push is green. it failes only on the merge

<tomprince> Yeah, it is only going to fail when android-stuff tasks are run.

<apavel|sheriffduty> tomprince: ok, thank you for looking

<tomprince> But it will fail everytime they are scheduled to run.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f8bb48d6a644
[taskgraph] Add a tag to help differentiate 'android-stuff' from builds, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/c2b4cd16c551
[lint] Enable codespell on tools/tryselect, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/31e16c2d9429
[tryselect] Implement |mach try chooser| r=sclements

Comment 12

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/f8bb48d6a644
https://hg.mozilla.org/mozilla-central/rev/c2b4cd16c551
https://hg.mozilla.org/mozilla-central/rev/31e16c2d9429

Comment 13

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/94af6f1447b7
https://hg.mozilla.org/releases/mozilla-esr60/rev/dedb59aa77fd