1412121 - [tryselect] Attempt to download taskgraph from CI instead of generating it

Status: RESOLVED WONTFIX

(Developer Infrastructure :: Try, enhancement)

Description

[Andrew Halberstadt [:ahal]]

I noticed task graph generation seems to have slowed down, it takes ~40 seconds on my machine. Instead we could download a taskgraph from CI, which only takes ~20 seconds for me (largely due to calling into vcs several times).

If for some reason we can't find a taskgraph artifact to download, or something goes wrong trying to find the parent revision, we can fallback to generating the taskgraph.

Comment 1

[Andrew Halberstadt [:ahal]]

Bug 1405588 is adding the 'base_ref' functionality to mozversioncontrol that this will depend on.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1412121 - [taskgraph] Upload 'target_task_graph' artifact from the decision task,

This is needed to |mach try| can download this taskgraph from CI rather than
generate it locally. Downloading is twice as fast on my machine + connection.

Review commit: https://reviewboard.mozilla.org/r/201772/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/201772/

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1412121 - [tryselect] Generate full graph instead of just labels,

Refactor tasks.py to return a taskgraph object instead of a list of labels.
This gives the |mach try| subcommands more capabilities for choosing tasks.

Since we are making large changes to this area anyway, and the artifacts
already contain an entire taskgraph, this is a good time to make this change.

Review commit: https://reviewboard.mozilla.org/r/201774/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/201774/

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1412121 - [tryselect] First attempt to download the taskgraph from CI,

Re-generating taskgraphs can take ~40 seconds on my machine. Downloading the
taskgraph from CI only takes around 20. So first attempt to download it based
on the parent revision.

If something goes wrong or the 'taskcluster' directory was modified locally, we
fall back to generating it.

Review commit: https://reviewboard.mozilla.org/r/201776/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/201776/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Is this worthwhile if we speed up taskgraph generation with `--fast` as in bug 1420449?

Comment 6

[Andrew Halberstadt [:ahal]]

Yeah possibly, I made it block so we can test it out again (sorry should have also cleared the reviews). With the changes in bug 1420449 downloading might take just as long or longer, so we'll need to re-evaluate.

Comment 7

[Andrew Halberstadt [:ahal]]

I set up a watchman target that mostly solves this:
https://firefox-source-docs.mozilla.org/tools/try/tasks.html#configuring-watchman

I think time would be better spent figuring out how to get that trigger automatically bootstrapped for everyone than trying to land this.