1378212 - Ability to use latest parameters.yml file from given tree

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

It's somewhat of a pain that users of ./mach taskgraph need to go and find a parameters.yml file to use. It would be great if we could specify something like:

./mach taskgraph target -p tree=mozilla-central

This would find the latest decision task on that tree and automatically download + use its parameters.yml file. Making this process super easy will be a requirement if we want to use |mach taskgraph| to generate the list of tasks available to try.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1378212 - Give cron-based decision tasks unique index routes,

We'd like to use routes like:
index.gecko.v2.mozilla-central.latest.firefox.decision

to automatically find the latest parameters.yml file from a decision task. Unfortunately,
sometimes the latest decision task happens to be triggered by a cron job which uses a different
target_tasks method and therefore won't generate the expected tasks.

This patch *should* create a different index for these types of decision tasks, though I'm not
sure how I can test it before pushing.

Review commit: https://reviewboard.mozilla.org/r/154272/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/154272/

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1378212 - Default to grabbing 'parameters.yml' file from latest mozilla-central decision task,

Review commit: https://reviewboard.mozilla.org/r/154274/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/154274/

Comment 3

[Andrew Halberstadt [:ahal]]

To re-iterate the commit message, I don't know how to test that first commit. It *looks* like it should work, but is mostly just a shot in the dark at this point.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8883391 [details]
Bug 1378212 - Give cron-based decision tasks unique index routes,

https://reviewboard.mozilla.org/r/154272/#review159556

I think this is good.  I'm not sure how to test it either :)

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8883392 [details]
Bug 1378212 - Default to grabbing 'parameters.yml' file from latest mozilla-central decision task,

https://reviewboard.mozilla.org/r/154274/#review159558

::: taskcluster/taskgraph/parameters.py:78
(Diff revision 1)
>      except IOError:
> -        # fetching parameters.yml using task task-id or supplied url
> +        # fetching parameters.yml using task task-id, tree or supplied url
> +        task_id = None
>          if filename.startswith("task-id="):
>              task_id = filename.split("=")[1]
> +        elif filename.startswith("tree="):

The taskgraph code is consistent in referring to this as a "project".  It would be nice to continue that consistency (org-wide, it's known by a lot of names, of course)

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 8883391 [details]
Bug 1378212 - Give cron-based decision tasks unique index routes,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/154272/diff/1-2/

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 8883392 [details]
Bug 1378212 - Default to grabbing 'parameters.yml' file from latest mozilla-central decision task,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/154274/diff/1-2/

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8883392 [details]
Bug 1378212 - Default to grabbing 'parameters.yml' file from latest mozilla-central decision task,

https://reviewboard.mozilla.org/r/154274/#review159580

::: taskcluster/mach_commands.py:44
(Diff revision 2)
>                              dest="format", const="json",
>                              help="Output task graph as a JSON object"),
>              CommandArgument('--labels', '-L', action="store_const",
>                              dest="format", const="labels",
>                              help="Output the label for each task in the task graph (default)"),
> -            CommandArgument('--parameters', '-p', required=True,
> +            CommandArgument('--parameters', '-p', default="project=mozilla-central",

++

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5d448af9b09e
Give cron-based decision tasks unique index routes, r=dustin
https://hg.mozilla.org/integration/autoland/rev/051408cf1c53
Default to grabbing 'parameters.yml' file from latest mozilla-central decision task, r=dustin

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/5d448af9b09e
https://hg.mozilla.org/mozilla-central/rev/051408cf1c53

Comment 11

[Gregory Szorc [:gps]]

Thank you for implementing this feature. I just casually wrote a taskgraph patch and unlike previous times, I didn't have to worry about juggling a parameters.yml file: it just worked.