1395717 - Use 'jobs-from' filename in the label for source-test tasks

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

I'd like the filename of the jobs-from config (e.g, mozlint.yml, python-tests.yml, doc.yml, etc) to show up in the final task label (sans extension).

This will make it much easier to select all tasks within a file with |mach try fuzzy|. Currently if you want to run all "python-test" tasks, you need to remember what's defined in python-tests.yml and hunt them all down manually. With this change we could just run:
./mach try fuzzy -q "'python-test"

It might make sense for these filenames to be part of the label in other types of tasks too, but for now let's just limit it to source-test.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

This creates a new "from" attribute that contains the filename of the 'jobs-from' config
that they were defined in. If there was no 'jobs-from' and they were instead defined with
the 'jobs' config, this attribute will not be present

Review commit: https://reviewboard.mozilla.org/r/175108/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/175108/

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8903316 [details]
Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

https://reviewboard.mozilla.org/r/175108/#review180544

If this could just be a key in the job dictionary, I'd prefer that..

::: taskcluster/taskgraph/loader/transform.py:44
(Diff revision 1)
>  
> +        for filename in config.get('jobs-from', []):
> +            for name, job in load_yaml(path, filename).iteritems():
> +                if defaults:
> +                    job = merge(defaults, job)
> +                job.setdefault('attributes', {})['from'] = filename

Why not just `job['job-from'] = filename` here?  Why does this need to be an attribute?

Comment 3

[Andrew Halberstadt [:ahal]]

Sure that's fine. I was mostly just trying to avoid updating all the schemas :p (and attributes looked like they contained similar types of information)

Comment 4

[Andrew Halberstadt [:ahal]]

Comment on attachment 8903316 [details]
Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/175108/diff/1-2/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8903316 [details]
Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

https://reviewboard.mozilla.org/r/175108/#review181412

::: taskcluster/taskgraph/loader/transform.py:37
(Diff revision 2)
> -        jobs_from = itertools.chain.from_iterable(
> -            load_yaml(path, filename).iteritems()
> -            for filename in config.get('jobs-from', {}))
> -        for name, job in itertools.chain(jobs, jobs_from):
>              if defaults:
>                  job = merge(defaults, job)

Shouldn't this have `job-from` too?

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 8903316 [details]
Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

https://reviewboard.mozilla.org/r/175108/#review181412

> Shouldn't this have `job-from` too?

Yeah, I suppose so. Should it just be `kind.yml`? Is that guaranteed to be the name of the file?

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 8903316 [details]
Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/175108/diff/2-3/

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8903316 [details]
Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/175108/diff/3-4/

Comment 9

[Andrew Halberstadt [:ahal]]

Comment on attachment 8903316 [details]
Bug 1395717 - Use filename source-test tasks are defined in as part of their label,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/175108/diff/4-5/

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f366810cead4
Use filename source-test tasks are defined in as part of their label, r=dustin

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/f366810cead4