1410969 - Generate an artifact listing per-file bug component info

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

We have in-repo file metadata, such as the bug component for a file.

Let's have TaskCluster run a command to generate and upload an artifact so this metadata can be consumed via a bulk and static import of said file.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1410969 - Properly import AutoTry;

The AutoTry code got moved around in 27b5181a3c77 and c94bf66df122
(bug 1382775). This broke `mach test` without arguments.

This commit imports the new location and uses the new API. As the
inline TODO states, there is still room for improvement. This will
be done in a separate commit.

Review commit: https://reviewboard.mozilla.org/r/192154/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/192154/

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1410969 - Return dict from find_paths_and_tags();

This will make the return value more easily extensible and will help
consumers know what the data structures represent.

Review commit: https://reviewboard.mozilla.org/r/192156/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/192156/

Comment 3

[Gregory Szorc [:gps]]

Attachment: Bug 1410969 - Consolidate code for resolving path metadata;

find_paths_and_tags() can instantiate a BuildReader to reader moz.build
metadata. We switch `mach test` to call this to avoid some redundant
code.

To facilitate `mach try`'s use case, we add test flavors to the
result of find_paths_and_metadata() and change the function name to
reflect its more generic purpose.

Review commit: https://reviewboard.mozilla.org/r/192158/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/192158/

Comment 4

[Gregory Szorc [:gps]]

Attachment: Bug 1410969 - Properly resolve wildcards in `mach file-info`;

I likely regressed this as part of e9416a307987 (bug 1397406).

Before this change, '*' in filenames didn't work.

Review commit: https://reviewboard.mozilla.org/r/192160/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/192160/

Comment 5

[Gregory Szorc [:gps]]

Attachment: Bug 1410969 - Add support for writing JSON from `mach file-info`;

The feature should be pretty self-explanatory. It will enable easier
machine readable output.

Review commit: https://reviewboard.mozilla.org/r/192162/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/192162/

Comment 6

[Gregory Szorc [:gps]]

Attachment: Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata;

This should be pretty self-explanatory. We now have tasks that produce
JSON files with our in-repo per-file Bugzilla metadata.

Review commit: https://reviewboard.mozilla.org/r/192164/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/192164/

Comment 7

[Gregory Szorc [:gps]]

I may still need another iteration or two on this. But Try results look promising:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=54e2af69ff71b42e3d8c7b2358ffc5754fe3ac8d
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6153ce8eb12eb3829105f85c9d9d50686d6d9067

Comment 8

[Gregory Szorc [:gps]]

Comment on attachment 8921176 [details]
Bug 1410969 - Return dict from find_paths_and_tags();

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192156/diff/1-2/

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8921177 [details]
Bug 1410969 - Consolidate code for resolving path metadata;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192158/diff/1-2/

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8921178 [details]
Bug 1410969 - Properly resolve wildcards in `mach file-info`;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192160/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8921179 [details]
Bug 1410969 - Add support for writing JSON from `mach file-info`;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192162/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8921180 [details]
Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192164/diff/1-2/

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8921179 [details]
Bug 1410969 - Add support for writing JSON from `mach file-info`;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192162/diff/2-3/

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8921180 [details]
Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192164/diff/2-3/

Comment 15

[Andrew Halberstadt [:ahal]]

Comment on attachment 8921175 [details]
Bug 1410969 - Properly import AutoTry;

https://reviewboard.mozilla.org/r/192154/#review197632

Oops, thanks for fixing this. We'll have to add some python/cram tests for |mach test| at some point.

Comment 16

[Andrew Halberstadt [:ahal]]

Comment on attachment 8921176 [details]
Bug 1410969 - Return dict from find_paths_and_tags();

https://reviewboard.mozilla.org/r/192156/#review197636

Comment 17

[Andrew Halberstadt [:ahal]]

Comment on attachment 8921177 [details]
Bug 1410969 - Consolidate code for resolving path metadata;

https://reviewboard.mozilla.org/r/192158/#review197640

::: commit-message-3dc84:3
(Diff revision 2)
> +Bug 1410969 - Consolidate code for resolving path metadata; r?ahal
> +
> +find_paths_and_tags() can instantiate a BuildReader to reader moz.build

nit: read moz.build

::: tools/tryselect/selectors/syntax.py:462
(Diff revision 2)
> -    def find_paths_and_tags(self, verbose, detect_paths):
> -        paths, tags = set(), set()
> +    def find_paths_and_metadata(self, verbose, detect_paths):
> +        paths, tags, flavors = set(), set(), set()

I think it would make sense to move this function to somewhere under `mozbuild` (except change the signature to accept a list of paths instead of using `files_changed`).

We can worry about that later though.

Comment 18

[Andrew Halberstadt [:ahal]]

Comment on attachment 8921178 [details]
Bug 1410969 - Properly resolve wildcards in `mach file-info`;

https://reviewboard.mozilla.org/r/192160/#review197646

Comment 19

[Andrew Halberstadt [:ahal]]

Comment on attachment 8921179 [details]
Bug 1410969 - Add support for writing JSON from `mach file-info`;

https://reviewboard.mozilla.org/r/192162/#review197648

::: python/mozbuild/mozbuild/frontend/mach_commands.py:130
(Diff revision 3)
>          for component, files in sorted(components.items(), key=lambda x: (x is None, x)):
>              print('%s :: %s' % (component.product, component.component) if component else 'UNKNOWN')
>              for f in sorted(files):
>                  print('  %s' % f)

Probably cleaner if `--format` defaults to whatever you want to call this output.

Comment 20

[Andrew Halberstadt [:ahal]]

Comment on attachment 8921180 [details]
Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata;

https://reviewboard.mozilla.org/r/192164/#review197650

Comment 21

[Gregory Szorc [:gps]]

Comment on attachment 8921179 [details]
Bug 1410969 - Add support for writing JSON from `mach file-info`;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192162/diff/3-4/

Comment 22

[Gregory Szorc [:gps]]

Comment on attachment 8921180 [details]
Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192164/diff/3-4/

Comment 23

[Gregory Szorc [:gps]]

Comment on attachment 8921180 [details]
Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192164/diff/4-5/

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8921179 [details]
Bug 1410969 - Add support for writing JSON from `mach file-info`;

https://reviewboard.mozilla.org/r/192162/#review197648

> Probably cleaner if `--format` defaults to whatever you want to call this output.

Changed this in latest series.

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8921177 [details]
Bug 1410969 - Consolidate code for resolving path metadata;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192158/diff/2-3/

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8921178 [details]
Bug 1410969 - Properly resolve wildcards in `mach file-info`;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192160/diff/2-3/

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8921179 [details]
Bug 1410969 - Add support for writing JSON from `mach file-info`;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192162/diff/4-5/

Comment 28

[Gregory Szorc [:gps]]

Comment on attachment 8921180 [details]
Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/192164/diff/5-6/

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f4058e6c62b3
Properly import AutoTry; r=ahal
https://hg.mozilla.org/integration/autoland/rev/7506fda0a6e7
Return dict from find_paths_and_tags(); r=ahal
https://hg.mozilla.org/integration/autoland/rev/7564ed8dff3e
Consolidate code for resolving path metadata; r=ahal
https://hg.mozilla.org/integration/autoland/rev/7f4ba82f99ea
Properly resolve wildcards in `mach file-info`; r=ahal
https://hg.mozilla.org/integration/autoland/rev/d65c6f54ff66
Add support for writing JSON from `mach file-info`; r=ahal
https://hg.mozilla.org/integration/autoland/rev/e1955af99935
TaskCluster task for generating JSON files with Bugzilla metadata; r=ahal

Comment 30

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/f4058e6c62b3
https://hg.mozilla.org/mozilla-central/rev/7506fda0a6e7
https://hg.mozilla.org/mozilla-central/rev/7564ed8dff3e
https://hg.mozilla.org/mozilla-central/rev/7f4ba82f99ea
https://hg.mozilla.org/mozilla-central/rev/d65c6f54ff66
https://hg.mozilla.org/mozilla-central/rev/e1955af99935