Closed Bug 1277417 Opened 3 years ago Closed 3 years ago

Output JSON from `mach taskgraph` commands

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

Attachments

(1 file)

These commands currently output a list of task labels and descriptions, which isn't too useful.  Instead, they should output JSON and recommend using `jq` to filter that down as necessary.
The JSON output is suitable for processing with `jq` to extract features of
interest.

Review commit: https://reviewboard.mozilla.org/r/58064/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/58064/
Attachment #8760496 - Flags: review?(ahalberstadt)
Duplicate of this bug: 1278411
Comment on attachment 8760496 [details]
Bug 1277417: output task information in JSON or just labels;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/58064/diff/1-2/
Attachment #8760496 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8760496 [details]
Bug 1277417: output task information in JSON or just labels;

https://reviewboard.mozilla.org/r/58064/#review55054

Looks good, with a couple suggestions. Feel free to re-flag if you do implement the suggestions and it ends up being non-trivial.

::: taskcluster/mach_commands.py:36
(Diff revision 2)
> +            CommandArgument('--json', '-J', action="store_const",
> +                            dest="format", const="json",
> +                            help="Output each task in the task graph as a JSON object"),
> +            CommandArgument('--labels', '-L', action="store_const",
> +                            dest="format", const="labels",
> +                            help="Output the label for each task in the task graph (default)"),

This is likely fine if there will be just two formats, but consider a --format parameter with action="choices" and default="label" instead.

::: taskcluster/mach_commands.py:201
(Diff revision 2)
> +    def show_taskgraph_json(self, taskgraph):
> +        named_links_dict = taskgraph.graph.named_links_dict()
> +        # this dictionary may be keyed by label or by taskid, so let's just call it 'key'
> +        for key in taskgraph.graph.visit_postorder():
> +            task = taskgraph.tasks[key]
> +            task_json = {
> +                'label': task.label,
> +                'attributes': task.attributes,
> +                'dependencies': named_links_dict.get(key, {}),
> +                'task': task.task
> +            }

Would be good to put a to_json method, both on TaskGraph and Task (with TaskGraph one calling Task one). That way it can also be re-used below.

::: taskcluster/mach_commands.py:214
(Diff revision 2)
> +            json.dump(task_json, sys.stdout, sort_keys=True, indent=2)
> +            sys.stdout.write('\n')

Why not:

    print(json.dumps(...))

::: taskcluster/taskgraph/decision.py:117
(Diff revision 2)
>  def taskgraph_to_json(taskgraph):
>      tasks = taskgraph.tasks
> +    named_links_dict = taskgraph.graph.named_links_dict()
>  
>      def tojson(task):
> -        return {
> +        task_json = {
>              'label': task.label,
>              'task': task.task,
>              'attributes': task.attributes,
> -            'dependencies': []
> +            'dependencies': named_links_dict.get(task.label, {})
>          }

A to_json method on the Task/TaskGraph classes could be re-used here.
https://reviewboard.mozilla.org/r/58064/#review55054

> This is likely fine if there will be just two formats, but consider a --format parameter with action="choices" and default="label" instead.

I want to add --dot later for dot (graph) output
Comment on attachment 8760496 [details]
Bug 1277417: output task information in JSON or just labels;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/58064/diff/2-3/
Attachment #8760496 - Flags: review+ → review?
Attachment #8760496 - Flags: review? → review?(ahalberstadt)
Comment on attachment 8760496 [details]
Bug 1277417: output task information in JSON or just labels;

https://reviewboard.mozilla.org/r/58064/#review55146

::: taskcluster/mach_commands.py:203
(Diff revisions 2 - 3)
>          for label in taskgraph.graph.visit_postorder():
>              print(label)
>  
>      def show_taskgraph_json(self, taskgraph):
> -        named_links_dict = taskgraph.graph.named_links_dict()
> -        # this dictionary may be keyed by label or by taskid, so let's just call it 'key'
> +        # JSON output is a sequence of JSON objects, rather than a single object, so
> +        # disasseble the dictionary

nit: disassemble
Attachment #8760496 - Flags: review?(ahalberstadt) → review+
My keyboarding instructor is so ashamed of me..
Comment on attachment 8760496 [details]
Bug 1277417: output task information in JSON or just labels;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/58064/diff/3-4/
https://hg.mozilla.org/mozilla-central/rev/cf46a59ce5e1
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Blocks: 1280904
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.