1477747 - action tasks that create other tasks should create a task-graph.json

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Aki Sasaki (not active)]

CoT verify requires that each decision and action task create a task-graph.json artifact containing all children tasks. This is currently true for the decision task, but aiui the `retrigger` and `backfill` action tasks create task-graph-\d+.json files rather than a task-graph.json. Aiui, these are to allow for multiple copies of the same set of tasks, but since cot verification doesn't know about them, the verify step breaks if these graphs contain any scriptworker tasks.

We could change cot verification to try to download every task-graph-\d+.json file as well as task-graph.json, but this is messy. I think it makes more sense to combine all of the task-graph-\d+.json files into a single task-graph.json file; this could be done at the end of the action if that makes the most sense.

Until we do this, backfill- and retrigger- action graphs will not be cot verifiable.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

How would they be joined?  They often contain tasks with the same label..

Comment 2

[Aki Sasaki (not active)]

Aiui, the dict is keyed by taskId, which should be unique.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, so this is just for task-graph.json.  Hm, should be doable.

Comment 4

[Aki Sasaki (not active)]

+1, just for task-graph.json. I think we could either a) build a set of tasks in memory to submit for each pass, then submit them together and write one task-graph.json file, or b) submit each subgraph with intermediary task-graph-\d+.json files, and then run something to glom them all into a master task-graph.json file. Either approach works for my purposes.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I'm unlikely to find time to work on this for quite a while.. if someone else can do it, that'd be great.  Otherwise, assign to me and it'll find a spot in taskwarrior :)

Comment 6

[Aki Sasaki (not active)]

I've got a wip patch that I think will work. I'm going to test it locally then submit it for review.

Comment 7

[Aki Sasaki (not active)]

Attachment: bug 1477747 - combine action task-graph-{suffix}.json into one.

Review commit: https://reviewboard.mozilla.org/r/259126/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/259126/

Comment 8

[Aki Sasaki (not active)]

I tested the above with an input.yml of

downstream: false
times: 4

and

./mach taskgraph test-action-callback -p ../parameters.yml --input ../input.yml --task-group-id EPG6Xq88Tq-oONjcws1HPA --task-id Xl6vOGLERZ-Ndo3NqVtjfQ retrigger

We've got a task-graph.json artifact with 4 copies of the task, but different taskIds.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8994582 [details]
bug 1477747 - combine action task-graph-{suffix}.json into one.

https://reviewboard.mozilla.org/r/259126/#review266466

This looks good!  I think there's room for lots of follow-ups around actions, such as making this sort of "automatic" if using create_tasks.  But that can come later.  Enough is changing in actions right now!

Comment 10

[Aki Sasaki (not active)]

Thanks! I suppose we could have `create_tasks` read from an existing task-graph.json, update it with the new graph, and write it every time, if suffix is not None. I'm also ok with leaving this for a followup.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b1f82bccc066
combine action task-graph-{suffix}.json into one. r=dustin

Comment 12

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/b1f82bccc066

Comment 13

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-release/rev/ad03a9634f51

Comment 14

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/f4109139c7b3