Closed Bug 1477747 Opened 3 years ago Closed 3 years ago

action tasks that create other tasks should create a task-graph.json

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(firefox-esr60 fixed, firefox62 fixed, firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox-esr60 --- fixed
firefox62 --- fixed
firefox63 --- fixed

People

(Reporter: aki, Assigned: aki)

Details

Attachments

(1 file)

CoT verify requires that each decision and action task create a task-graph.json artifact containing all children tasks. This is currently true for the decision task, but aiui the `retrigger` and `backfill` action tasks create task-graph-\d+.json files rather than a task-graph.json. Aiui, these are to allow for multiple copies of the same set of tasks, but since cot verification doesn't know about them, the verify step breaks if these graphs contain any scriptworker tasks.

We could change cot verification to try to download every task-graph-\d+.json file as well as task-graph.json, but this is messy. I think it makes more sense to combine all of the task-graph-\d+.json files into a single task-graph.json file; this could be done at the end of the action if that makes the most sense.

Until we do this, backfill- and retrigger- action graphs will not be cot verifiable.
How would they be joined?  They often contain tasks with the same label..
Aiui, the dict is keyed by taskId, which should be unique.
Ah, so this is just for task-graph.json.  Hm, should be doable.
+1, just for task-graph.json. I think we could either a) build a set of tasks in memory to submit for each pass, then submit them together and write one task-graph.json file, or b) submit each subgraph with intermediary task-graph-\d+.json files, and then run something to glom them all into a master task-graph.json file. Either approach works for my purposes.
I'm unlikely to find time to work on this for quite a while.. if someone else can do it, that'd be great.  Otherwise, assign to me and it'll find a spot in taskwarrior :)
I've got a wip patch that I think will work. I'm going to test it locally then submit it for review.
I tested the above with an input.yml of

downstream: false
times: 4

and

./mach taskgraph test-action-callback -p ../parameters.yml --input ../input.yml --task-group-id EPG6Xq88Tq-oONjcws1HPA --task-id Xl6vOGLERZ-Ndo3NqVtjfQ retrigger

We've got a task-graph.json artifact with 4 copies of the task, but different taskIds.
Assignee: nobody → aki
Comment on attachment 8994582 [details]
bug 1477747 - combine action task-graph-{suffix}.json into one.

https://reviewboard.mozilla.org/r/259126/#review266466

This looks good!  I think there's room for lots of follow-ups around actions, such as making this sort of "automatic" if using create_tasks.  But that can come later.  Enough is changing in actions right now!
Attachment #8994582 - Flags: review?(dustin) → review+
Thanks! I suppose we could have `create_tasks` read from an existing task-graph.json, update it with the new graph, and write it every time, if suffix is not None. I'm also ok with leaving this for a followup.
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b1f82bccc066
combine action task-graph-{suffix}.json into one. r=dustin
https://hg.mozilla.org/mozilla-central/rev/b1f82bccc066
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
You need to log in before you can comment on or make changes to this bug.