1310058 - Let's figure out task-group metadata for tc-gh

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Brian Stack [:bstack]]

I'm in the process of moving tc-gh from using the scheduler to using the queue directly. Much of this should be a straightforward swap out since we've added the "task-group-drained" exchange, but there's one major difference that we need to sort out. As it stands, a .taskcluster.yml for tc-gh contains a top-level metadata dictionary that sets fields such as name/description/owner/source for the entire graph. This creates a few problems.

1. If we make this switch directly and just throw away the graph metadata, formerly required info is now un-required and just thrown off into the void. It's quite probable that people are using this info for something. In the common case it's pretty much just a clone of the task metadata, but there are more complex cases.

2. We want to be able to notify users of an entire task-group being complete in tc-gh with an email or irc similarly to travis telling you that all of your builds are done. If we don't associate this metadata with the task-group somehow, we'll not know who to notify or how.

Possible fixes for (1):
a) Just throw it away because probably nobody cares anyway. Update docs to specify as such.
b) Create a "v2" .taskcluster.yml that doesn't contain these fields
c) Do something semi-magical with them if they exist -- This means subsituting them into task metadata or something else like that.
d) do one of the fixes for (2) below

Possible fixes for (2):
a) Add metadata fields to task-groups in a similar way to how we have them on taskgraphs. This could happen in two ways: Either by adding a createTaskGroup endpoint to the queue that has to be called before tasks are created, or by taking metadata from the first task in a task-group and associating it with the group as a whole.
b) Have tc-gh load an arbitrary task from the group upon being alerted to it being finished and use the metadata fields from that to generate notifications
c) Store the top-level metadata fields in azure keyed on the task-group id that was created for the push. This could be seen as early work towards tc-gh storing more information about the state of a repo in preparation for a "repo dashboard" and shields like travis provides.

Personally I think I'm partial to 2.c with 2.a being a close second. Do you have any thoughts on the matter? Feel free to invite anyone else who might care about this to comment.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

For 1, that metadata appears in the inspector -- https://tools.taskcluster.net/task-graph-inspector/#Ee4inTKxTlO-LErfO3IxYA/ -- but I suspect it's unused otherwise.

I see multiple tasks in .taskcluster.yml as alternatives, rather than a collection of tasks in a single group (although both are possible).  In the cases where I've used it, I've generally written one task for pushes and one for master, for example.  The slightly more complex case of running a few tasks in parallel quickly runs into the limitation that you can't make those tasks depend on one another, so there's a fairly narrow range of useful behavior there.  I think the fix to that limitation is better support for decision tasks -- something that's outside the scope of tc-github.

All of which means that the more common case will be for the metadata on the task that is created to be more important than that on the graph/group.  As a datapoint, I don't think mozilla-taskcluster does anything with the top-level metadata in .taskcluster.yml for Gecko.

I do like option 2.c, especially if tc-github is the tool performing these notifications.  If, on the other hand, you'd want tc-notify to grow the ability to notify on task-group completion, then I think the metadata should be stored in the queue.

Comment 2

[Brian Stack [:bstack]]

Great. In my ideal world tc-gh is the one doing the notifications here so that makes sense. I'll begin work on that now!