1574189 - Allow repeated `dependencies` in taskgraph task definitions

Status: RESOLVED MOVED

(Firefox Build System :: Task Configuration, task)

Description

[Nick Alexander :nalexander [he/him]]

This has now bitten me twice, so it's time to file. The dependencies: block is a map from string alias to string job name. The TC API rejects repeated dependencies in the submitted task definition; for example:

[task 2019-08-15T04:34:27.531Z] Schema Validation Failed!
[task 2019-08-15T04:34:27.531Z] Rejecting Schema: https://schemas.taskcluster.net/queue/v1/create-task-request.json#
[task 2019-08-15T04:34:27.531Z] Errors:
[task 2019-08-15T04:34:27.531Z]   * data.dependencies should NOT have duplicate items (items ## 1 and 0 are identical)
[task 2019-08-15T04:34:27.531Z] 
[task 2019-08-15T04:34:27.531Z] ---
[task 2019-08-15T04:34:27.531Z] 
[task 2019-08-15T04:34:27.531Z] * method:     createTask
[task 2019-08-15T04:34:27.531Z] * errorCode:  InputValidationError
[task 2019-08-15T04:34:27.531Z] * statusCode: 400
[task 2019-08-15T04:34:27.531Z] * time:       2019-08-15T04:34:27.496Z
[task 2019-08-15T04:34:27.535Z] 
[task 2019-08-15T04:34:27.535Z] Schema Validation Failed!
[task 2019-08-15T04:34:27.535Z] Rejecting Schema: https://schemas.taskcluster.net/queue/v1/create-task-request.json#
[task 2019-08-15T04:34:27.535Z] Errors:
[task 2019-08-15T04:34:27.535Z]   * data.dependencies should NOT have duplicate items (items ## 7 and 6 are identical)
[task 2019-08-15T04:34:27.536Z] 
[task 2019-08-15T04:34:27.536Z] ---
[task 2019-08-15T04:34:27.536Z] 
[task 2019-08-15T04:34:27.536Z] * method:     createTask
[task 2019-08-15T04:34:27.536Z] * errorCode:  InputValidationError
[task 2019-08-15T04:34:27.536Z] * statusCode: 400
[task 2019-08-15T04:34:27.536Z] * time:       2019-08-15T04:34:27.501Z
[task 2019-08-15T04:34:27.613Z] 
[task 2019-08-15T04:34:27.613Z] Schema Validation Failed!
[task 2019-08-15T04:34:27.613Z] Rejecting Schema: https://schemas.taskcluster.net/queue/v1/create-task-request.json#
[task 2019-08-15T04:34:27.613Z] Errors:
[task 2019-08-15T04:34:27.613Z]   * data.dependencies should NOT have duplicate items (items ## 6 and 5 are identical)
[task 2019-08-15T04:34:27.613Z] 
[task 2019-08-15T04:34:27.613Z] ---
[task 2019-08-15T04:34:27.613Z] 
[task 2019-08-15T04:34:27.613Z] * method:     createTask
[task 2019-08-15T04:34:27.613Z] * errorCode:  InputValidationError
[task 2019-08-15T04:34:27.613Z] * statusCode: 400
[task 2019-08-15T04:34:27.613Z] * time:       2019-08-15T04:34:27.579Z

Fine, fair enough. But it's legitimately useful to be able to give the same target task name multiple aliases; for example, I want to refer to a Linux build for extracting one of the test archives, but there may already be a reference to that Linux build for extracting the package under test. Right now I need to conditionally change my alias so as to not have a repeated dependency.

Can we make the dependency handling code know that the target dependencies are a set, and manage multiple aliases to the same task?

Comment 1

[Nick Alexander :nalexander [he/him]]

tomprince: dustin: is there some fundamental reason I can't have multiple aliases to the same dependent task?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

There's good reason for TC not to allow this (since they're not aliases, just an array of taskIds), but I think it makes sense in task-graph generation. So this could be fixed by unique-ifying that array when generating the task definition.

Comment 3

[Tom Prince [:tomprince]]

Yeah, that seems reasonable.

Comment 4

[Andrew Halberstadt [:ahal]]

Moved to: https://github.com/taskcluster/taskgraph/issues/11