1275034 - create decision tasks with the assumption they use the big-graph scheduler

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

mozilla-taskcluster should, for tasks configured from .taskcluster.yml, provide a new TASK_GROUP_ID into which it has inserted the decision task, and skip using the scheduler at all.

The schedulerid for the new task should be designed appropriately, as well, so that we can limit access to retriggers, etc.  I think that should involve both the level and the project, e.g., `l-{{level}}-{{project}}`

Comment 1

[Greg Arndt [:garndt]]

I wouldn't be surprised if the scheduler ID also has a similar character limit as the other identifiers.

Comment 2

[Greg Arndt [:garndt]]

Yup, schedulerID is limited to 22 characters:
https://github.com/taskcluster/taskcluster-queue/blob/master/schemas/create-task-request.yml#L35

"level-1-mozilla-inbound" is 23 by itself.

Comment 3

[Greg Arndt [:garndt]]

So if we want to have a unique scheduler ID, we need something <= 22 characters.  Any suggestions on naming? If we include project, mozilla-inbound is 15 characters alone so we're also imposing a limit to how long the project names can be so it can fit into a scheduler ID.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Maybe we should compromise and just include the level, and enough to disambiguate these schedulers from non-gecko schedulers?  Something like `gecko-level-N`?  Then we can assign the relevant scopes to the per-level groups.  We generally don't want to give scopes specific to a project anyway.

Comment 5

[Greg Arndt [:garndt]]

Attachment: mozilla-taskcluster PR 85

I think this should schedule the tasks the way we've been wanting.  This should also allow the decision task to be passed arguments for creating additional tasks with the same scheduler and group id.

I attempted to look into m-t's testing situation to see about testing this ahead of time, and it's as to be expected, a mess.  I can still add tests but thought I would show you this before I enter PTO at the end of the week.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8770801 [details] [review]
mozilla-taskcluster PR 85

I'll try to land this soon

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: https://github.com/taskcluster/mozilla-taskcluster/pull/87

I updated this a little based on our discussion of setting the taskGroupId.  I don't know if that needs to be documented somewhere?  At the very least, probably in .taskcluster.yml, for which I'll submit another patch.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1275034: add comment about taskGroupId = taskId;

Review commit: https://reviewboard.mozilla.org/r/65598/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65598/

Comment 9

[Greg Arndt [:garndt]]

Comment on attachment 8772904 [details]
Bug 1275034: add comment about taskGroupId = taskId;

https://reviewboard.mozilla.org/r/65598/#review65654

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5406ddf4a856
add comment about taskGroupId = taskId; r=garndt

Comment 11

[Greg Arndt [:garndt]]

This has been landed and pushed to heroku

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5406ddf4a856