1178516 - scheduler: New big graph scheduler

Status: RESOLVED FIXED

(Taskcluster Graveyard :: Scheduler, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

Drop the concept of task-graphs... and just make dependencies between tasks.
This is partially prototyped and tested in bug 1107597.

I suggest we proceed with implementation strategy D from bug 1107597.
We need the following changes/features:
 - queue:     Add API to list taskIds by taskGroupId
 - queue:     Add API to page status of taskIds by taskGroupId
 - base:      Add system for specifying query-string parameters (for paging)
 - clients:   Add support for query-string parameters (for paging)
 - scheduler: listen for tasks, read task.extra.dependsOn as a list of dependent
              taskIds. Let "!<taskId>" denote: run when <taskId> is failed,
              and "?<taskId>" denote: run when <taskId> is resolved,
              no matter how the task is resolved.
 - scheduler: API to add a dependency
 - scheduler: API to remove a dependency
 - scheduler: API to list dependencies
 - scheduler: API to list depends-on (and remaining things a task depends-on)

The API to add/remove dependencies can be used for activating tasks that was
created with a self-dependency. Useful for cases where we want to disable some
tasks by default (perhaps we only run them every 10 pushes or so).

Note, we either find a new name. Or implement the old task-graphs API on the new
scheduler. I think perhaps we could call it dependencies.taskcluster.net or
something like that. It's definitely simplest to just keep the task-graph
scheduler around until we're not using it anymore.

Comment 1

[Armen [:armenzg]]

Hi jonasfj, what is your timeline for this?
Is there any wip I can test? Thanks!

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

@armenzg,
There is a prototype here:
  https://github.com/taskcluster/taskcluster-scheduler
It doesn't implement the API yet... just the method of making relations.

Comment 3

[Armen [:armenzg]]

I will NI myself to look at it.

Is this what we will switch to for mach taskcluster-graph? or this the server component?
Do you expect to have this on Q3? Q4?

Comment 4

[Armen [:armenzg]]

What I read on your original description and on the README of that repo makes sense to me.

Please let me know once you know the timelines for this project. It will help me time my own work around it.

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

status:
 + queue:     Add API to list taskIds by taskGroupId
 + queue:     Add API to page status of taskIds by taskGroupId
 + base:      Add system for specifying query-string parameters (for paging)
 + clients:   Add support for query-string parameters (for paging)

The "scheduler" stuff will be folded into the queue, this is a bit of work.
Mostly recognizing all the places it fits in... and making sure everything is hooked up correctly.

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

work in progress here:
  https://github.com/taskcluster/taskcluster-queue/pull/81

It needs testing and a fair bit of review to make sure we got the idempotency correct.

Comment 7

[Armen [:armenzg]]

Hi jonasfj, is your timeline for this end of Q1? is it one of your deliverables? Thanks!

Is this to remove the upper limit of tasks in a graph? (It would make bug 1250988 unncessary)

Comment 8

[Jonas Finnemann Jensen (:jonasfj)]

> is your timeline for this end of Q1? is it one of your deliverables? 
> Is this to remove the upper limit of tasks in a graph?
Yes, Yes, Yes :)

It also improves error messages, etc. End of Q1 is the plan.

Comment 9

[Jonas Finnemann Jensen (:jonasfj)]

Update: scheduler was merged into queue, all this is done I hope.
We still have to do a task-group inspector at some point, but nothing is blocking anyone from using this.