1329235 - Add support for task priority adjustments in try

Status: RESOLVED DUPLICATE of bug 1443974

(Firefox Build System :: Task Configuration, task)

Description

[(not currently active) Ted Mielczarek]

When testing build system changes on try that I expect to have a build time impact, I frequently retrigger the build jobs a number of times to get enough data for a good perfherder comparison. I suspect developers do this for testing changes that they expect to have an impact on Talos results as well (but I almost never do that, so I don't know for sure).

I would be happy to have these jobs run when we have spare capacity instead of taking up queue space that other people need for their try pushes, but there's no easy way to do that right now.

I'd suggest we add support for task priority "low" in the queue, to indicate that the task is not urgent and can wait until capacity is available. I'm not too concerned about the specifics here, but we'd probably want to have some maximum amount of time that low priority jobs would wait just so that they don't sit forever if we don't ever have excess capacity.

As a follow-up it'd be nice to be able to get an email when all the tasks I triggered are finished, so that I can fire off a bunch of low-priority retriggers and not have to keep looking at try to see if they're finished.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

We do have support for low priority tasks:
  task.priority = 'low'

But I don't remember if we've given out the scope.

The complicated part here is probably integrating the with the retriggering logic.
Same thing goes for getting notifications, the support is there, using a dummy tasks and tc-notify.

How did you create the tasks, ie. what retrigger button did you use?

Comment 2

[(not currently active) Ted Mielczarek]

Currently I'm just hitting "R" on treeherder. I'd be fine with writing a custom script to trigger builds like this.

Comment 3

[Greg Arndt [:garndt]]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #1)
> We do have support for low priority tasks:
>   task.priority = 'low'

It doesn't appear we have low, just normal and high

https://github.com/taskcluster/taskcluster-queue/blob/master/schemas/create-task-request.yml#L104-L106


In the future this will change based on bug 1231781 and this conversation:

https://github.com/taskcluster/taskcluster-rfcs/issues/58

However, that still doesn't fix the issue of being able to specify a priority when retriggering jobs.  Currently they will be retriggered with whatever priority the task definition specifies (or normal as the default).  Maybe this is solved when we move retriggers to action tasks defined in tree, and priority is something that can be overridden?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I think we also want support for elevated-priority tasks on try.  So basically we need a way to run try with --this-is-important or --this-is-not-important to adjust the priority.

Comment 5

[Chris Cooper [:coop] (he/him)]

I spoke to Travis about this today. This is something that dcamp (channeling relman, no doubt) is very anxious to see implemented to assist with getting Try results more quickly for chemspill situations.

Any estimate how much work this would entail? I'd like to try to squeeze this into H1 this year if we can.

It sounds like we already have a binary priority system (normal and high) and "just" need to add a way to toggle those priorities manually, possibly adding an additional priority state or two (critical and low?).

critical: speculative patch for chemspill, or chemspill release itself
high:     Releases (including nightly)
normal:   Try and everything else without a priority set
low:      Manually set by conscientious people, and for long-running jobs or batch re-triggers

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

The discussion in our quarterly planning was that it will be possible to *submit* tasks at an elevated priority.  We have seven priorities now, so this just means allowing try to use one of two priorities depending on the option.  Anyone in the taskgraph module could accomplish this.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 1443974 will address this.