1364266 - Task priority to be defined based on project branch

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Greg Arndt [:garndt]]

Tasks currently default to "normal" priority for now.  Once we have 7 priority levels within the queue, task priority should be based on the project branch that the task originated from.

Before landing in-tree scheduling changes for this, roles need to be updated within taskcluster roles to include scopes for being able to create tasks of a certain level.

this perhaps can be done inside of the scm level roles rather than per project.

Comment 1

[Greg Arndt [:garndt]]

Attachment: priority levels

Aki, can I get your feedback on the levels here? I based it on what was found in buildbot, and translated the numbers in buildbot to the word-based priority levels of taskcluster.  I can't test this out on try yet because it hasn't landed in the queue but thought I would get your eyes on it first.

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 8867336 [details]
priority levels

This seems to match, and leaves "lowest" for idle time jobs.

Comment 3

[Greg Arndt [:garndt]]

Ok, it looks like the priority patch for the queue will land Monday morning so we can test this and deploy afterwards.

Comment 4

[Gregory Szorc [:gps]]

I'm mostly fine porting a priorities mechanism from how it exists in buildbot today. However, longer term I have some (possibly) radical suggestions:

* Prioritize try and project branches over mozilla-central, autoland, and inbound.
* Lower priority of try pushes using "-u all" or "-p all"
* Lower priority for certain retries on try. e.g. if someone schedules all mochitests to run 20 times, those subsequent runs should be lower priority so they don't dominate the queue)
* Raise priority of single/limited retries everywhere (this will help flush out intermittents sooner)

These can all be discussed and implemented in a follow-up bug of course. I just wanted to say that I think the current prioritization mechanism can be improved in case anyone was planning to tweak things.

Comment 5

[Greg Arndt [:garndt]]

I definitely like that list, and we have some others we would like to add such as interactive tasks should be higher priority because a human is actually waiting on them.

Having priority based on just the project and nothing else is a bit heavy handed, but a step in the right direction towards moving away from first come, first served.

Comment 6

[Greg Arndt [:garndt]]

Attachment: Bug 1364266 - Specify task priority within task definition

Tasks should be assigned a priority based on the branch they originated from. It
is important that certain branches receive preferential treatment, such as a release
branch task being executed before a task from Try. Branch priority mirrors
the priorities defined within buildbot.

Review commit: https://reviewboard.mozilla.org/r/140384/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/140384/

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8868790 [details]
Bug 1364266 - Specify task priority within task definition

https://reviewboard.mozilla.org/r/140384/#review144082

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6a2176a26495
Specify task priority within task definition r=dustin

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/183c35371572
Specify task priority within task definition r=dustin a=merge

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/183c35371572

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6a2176a26495

Comment 12

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/b31a663614e8
Backed out changeset 6a2176a26495 on suspicion of overwhelming the taskcluster queue

Comment 13

[Bill Gianopoulos [:WG9s]]

Just a silly question, but does Amazon provide any kind of consulting service that we perhaps should be using in trying to optimize our usage of their services, rather than flying blind here making stuff up?

Comment 14

[Greg Arndt [:garndt]]

Not silly, quite a fair question.  As the team investigates we certainly will use Microsoft's services to help us out.

Comment 15

[Greg Arndt [:garndt]]

In retrospect, backing this patch out indeed was the wrong choice and was not contributing to the issues we were seeing.  To the point of comment 13, it was flying blind.  We now have some metrics being captured by the queue that allows us to pinpoint where this issue is, which is in publishing pulse messages and nothing to do with Azure like we originally thought. This patch should be safe to land again.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5a676e5924b68ad90da251c70fa9ea776b5cfe2f
Bug 1364266 - Specify task priority within task definition r=dustin

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/5a676e5924b6