Closed Bug 1364266 Opened 5 years ago Closed 5 years ago

Task priority to be defined based on project branch

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla55

People

(Reporter: garndt, Assigned: garndt)

References

Details

Attachments

(2 files)

Tasks currently default to "normal" priority for now.  Once we have 7 priority levels within the queue, task priority should be based on the project branch that the task originated from.

Before landing in-tree scheduling changes for this, roles need to be updated within taskcluster roles to include scopes for being able to create tasks of a certain level.

this perhaps can be done inside of the scm level roles rather than per project.
Attached file priority levels
Aki, can I get your feedback on the levels here? I based it on what was found in buildbot, and translated the numbers in buildbot to the word-based priority levels of taskcluster.  I can't test this out on try yet because it hasn't landed in the queue but thought I would get your eyes on it first.
Attachment #8867336 - Flags: feedback?(aki)
Comment on attachment 8867336 [details]
priority levels

This seems to match, and leaves "lowest" for idle time jobs.
Attachment #8867336 - Flags: feedback?(aki) → feedback+
Ok, it looks like the priority patch for the queue will land Monday morning so we can test this and deploy afterwards.
I'm mostly fine porting a priorities mechanism from how it exists in buildbot today. However, longer term I have some (possibly) radical suggestions:

* Prioritize try and project branches over mozilla-central, autoland, and inbound.
* Lower priority of try pushes using "-u all" or "-p all"
* Lower priority for certain retries on try. e.g. if someone schedules all mochitests to run 20 times, those subsequent runs should be lower priority so they don't dominate the queue)
* Raise priority of single/limited retries everywhere (this will help flush out intermittents sooner)

These can all be discussed and implemented in a follow-up bug of course. I just wanted to say that I think the current prioritization mechanism can be improved in case anyone was planning to tweak things.
I definitely like that list, and we have some others we would like to add such as interactive tasks should be higher priority because a human is actually waiting on them.

Having priority based on just the project and nothing else is a bit heavy handed, but a step in the right direction towards moving away from first come, first served.
Blocks: 1231781
Comment on attachment 8868790 [details]
Bug 1364266 - Specify task priority within task definition

https://reviewboard.mozilla.org/r/140384/#review144082
Attachment #8868790 - Flags: review?(dustin) → review+
Pushed by garndt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6a2176a26495
Specify task priority within task definition r=dustin
Pushed by kwierso@gmail.com:
https://hg.mozilla.org/mozilla-central/rev/183c35371572
Specify task priority within task definition r=dustin a=merge
https://hg.mozilla.org/mozilla-central/rev/183c35371572
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Backout by philringnalda@gmail.com:
https://hg.mozilla.org/mozilla-central/rev/b31a663614e8
Backed out changeset 6a2176a26495 on suspicion of overwhelming the taskcluster queue
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Just a silly question, but does Amazon provide any kind of consulting service that we perhaps should be using in trying to optimize our usage of their services, rather than flying blind here making stuff up?
Not silly, quite a fair question.  As the team investigates we certainly will use Microsoft's services to help us out.
In retrospect, backing this patch out indeed was the wrong choice and was not contributing to the issues we were seeing.  To the point of comment 13, it was flying blind.  We now have some metrics being captured by the queue that allows us to pinpoint where this issue is, which is in publishing pulse messages and nothing to do with Azure like we originally thought. This patch should be safe to land again.
https://hg.mozilla.org/mozilla-central/rev/5a676e5924b6
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
Duplicate of this bug: 1348773
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.