Closed Bug 1364266 Opened 5 years ago Closed 5 years ago
Task priority to be defined based on project branch
Tasks currently default to "normal" priority for now. Once we have 7 priority levels within the queue, task priority should be based on the project branch that the task originated from. Before landing in-tree scheduling changes for this, roles need to be updated within taskcluster roles to include scopes for being able to create tasks of a certain level. this perhaps can be done inside of the scm level roles rather than per project.
Aki, can I get your feedback on the levels here? I based it on what was found in buildbot, and translated the numbers in buildbot to the word-based priority levels of taskcluster. I can't test this out on try yet because it hasn't landed in the queue but thought I would get your eyes on it first.
Attachment #8867336 - Flags: feedback?(aki)
Comment on attachment 8867336 [details] priority levels This seems to match, and leaves "lowest" for idle time jobs.
Attachment #8867336 - Flags: feedback?(aki) → feedback+
Ok, it looks like the priority patch for the queue will land Monday morning so we can test this and deploy afterwards.
I'm mostly fine porting a priorities mechanism from how it exists in buildbot today. However, longer term I have some (possibly) radical suggestions: * Prioritize try and project branches over mozilla-central, autoland, and inbound. * Lower priority of try pushes using "-u all" or "-p all" * Lower priority for certain retries on try. e.g. if someone schedules all mochitests to run 20 times, those subsequent runs should be lower priority so they don't dominate the queue) * Raise priority of single/limited retries everywhere (this will help flush out intermittents sooner) These can all be discussed and implemented in a follow-up bug of course. I just wanted to say that I think the current prioritization mechanism can be improved in case anyone was planning to tweak things.
I definitely like that list, and we have some others we would like to add such as interactive tasks should be higher priority because a human is actually waiting on them. Having priority based on just the project and nothing else is a bit heavy handed, but a step in the right direction towards moving away from first come, first served.
Comment on attachment 8868790 [details] Bug 1364266 - Specify task priority within task definition https://reviewboard.mozilla.org/r/140384/#review144082
Attachment #8868790 - Flags: review?(dustin) → review+
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/autoland/rev/6a2176a26495 Specify task priority within task definition r=dustin
Pushed by email@example.com: https://hg.mozilla.org/mozilla-central/rev/183c35371572 Specify task priority within task definition r=dustin a=merge
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Backout by firstname.lastname@example.org: https://hg.mozilla.org/mozilla-central/rev/b31a663614e8 Backed out changeset 6a2176a26495 on suspicion of overwhelming the taskcluster queue
Just a silly question, but does Amazon provide any kind of consulting service that we perhaps should be using in trying to optimize our usage of their services, rather than flying blind here making stuff up?
Not silly, quite a fair question. As the team investigates we certainly will use Microsoft's services to help us out.
In retrospect, backing this patch out indeed was the wrong choice and was not contributing to the issues we were seeing. To the point of comment 13, it was flying blind. We now have some metrics being captured by the queue that allows us to pinpoint where this issue is, which is in publishing pulse messages and nothing to do with Azure like we originally thought. This patch should be safe to land again.
https://hg.mozilla.org/integration/mozilla-inbound/rev/5a676e5924b68ad90da251c70fa9ea776b5cfe2f Bug 1364266 - Specify task priority within task definition r=dustin
Status: REOPENED → RESOLVED
Closed: 5 years ago → 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.