1533314 - assess the build workerTypes usage in mobile world and experiment the priorities inclusion

Status: NEW

(Release Engineering :: Release Automation, enhancement)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

When work from bug 1519493 for steps 1,2 has landed, we've started seeing bustages in some repos. These bustages raise some of the assumptions we've made about scopes, mainly:

queue:create-task:highest:aws-provisioner-v1/mobile-1-* does NOT automatically infer queue:create-task:lowest:aws-provisioner-v1/mobile-1-b-andrcmp at runtime.

The logic above was inferred from two previous assumptions:
i. a priority implies all lower priorities. E.g. queue:create-task:highest:some/worker-type implies all lowest, medium, high,etc
ii. globstar mobile-1-* implies mobile-1-b-andrcmp

This is wrong and doesn't work this way.
Johan helped me out with understanding this and suggested two possible solutions:

a. we make taskclusterProxy smarter so it has a better understanding of priorities
b. we try to change the scope given to a decision from queue:create-task:highest:aws-provisioner-v1/mobile-1-* to assume:role:something-something. Apparently b. is a significant difference than what we currently have in gecko world.

The main problem could be that queue:create-task:highest:aws-provisioner-v1/mobile-1-* implies the other (more specific one) when you run the decision task, butwhen the decision task comes to create the task, then it strictly uses the scope that it was given.

Since this is breaking developer's workflow, we will do this:

1. for now, switch everywhere to highest to match what we currently have in ciadmin world
2. leave it like that for a couple of days/weeks to assess the load on the workerTypes as well. Johan suggested that this might be a good idea to evaluate anyway since we're presuambly not provisioning the maximum amount of machines.
3. idea here is to glob star as much as we can in ciadmin and be more specific in projects themselves. But the build workers are to be migrated anyway towards mobile-{level}-b-linux so this should help in the near future.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [a-c] Bug 1533314 - switch all priorities to highest.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [ref-browser] Bug 1533314 - switch all priorities to highest.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Tom also suggested we should be moving towards assuming roles instead of specific scopes per tasks, similarly to what we currently have in gecko. Example gist can be found here

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: more cleanup that can be done here: https://github.com/mozilla-mobile/fenix/blob/master/.taskcluster.yml#L58

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Found in triaging. Not actively working on this, returning to the pool.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/782bfb093de4
[components] Bug 1533314 - switch all priorities to highest.