1307771 - rename opt-* and dbg-* and android-* workerTypes to gecko-L-b-*

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Some of these can probably be combined with gecko-L-b-linux, with a corresponding bump in its capacity, but `gecko-L-b-cross-osx` et al. is OK too.

Comment 1

[Haard Panchal]

Hello! I'm new to bugzilla and would like to work on this issue. It would be great if I could get some tips and more detail on the same.

Comment 2

[Matt Drake [:drake]]

Same. I would like to work on some of these bugs. Need some info on where to start, how to get started, that sort of thing.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Hi!  This might actually be a fun bug to work on together, as there's a fair bit to learn and you might help one another when you get stuck.

You can see the list of workerTypes we have at
  https://tools.taskcluster.net/aws-provisioner/

For a bit of background on what a workerType is, you may want to read through
  https://docs.taskcluster.net/manual
You can probably skip most of the first chapter about API access.

You'll see that some of our workerTypes have nice consistent names starting with gecko-.., while others are basically made up (opt-this, dbg-that, etc.).  The idea is to use consistent names for everything.

The things that *use* these worker types are tasks created from the Firefox source code (Gecko).  You'll find some documentation of that at
  https://gecko.readthedocs.io/en/latest/taskcluster/taskcluster

And in particular you'll find references to the workerTypes under taskcluster/ in the gecko source code.

The process for making these changes is this:

 * create workerTypes with the correct names (I can do this, as it involves handling some passwords)
 * find and change the references in the source code to point to workerTypes with the correct names
   * write a patch
   * get review
   * land the patch
 * wait for those changes to be distributed everywhere so that the old workerTypes are no longer used (could be a few months)
 * remove the old workerTypes

Once you're familiar with this code, there are lots of other things to work on, too :)

Comment 4

[Haard Panchal]

Thanks Dustin! I'll start working on it right away.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Haard, how is it going?  Have you had a chance to talk with Matt?

Comment 6

[Haard Panchal]

I have gone through the code under taskcluster/ and found references to opt-*, dbg-* and android-*. But in order to change that to gecko-* correspondingly, we would have to create those worker types in https://tools.taskcluster.net/aws-provisioner/#gecko-t-win7-32/ , right?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Yes, that's right.  I can create them by copying the existing definitions.  Can you give me a list of suggested workerType names, and what type to copy them from?

Comment 8

[Haard Panchal]

Here you go, Dustin:
Previous       Final
dbg-linux32 -> gecko-L-b-linux32
dbg-linux64 -> gecko-L-b-linux64
dbg-macosx64 -> gecko-L-b-macosx64
opt-linux32 -> gecko-L-b-linux32
opt-linux64 -> gecko-L-b-linux64
opt-macosx64 -> gecko-L-b-macosx64
android-api-15 -> gecko-L-b-android
Should there be any more?

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Sounds good.  Looking in the source, only a few are still used:

dbg-linux32
dbg-linux64
dbg-macosx64
taskcluster/ci/build/macosx.yml:    worker-type: aws-provisioner-v1/dbg-macosx64
opt-linux32
opt-linux64
opt-macosx64
taskcluster/ci/static-analysis/kind.yml:        worker-type: aws-provisioner-v1/opt-macosx64
taskcluster/ci/build/macosx.yml:    worker-type: aws-provisioner-v1/opt-macosx64
android-api-15
taskcluster/ci/build/android.yml:    worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/build/android.yml:    worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/build/android.yml:    worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/build/android.yml:    worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/build/android.yml:    worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/android-stuff/kind.yml:        worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/android-stuff/kind.yml:        worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/android-stuff/kind.yml:        worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/android-stuff/kind.yml:        worker-type: aws-provisioner-v1/android-api-15
taskcluster/ci/l10n/kind.yml:        worker-type: aws-provisioner-v1/android-api-15

As you have done, we can combine the dbg- and opt-macosx64, so we just need gecko-L-b-macosx64 and gecko-L-b-android.  I've created those six (three levels for each).

So all that remains to be done is to change the references that appear above to point to the new names.

Comment 10

[Haard Panchal]

I found out the references to the old names, but I not sure which 'level' of the worker-type to use, as I do not about the level concept, and what the difference is between two different levels. Where can I find more information on this?

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

The levels are outlined here:
  https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/

We handle the levels by substituting `config.params['level']`, handled here
  https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/task.py#484
so you can just write
  worker-type: aws-provisioner/gecko-{level}-b-android
and you should get the correct output.

Comment 12

[Haard Panchal]

Attachment: renaming.patch

Please review the patch!

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8804197 [details] [diff] [review]
renaming.patch

Please set the r? flag when requesting a review :)

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8804197 [details] [diff] [review]
renaming.patch

Review of attachment 8804197 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good, except that you've re-introduced the deleted android-partner.yml.  Can you remove that file and attach a new patch?

Comment 15

[Haard Panchal]

Sure!

Comment 16

[Haard Panchal]

Attachment: Removed android-partner.yml

I hope this one is correct :)

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8804811 [details] [diff] [review]
Removed android-partner.yml

Review of attachment 8804811 [details] [diff] [review]:
-----------------------------------------------------------------

I'll test this in try..

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2ecbe1b5f8b

Comment 19

[Haard Panchal]

Will the patch be applied in the original source code or I'll have to do it myself?

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Looks like the try test was good, so I'll land this now.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c1d12c6ae210912b1e5005ff1cd952cdc605e593
Bug 1307771: rename opt-* and dbg-* and android-* workerTypes to gecko-L-b-*; r=dustin

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c1d12c6ae210