1519493 - Automated permission handling for mobile products

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Johan and myself gave this an initial glance earlier today. Before we report our status update and plan for Q1 with respect to this, let's recap a bit what this bug is all about, in a tl;dr.

Context

Since its debut, Taskcluster relied on scopes for its security model. These scopes are managed in a web UI and are usually a blocker for developers but also a hot potato for TC folks. They often need to drop work in order to click-click-click to add a certain scope for a missing user or alike. In order to avoid this in the future and improve this story, a self-serve model has been created.

tl;dr - it relies on two pieces of logic:

• ciconfigs1 - where no-code all-configs reside
• ciadmin2 - the deployment and manipulation of scopes logic resides
• Dustin has a nice blogpost describing this3

The idea is to self serve the generation and management of the scopes to various subteams - under the review model - but to be applied by a small number of people that hold the key to TC.

In the mobile world, we've ramped-up four different mobile products so far: focus, android-components, reference-browser and fenix. For each of these, setting up the scopes story has always been slightly messy. Setting up staging releases even more difficult. And so on.

Goal of this bug is to solve the scope issue in a way that's clean, simple, effective, generative and almost automated for future new products that may need ramping-up.

Johan and I spent some time today to discuss this bug, understand ciadmin, figure out how to integrate it in mobile world and how to break it down into smaller pieces that me and Johan can parallelize in the next weeks. That looks like the following:

Step 1:

a) Description: enable ciadmin for the github world (mobile world implicitly) in a backwards compatible way
b) What needs to happen:

• right now, the scopes are managed in the web UI. We need to change that and switch to ciadmin in a backwards compatible way
• enrich ciconfigs so that we also have github-based mobile products along with their scopes
• amend ciadmin repository to encompass the github world as well. Right now it only supports gecko + mercurial
• once these both are amended, get review from Dustin and Tom Prince
• before we can merge things, we can easily test the effects of our changes with ci-admin diff (ran in a clean state against production and in a "dirty" state with our latest changes)

c) What are the effects once this is done:

• UI should(?) say that roles are generated automatically and "please don't edit". Nothing else breaks. The new way of adding scopes becomes via ciadmin, instead of web UI click-click-click

Step 2:

a) Description: test adding a new dummy scope without touching the UI
b) What needs to happen:

• with step 1 completed, we should be able to amend scopes in a generative way, without touching the UI. We could try to add a dummy scope to an existing product to make sure we can validate our new model.
• this means we'd have to tweak ci-condfigs via Phabricator, get review and approval to land that
• once ciconfigs changes are merged, we'd need someone that has access to run a ciadmin apply to deploy most recent changes

c) What are the effects once this is done:

• we validate our model and rely on UI no more. Better security and management of the scopes and one step closer to automation

Step 3:

a) Description: extend to multiple sets of scopes as pre-requisite for automation
b) What needs to happen:

• improve the ciadmin story so that we can generate pre-sets of scopes in ciconfigs. TBD how we'll organize this but likely we need to separate per product/groups/etc the sets of scopes from ciconfigs so that it's easier to add new sets of scopes, in a generative way for
  ** how does it look like for a new RelEnger?
  ** how does it look like for a new developer in the mobile world?
  ** how does it look like for a new project/product that needs to be added?

c) What are the effects once this is done:

• we should be in a position where adding new groups/users/products is a matter of one-liner(ish) in ciconfigs

Step 4:

a) Description: extend permissions to handle hooks as well
b) What needs to happen:

• taking this a step further, we could improve the generative behavior to generate hooks and staging hooks altogether. This means that for a given new product we should be able to easily provide the associated hooks (e.g. to trigger nightlies or alike)

c) What are the effects once this is done:

• we're taking the automation one step further. For a given new product we should be in a position to easily provide the end-to-end set of scopes, the staging releases to allow them to run their testing releases in PRs but also the hooks to trigger those staging releases. All in a nicely generative way.

Step 5 (in conjunction with bug 1519494):

a) Description: automatize ourselves out of the job in ramping up new mobile products
b) What needs to happen:

• with work from bug 1519494 completed, we should be able to factorize and give new teams the full RelEnbg pipeline (scopes, staging releases, hooks, graphs)

c) What are the effects once this is done:

• enables us to self-serve everything and allow others use our "RelEng pipeline"

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [ciadmin] initial stub to tweak it

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [ciconfigs] initial stub to tweak it

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Today, Johan and me attempted to solve step 1. With initial patches above in ciadmin and ciconfigs we managed to get a look at the mobile-world related roles in TC without using the Web UI. The patches need polishing and extensions to include all the mobile products; full support is needed in ciadmin as it's now chained to gecko/mercurial world.

More to come soon.

Comment 5

[Tom Prince [:tomprince]]

(Note that both ci-admin and ci-configuration are managed in phabricator)

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

A bunch of releaseduty related automation issues took priority today on my plate and I won't be able to address this by EOW. I spoke with Tom last night and he confirmed that there are indeed ciadmin/ciconfigs assumptions that the repos are all hg.mozilla.org but ideally it should be straightforward to add support for github.

What I currently have now are some hacks on top of the patches from two weeks ago that are in these bug. If I manage to get some time by EOD, I'll polish and paste here; if not, I'll continue the other Monday when I'm back.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Status update: am currently polishing the patches to enable step 1. After briefly talking in RelEng automation meeting earlier today, the path going forward is:

• we identify cleanup items and nail them one by one
• we land the ciadmin stuff for android-components, reference-browser and fenix. We ignore focus-android for now
• if we can nicely clean the latter, we can switch it too. Otherwise, we can leave it be in web UI until further notice.

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1519493 - step 1 in automating permissions in mobile world.

This is a preliminary ask-for-feedback.

• not to be merged until we perform the cleanup in bug 1526017
• this is experimental to understand if the mapping logic can be optimized
• I'm not sure how to target the "staging releases" scopes yet other than mentioning the job ["pull-request", "branch" or alike]

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1519493 - step 1 in automating permissions in mobile world.

This is a preliminary ask-for-feedback in adding support for git in the ciadmin repo.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1519493 - fix checker in ci-admin r=tomprince

Fix checker which broke with D19246

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Status update: steps 1 and 2 from comment 1 are now done and deployed.

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1519493 - create cron roles for hooks to use instead. r=jlorenzo

Create cron prod/staging roles for hooks to use instead.

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

The hooks we have in the mobile world, use dedicated assume:hook-id... roles. Those roles are now pointing to equivalent cron:nightly and cron:nightly-staging roles which are maintained under ciadmin.

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

https://tools.taskcluster.net/auth/roles/hook-id%3Aproject-mobile%2Fandroid-components-snapshot-release now points to assume:repo:github.com/mozilla-mobile/android-components:cron:nightly.

In case things go south, we can point them back to the list of scopes as follows:

project:mobile:android-components:releng:beetmover:action:push-to-maven
project:mobile:android-components:releng:beetmover:bucket:maven-snapshot-production
queue:create-task:highest:aws-provisioner-v1/mobile-3-b-andrcmp
queue:create-task:highest:aws-provisioner-v1/mobile-3-decision
queue:create-task:highest:aws-provisioner-v1/mobile-3-images
queue:create-task:highest:scriptworker-prov-v1/mobile-beetmover-v1
queue:route:index.project.mobile.android-components.cache.level-3.docker-images.v1.*
queue:route:statuses
queue:scheduler-id:focus-nightly-sched

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

https://tools.taskcluster.net/auth/roles/hook-id%3Aproject-mobile%2Ffenix-nightly now points to assume:repo:github.com/mozilla-mobile/fenix:cron:nightly.

In case things go south, we can point them back to the list of scopes as follows:

project:mobile:fenix:releng:googleplay:product:fenix
project:mobile:fenix:releng:googleplay:product:fenix:beta
project:mobile:fenix:releng:googleplay:product:fenix:nightly
project:mobile:fenix:releng:signing:cert:beta-signing
project:mobile:fenix:releng:signing:cert:nightly-signing
project:mobile:fenix:releng:signing:cert:release-signing
project:mobile:fenix:releng:signing:format:autograph_apk
queue:create-task:highest:aws-provisioner-v1/gecko-focus
queue:create-task:highest:aws-provisioner-v1/github-worker
queue:create-task:highest:aws-provisioner-v1/mobile-3-b-fenix
queue:create-task:highest:aws-provisioner-v1/mobile-3-decision
queue:create-task:highest:scriptworker-prov-v1/mobile-pushapk-v1
queue:create-task:highest:scriptworker-prov-v1/mobile-signing-v1
queue:route:index.project.mobile.fenix.signed-nightly.*
queue:route:index.project.mobile.fenix.v2.beta.*
queue:route:[e-mail]-failed
queue:route:statuses
queue:route:tc-treeherder.v2.fenix.*
queue:scheduler-id:focus-nightly-sched
secrets:get:project/mobile/fenix/adjust
secrets:get:project/mobile/fenix/beta/adjust
secrets:get:project/mobile/fenix/beta/leanplum
secrets:get:project/mobile/fenix/beta/sentry
secrets:get:project/mobile/fenix/leanplum
secrets:get:project/mobile/fenix/sentry

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

https://tools.taskcluster.net/auth/roles/hook-id%3Aproject-mobile%2Ffenix-nightly-staging now points to assume:repo:github.com/mozilla-mobile/fenix:cron:nightly-staging.

In case things go south, we can point them back to the list of scopes as follows:

project:mobile:fenix:releng:googleplay:product:fenix:dep
project:mobile:fenix:releng:signing:cert:dep-signing
project:mobile:fenix:releng:signing:format:autograph_apk
queue:create-task:highest:aws-provisioner-v1/android-components-g
queue:create-task:highest:aws-provisioner-v1/github-worker
queue:create-task:highest:aws-provisioner-v1/mobile-1-b-fenix
queue:create-task:highest:aws-provisioner-v1/mobile-1-decision
queue:create-task:highest:scriptworker-prov-v1/mobile-pushapk-dep-v1
queue:create-task:highest:scriptworker-prov-v1/mobile-signing-dep-v1
queue:create-task:highest:scriptworker-prov-v1/mobile-signing-v1
queue:route:index.project.mobile.fenix.staging-signed-nightly.*
queue:route:index.project.mobile.fenix.v2.staging.*
queue:route:[e-mail]-failed
queue:route:statuses
queue:route:tc-treeherder.v2.fenix.*
queue:scheduler-id:focus-nightly-sched
secrets:get:garbage/staging/project/mobile/fenix/adjust
secrets:get:garbage/staging/project/mobile/fenix/leanplum
secrets:get:garbage/staging/project/mobile/fenix/sentry

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

https://tools.taskcluster.net/auth/roles/hook-id%3Aproject-mobile%2Freference-browser-nightly now points to assume:assume:repo:github.com/mozilla-mobile/reference-browser:cron:nightly.

In case things go south we can point them back to the list of scopes as follows:

project:mobile:focus:releng:googleplay:product:reference-browser
project:mobile:reference-browser:releng:googleplay:product:reference-browser
project:mobile:reference-browser:releng:signing:cert:release-signing
project:mobile:reference-browser:releng:signing:format:autograph_apk_reference_browser
queue:create-task:highest:aws-provisioner-v1/gecko-focus
queue:create-task:highest:aws-provisioner-v1/github-worker
queue:create-task:highest:aws-provisioner-v1/mobile-3-b-ref-browser
queue:create-task:highest:aws-provisioner-v1/mobile-3-decision
queue:create-task:highest:scriptworker-prov-v1/mobile-pushapk-v1
queue:create-task:highest:scriptworker-prov-v1/mobile-signing-v1
queue:route:index.project.mobile.reference-browser.nightly.*
queue:route:index.project.mobile.reference-browser.signed-nightly.*
queue:route:[e-mail]-failed
queue:route:[e-mail]-failed
queue:route:statuses
queue:route:tc-treeherder.v2.reference-browser.*
queue:scheduler-id:focus-nightly-sched
secrets:get:project/mobile/reference-browser/nimbledroid
secrets:get:project/mobile/reference-browser/sentry

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

https://tools.taskcluster.net/auth/roles/hook-id%3Aproject-mobile%2Freference-browser-nightly-staging now points to assume:assume:repo:github.com/mozilla-mobile/reference-browser:cron:nightly-staging.

Sadly, I forgot to back these up but it should be less harmful as this is staging. I already triggered the hook and it works as expected so I don't forsee any issues in the future.

Comment 19

[Mihai Tabara [:mtabara]⌚️GMT]

Steps 1-2-3 are done.
Step 4 is done partially because we don't have a consistent way of generating uniform hooks for all mobile projects. Not yet. With citaskgraph or decisionlib adoption, we will be able to standardize our way of generating the hooks content. When that happens, we will be able to generate nicely the contents of the hooks and also associate the corresponding scopes.

Until then, I think we can close this. Feel free to reopen if I missed anything.