Bugzilla

Johan and myself gave this an initial glance earlier today. Before we report our status update and plan for Q1 with respect to this, let's recap a bit what this bug is all about, in a tl;dr.

--- Context ---
Since its debut, Taskcluster relied on scopes for its security model. These scopes are managed in a web UI and are usually a blocker for developers but also a hot potato for TC folks. They often need to drop work in order to click-click-click to add a certain scope for a missing user or alike. In order to avoid this in the future and improve this story, a self-serve model has been created. 

tl;dr - it relies on two pieces of logic:
* ciconfigs[1] - where no-code all-configs reside
* ciadmin[2] - the deployment and manipulation of scopes logic resides
* Dustin has a nice blogpost describing this[3]

The idea is to self serve the generation and management of the scopes to various subteams - under the review model - but to be applied by a small number of people that hold the key to TC. 
---

In the mobile world, we've ramped-up four different mobile products so far: focus, android-components, reference-browser and fenix. For each of these, setting up the scopes story has always been slightly messy. Setting up staging releases even more difficult. And so on.

Goal of this bug is to solve the scope issue in a way that's clean, simple, effective, generative and almost automated for future new products that may need ramping-up.

Johan and I spent some time today to discuss this bug, understand ciadmin, figure out how to integrate it in mobile world and how to break it down into smaller pieces that me and Johan can parallelize in the next weeks. That looks like the following:


Step 1:
a) Description: enable ciadmin for the github world (mobile world implicitly) in a backwards compatible way
b) What needs to happen:
* right now, the scopes are managed in the web UI. We need to change that and switch to ciadmin in a backwards compatible way
* enrich ciconfigs so that we also have github-based mobile products along with their scopes
* amend ciadmin repository to encompass the github world as well. Right now it only supports gecko + mercurial
* once these both are amended, get review from Dustin and Tom Prince
* before we can merge things, we can easily test the effects of our changes with `ci-admin diff` (ran in a clean state against production and in a "dirty" state with our latest changes)
c) What are the effects once this is done:
* UI should(?) say that roles are generated automatically and "please don't edit". Nothing else breaks. The new way of adding scopes becomes via ciadmin, instead of web UI click-click-click


Step 2:
a) Description: test adding a new dummy scope without touching the UI
b) What needs to happen:
* with step 1 completed, we should be able to amend scopes in a generative way, without touching the UI. We could try to add a dummy scope to an existing product to make sure we can validate our new model. 
* this means we'd have to tweak ci-condfigs via Phabricator, get review and approval to land that
* once ciconfigs changes are merged, we'd need someone that has access to run a `ciadmin apply` to deploy most recent changes
c) What are the effects once this is done:
* we validate our model and rely on UI no more. Better security and management of the scopes and one step closer to automation


Step 3:
a) Description: extend to multiple sets of scopes as pre-requisite for automation
b) What needs to happen:
* improve the ciadmin story so that we can generate pre-sets of scopes in ciconfigs. TBD how we'll organize this but likely we need to separate per product/groups/etc the sets of scopes from ciconfigs so that it's easier to add new sets of scopes, in a generative way for
** how does it look like for a new RelEnger? 
** how does it look like for a new developer in the mobile world?
** how does it look like for a new project/product that needs to be added? 
c) What are the effects once this is done:
* we should be in a position where adding new groups/users/products is a matter of one-liner(ish) in ciconfigs


Step 4:
a) Description: extend permissions to handle hooks as well
b) What needs to happen:
* taking this a step further, we could improve the generative behavior to generate hooks and staging hooks altogether. This means that for a given new product we should be able to easily provide the associated hooks (e.g. to trigger nightlies or alike)
c) What are the effects once this is done:
* we're taking the automation one step further. For a given new product we should be in a position to easily provide the end-to-end set of scopes, the staging releases to allow them to run their testing releases in PRs but also the hooks to trigger those staging releases. All in a nicely generative way. 


Step 5 (in conjunction with bug 1519494):
a) Description: automatize ourselves out of the job in ramping up new mobile products
b) What needs to happen:
* with work from bug 1519494 completed, we should be able to factorize and give new teams the full RelEnbg pipeline (scopes, staging releases, hooks, graphs)
c) What are the effects once this is done:
* enables us to self-serve everything and allow others use our "RelEng pipeline"

[1]: https://hg.mozilla.org/build/ci-configuration/
[2]: https://hg.mozilla.org/build/ci-admin/
[3]: http://code.v.igoro.us/posts/2018/06/ci-admin.html

Johan and myself gave this an initial glance earlier today. Before we report our status update and plan for Q1 with respect to this, let's recap a bit what this bug is all about, in a tl;dr.

## Context
Since its debut, Taskcluster relied on scopes for its security model. These scopes are managed in a web UI and are usually a blocker for developers but also a hot potato for TC folks. They often need to drop work in order to click-click-click to add a certain scope for a missing user or alike. In order to avoid this in the future and improve this story, a self-serve model has been created. 

tl;dr - it relies on two pieces of logic:
* ciconfigs[1] - where no-code all-configs reside
* ciadmin[2] - the deployment and manipulation of scopes logic resides
* Dustin has a nice blogpost describing this[3]

The idea is to self serve the generation and management of the scopes to various subteams - under the review model - but to be applied by a small number of people that hold the key to TC. 


In the mobile world, we've ramped-up four different mobile products so far: focus, android-components, reference-browser and fenix. For each of these, setting up the scopes story has always been slightly messy. Setting up staging releases even more difficult. And so on.

Goal of this bug is to solve the scope issue in a way that's clean, simple, effective, generative and almost automated for future new products that may need ramping-up.

Johan and I spent some time today to discuss this bug, understand ciadmin, figure out how to integrate it in mobile world and how to break it down into smaller pieces that me and Johan can parallelize in the next weeks. That looks like the following:


Step 1:
a) Description: enable ciadmin for the github world (mobile world implicitly) in a backwards compatible way
b) What needs to happen:
* right now, the scopes are managed in the web UI. We need to change that and switch to ciadmin in a backwards compatible way
* enrich ciconfigs so that we also have github-based mobile products along with their scopes
* amend ciadmin repository to encompass the github world as well. Right now it only supports gecko + mercurial
* once these both are amended, get review from Dustin and Tom Prince
* before we can merge things, we can easily test the effects of our changes with `ci-admin diff` (ran in a clean state against production and in a "dirty" state with our latest changes)

c) What are the effects once this is done:
* UI should(?) say that roles are generated automatically and "please don't edit". Nothing else breaks. The new way of adding scopes becomes via ciadmin, instead of web UI click-click-click


Step 2:
a) Description: test adding a new dummy scope without touching the UI
b) What needs to happen:
* with step 1 completed, we should be able to amend scopes in a generative way, without touching the UI. We could try to add a dummy scope to an existing product to make sure we can validate our new model. 
* this means we'd have to tweak ci-condfigs via Phabricator, get review and approval to land that
* once ciconfigs changes are merged, we'd need someone that has access to run a `ciadmin apply` to deploy most recent changes

c) What are the effects once this is done:
* we validate our model and rely on UI no more. Better security and management of the scopes and one step closer to automation


Step 3:
a) Description: extend to multiple sets of scopes as pre-requisite for automation
b) What needs to happen:
* improve the ciadmin story so that we can generate pre-sets of scopes in ciconfigs. TBD how we'll organize this but likely we need to separate per product/groups/etc the sets of scopes from ciconfigs so that it's easier to add new sets of scopes, in a generative way for
** how does it look like for a new RelEnger? 
** how does it look like for a new developer in the mobile world?
** how does it look like for a new project/product that needs to be added? 

c) What are the effects once this is done:
* we should be in a position where adding new groups/users/products is a matter of one-liner(ish) in ciconfigs


Step 4:
a) Description: extend permissions to handle hooks as well
b) What needs to happen:
* taking this a step further, we could improve the generative behavior to generate hooks and staging hooks altogether. This means that for a given new product we should be able to easily provide the associated hooks (e.g. to trigger nightlies or alike)

c) What are the effects once this is done:
* we're taking the automation one step further. For a given new product we should be in a position to easily provide the end-to-end set of scopes, the staging releases to allow them to run their testing releases in PRs but also the hooks to trigger those staging releases. All in a nicely generative way. 


Step 5 (in conjunction with bug 1519494):
a) Description: automatize ourselves out of the job in ramping up new mobile products
b) What needs to happen:
* with work from bug 1519494 completed, we should be able to factorize and give new teams the full RelEnbg pipeline (scopes, staging releases, hooks, graphs)

c) What are the effects once this is done:
* enables us to self-serve everything and allow others use our "RelEng pipeline"

[1]: https://hg.mozilla.org/build/ci-configuration/
[2]: https://hg.mozilla.org/build/ci-admin/
[3]: http://code.v.igoro.us/posts/2018/06/ci-admin.html

Johan and myself gave this an initial glance earlier today. Before we report our status update and plan for Q1 with respect to this, let's recap a bit what this bug is all about, in a tl;dr.

## Context
Since its debut, Taskcluster relied on scopes for its security model. These scopes are managed in a web UI and are usually a blocker for developers but also a hot potato for TC folks. They often need to drop work in order to click-click-click to add a certain scope for a missing user or alike. In order to avoid this in the future and improve this story, a self-serve model has been created. 

tl;dr - it relies on two pieces of logic:
* ciconfigs[1] - where no-code all-configs reside
* ciadmin[2] - the deployment and manipulation of scopes logic resides
* Dustin has a nice blogpost describing this[3]

The idea is to self serve the generation and management of the scopes to various subteams - under the review model - but to be applied by a small number of people that hold the key to TC. 


In the mobile world, we've ramped-up four different mobile products so far: focus, android-components, reference-browser and fenix. For each of these, setting up the scopes story has always been slightly messy. Setting up staging releases even more difficult. And so on.

Goal of this bug is to solve the scope issue in a way that's clean, simple, effective, generative and almost automated for future new products that may need ramping-up.

Johan and I spent some time today to discuss this bug, understand ciadmin, figure out how to integrate it in mobile world and how to break it down into smaller pieces that me and Johan can parallelize in the next weeks. That looks like the following:


## Step 1:
a) Description: enable ciadmin for the github world (mobile world implicitly) in a backwards compatible way
b) What needs to happen:
* right now, the scopes are managed in the web UI. We need to change that and switch to ciadmin in a backwards compatible way
* enrich ciconfigs so that we also have github-based mobile products along with their scopes
* amend ciadmin repository to encompass the github world as well. Right now it only supports gecko + mercurial
* once these both are amended, get review from Dustin and Tom Prince
* before we can merge things, we can easily test the effects of our changes with `ci-admin diff` (ran in a clean state against production and in a "dirty" state with our latest changes)

c) What are the effects once this is done:
* UI should(?) say that roles are generated automatically and "please don't edit". Nothing else breaks. The new way of adding scopes becomes via ciadmin, instead of web UI click-click-click


## Step 2:
a) Description: test adding a new dummy scope without touching the UI
b) What needs to happen:
* with step 1 completed, we should be able to amend scopes in a generative way, without touching the UI. We could try to add a dummy scope to an existing product to make sure we can validate our new model. 
* this means we'd have to tweak ci-condfigs via Phabricator, get review and approval to land that
* once ciconfigs changes are merged, we'd need someone that has access to run a `ciadmin apply` to deploy most recent changes

c) What are the effects once this is done:
* we validate our model and rely on UI no more. Better security and management of the scopes and one step closer to automation


Step 3:
a) Description: extend to multiple sets of scopes as pre-requisite for automation
b) What needs to happen:
* improve the ciadmin story so that we can generate pre-sets of scopes in ciconfigs. TBD how we'll organize this but likely we need to separate per product/groups/etc the sets of scopes from ciconfigs so that it's easier to add new sets of scopes, in a generative way for
** how does it look like for a new RelEnger? 
** how does it look like for a new developer in the mobile world?
** how does it look like for a new project/product that needs to be added? 

c) What are the effects once this is done:
* we should be in a position where adding new groups/users/products is a matter of one-liner(ish) in ciconfigs


## Step 4:
a) Description: extend permissions to handle hooks as well
b) What needs to happen:
* taking this a step further, we could improve the generative behavior to generate hooks and staging hooks altogether. This means that for a given new product we should be able to easily provide the associated hooks (e.g. to trigger nightlies or alike)

c) What are the effects once this is done:
* we're taking the automation one step further. For a given new product we should be in a position to easily provide the end-to-end set of scopes, the staging releases to allow them to run their testing releases in PRs but also the hooks to trigger those staging releases. All in a nicely generative way. 


## Step 5 (in conjunction with bug 1519494):
a) Description: automatize ourselves out of the job in ramping up new mobile products
b) What needs to happen:
* with work from bug 1519494 completed, we should be able to factorize and give new teams the full RelEnbg pipeline (scopes, staging releases, hooks, graphs)

c) What are the effects once this is done:
* enables us to self-serve everything and allow others use our "RelEng pipeline"

[1]: https://hg.mozilla.org/build/ci-configuration/
[2]: https://hg.mozilla.org/build/ci-admin/
[3]: http://code.v.igoro.us/posts/2018/06/ci-admin.html

Johan and myself gave this an initial glance earlier today. Before we report our status update and plan for Q1 with respect to this, let's recap a bit what this bug is all about, in a tl;dr.

## Context
Since its debut, Taskcluster relied on scopes for its security model. These scopes are managed in a web UI and are usually a blocker for developers but also a hot potato for TC folks. They often need to drop work in order to click-click-click to add a certain scope for a missing user or alike. In order to avoid this in the future and improve this story, a self-serve model has been created. 

tl;dr - it relies on two pieces of logic:
* ciconfigs[1] - where no-code all-configs reside
* ciadmin[2] - the deployment and manipulation of scopes logic resides
* Dustin has a nice blogpost describing this[3]

The idea is to self serve the generation and management of the scopes to various subteams - under the review model - but to be applied by a small number of people that hold the key to TC. 


In the mobile world, we've ramped-up four different mobile products so far: focus, android-components, reference-browser and fenix. For each of these, setting up the scopes story has always been slightly messy. Setting up staging releases even more difficult. And so on.

Goal of this bug is to solve the scope issue in a way that's clean, simple, effective, generative and almost automated for future new products that may need ramping-up.

Johan and I spent some time today to discuss this bug, understand ciadmin, figure out how to integrate it in mobile world and how to break it down into smaller pieces that me and Johan can parallelize in the next weeks. That looks like the following:


## Step 1:
a) Description: enable ciadmin for the github world (mobile world implicitly) in a backwards compatible way
b) What needs to happen:
* right now, the scopes are managed in the web UI. We need to change that and switch to ciadmin in a backwards compatible way
* enrich ciconfigs so that we also have github-based mobile products along with their scopes
* amend ciadmin repository to encompass the github world as well. Right now it only supports gecko + mercurial
* once these both are amended, get review from Dustin and Tom Prince
* before we can merge things, we can easily test the effects of our changes with `ci-admin diff` (ran in a clean state against production and in a "dirty" state with our latest changes)

c) What are the effects once this is done:
* UI should(?) say that roles are generated automatically and "please don't edit". Nothing else breaks. The new way of adding scopes becomes via ciadmin, instead of web UI click-click-click


## Step 2:
a) Description: test adding a new dummy scope without touching the UI
b) What needs to happen:
* with step 1 completed, we should be able to amend scopes in a generative way, without touching the UI. We could try to add a dummy scope to an existing product to make sure we can validate our new model. 
* this means we'd have to tweak ci-condfigs via Phabricator, get review and approval to land that
* once ciconfigs changes are merged, we'd need someone that has access to run a `ciadmin apply` to deploy most recent changes

c) What are the effects once this is done:
* we validate our model and rely on UI no more. Better security and management of the scopes and one step closer to automation


## Step 3:
a) Description: extend to multiple sets of scopes as pre-requisite for automation
b) What needs to happen:
* improve the ciadmin story so that we can generate pre-sets of scopes in ciconfigs. TBD how we'll organize this but likely we need to separate per product/groups/etc the sets of scopes from ciconfigs so that it's easier to add new sets of scopes, in a generative way for
** how does it look like for a new RelEnger? 
** how does it look like for a new developer in the mobile world?
** how does it look like for a new project/product that needs to be added? 

c) What are the effects once this is done:
* we should be in a position where adding new groups/users/products is a matter of one-liner(ish) in ciconfigs


## Step 4:
a) Description: extend permissions to handle hooks as well
b) What needs to happen:
* taking this a step further, we could improve the generative behavior to generate hooks and staging hooks altogether. This means that for a given new product we should be able to easily provide the associated hooks (e.g. to trigger nightlies or alike)

c) What are the effects once this is done:
* we're taking the automation one step further. For a given new product we should be in a position to easily provide the end-to-end set of scopes, the staging releases to allow them to run their testing releases in PRs but also the hooks to trigger those staging releases. All in a nicely generative way. 


## Step 5 (in conjunction with bug 1519494):
a) Description: automatize ourselves out of the job in ramping up new mobile products
b) What needs to happen:
* with work from bug 1519494 completed, we should be able to factorize and give new teams the full RelEnbg pipeline (scopes, staging releases, hooks, graphs)

c) What are the effects once this is done:
* enables us to self-serve everything and allow others use our "RelEng pipeline"

[1]: https://hg.mozilla.org/build/ci-configuration/
[2]: https://hg.mozilla.org/build/ci-admin/
[3]: http://code.v.igoro.us/posts/2018/06/ci-admin.html

Johan and myself gave this an initial glance earlier today. Before we report our status update and plan for Q1 with respect to this, let's recap a bit what this bug is all about, in a tl;dr.

## Context
Since its debut, Taskcluster relied on scopes for its security model. These scopes are managed in a web UI and are usually a blocker for developers but also a hot potato for TC folks. They often need to drop work in order to click-click-click to add a certain scope for a missing user or alike. In order to avoid this in the future and improve this story, a self-serve model has been created. 

tl;dr - it relies on two pieces of logic:
* ciconfigs[1] - where no-code all-configs reside
* ciadmin[2] - the deployment and manipulation of scopes logic resides
* Dustin has a nice blogpost describing this[3]

The idea is to self serve the generation and management of the scopes to various subteams - under the review model - but to be applied by a small number of people that hold the key to TC. 


In the mobile world, we've ramped-up four different mobile products so far: focus, android-components, reference-browser and fenix. For each of these, setting up the scopes story has always been slightly messy. Setting up staging releases even more difficult. And so on.

Goal of this bug is to solve the scope issue in a way that's clean, simple, effective, generative and almost automated for future new products that may need ramping-up.

Johan and I spent some time today to discuss this bug, understand ciadmin, figure out how to integrate it in mobile world and how to break it down into smaller pieces that me and Johan can parallelize in the next weeks. That looks like the following:


## ~Step 1~:
a) Description: enable ciadmin for the github world (mobile world implicitly) in a backwards compatible way
b) What needs to happen:
* right now, the scopes are managed in the web UI. We need to change that and switch to ciadmin in a backwards compatible way
* enrich ciconfigs so that we also have github-based mobile products along with their scopes
* amend ciadmin repository to encompass the github world as well. Right now it only supports gecko + mercurial
* once these both are amended, get review from Dustin and Tom Prince
* before we can merge things, we can easily test the effects of our changes with `ci-admin diff` (ran in a clean state against production and in a "dirty" state with our latest changes)

c) What are the effects once this is done:
* UI should(?) say that roles are generated automatically and "please don't edit". Nothing else breaks. The new way of adding scopes becomes via ciadmin, instead of web UI click-click-click


## ~Step 2~:
a) Description: test adding a new dummy scope without touching the UI
b) What needs to happen:
* with step 1 completed, we should be able to amend scopes in a generative way, without touching the UI. We could try to add a dummy scope to an existing product to make sure we can validate our new model. 
* this means we'd have to tweak ci-condfigs via Phabricator, get review and approval to land that
* once ciconfigs changes are merged, we'd need someone that has access to run a `ciadmin apply` to deploy most recent changes

c) What are the effects once this is done:
* we validate our model and rely on UI no more. Better security and management of the scopes and one step closer to automation


## Step 3:
a) Description: extend to multiple sets of scopes as pre-requisite for automation
b) What needs to happen:
* improve the ciadmin story so that we can generate pre-sets of scopes in ciconfigs. TBD how we'll organize this but likely we need to separate per product/groups/etc the sets of scopes from ciconfigs so that it's easier to add new sets of scopes, in a generative way for
** how does it look like for a new RelEnger? 
** how does it look like for a new developer in the mobile world?
** how does it look like for a new project/product that needs to be added? 

c) What are the effects once this is done:
* we should be in a position where adding new groups/users/products is a matter of one-liner(ish) in ciconfigs


## Step 4:
a) Description: extend permissions to handle hooks as well
b) What needs to happen:
* taking this a step further, we could improve the generative behavior to generate hooks and staging hooks altogether. This means that for a given new product we should be able to easily provide the associated hooks (e.g. to trigger nightlies or alike)

c) What are the effects once this is done:
* we're taking the automation one step further. For a given new product we should be in a position to easily provide the end-to-end set of scopes, the staging releases to allow them to run their testing releases in PRs but also the hooks to trigger those staging releases. All in a nicely generative way. 


## Step 5 (in conjunction with bug 1519494):
a) Description: automatize ourselves out of the job in ramping up new mobile products
b) What needs to happen:
* with work from bug 1519494 completed, we should be able to factorize and give new teams the full RelEnbg pipeline (scopes, staging releases, hooks, graphs)

c) What are the effects once this is done:
* enables us to self-serve everything and allow others use our "RelEng pipeline"

[1]: https://hg.mozilla.org/build/ci-configuration/
[2]: https://hg.mozilla.org/build/ci-admin/
[3]: http://code.v.igoro.us/posts/2018/06/ci-admin.html