1218512 - Limit scopes for aws-provisioner2

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This service currently has a '*' scope, which is too broad.

John, AIUI the provisioner secrets component uses this clientId to create temporary credentials based on the workerType's scopes.  So at best we could limit this client's scopes to the union of all workerTypes' scopes, plus whatever scopes it needs to make other API calls (probably nothing - it's just polling queues, right?)

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

See bug 1217088.
The entire story is there... Should be easy... Most of it is button pushing...

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

that means that the temporary credentials that are issued to the instances are only valid for those operations.  If a worker needs to have more scopes, then we need to bake in a set of credentials to the worker type definition or we need to use these temp credentials to fetch more permissive credentials.

We would need to have credentials limited to:

1. all workerType management functions
2. queue.pendingTasks
3. azure tables when we switch to sas
4. all scopes in all worker types's scope lists

I guess we could have two sets of credentials, one that the provisioner uses internally to do its operations that is not used for generating temporary credentials, and a second set that has the ability to generate temporary credentials.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Well, if a worker needs to have more scopes, we just modify the worker's role to give it those scopes -- easy :)

So I think that the aws-provisioner's role should look like this:

  queue:pending-tasks:aws-provisioner-v1/* (2)
  auth:azure-table-access:taskclusterv1/SomeTableName (3)
  assume:worker-type:aws-provisioner-v1/* (4)

per irc, (1) means aws-provisioner:<verb>:.. scopes, which the provisioner itself doesn't need, since it doesn't call its own API methods.  And (3) sounds like it's not needed yet, but is easily accomplished when it is needed.

I'll get started reflecting scope lists from the aws provisioner's workerTypes list into roles.  I'll also replace those lists with ['assume:worker-type:blahblah'].  Then I'll update the aws-provisioner's role as above.  If nothing breaks, then the last bit will be to just dynamically assign the workerType's role in the aws provisioner's code, removing the scopes configuration in the workerTypes list.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

The scope lists for all of the workerTypes is ["*"].  That's partly because bug 1134342 isn't done yet, but we can be a *little* more restrictive given the magic of roles.

I set each workerType's scopes to `assume:worker-type:aws-provisioner-v1/<workerType>`.

I've configured the role `worker-type:aws-provisioner-v1/*` to
    assume:repo:*
    assume:worker-id:*
    queue:create-artifact:*
    queue:pending-tasks:aws-provisioner-v1/*  <-- parameterized roles would be helpful here
    queue:poll-task-urls
    queue:resolve-task
    queue:claim-task

which captures all of the scopes that might be necessary for a build from a repo, as well as those the worker itself requires, and includes the (more or less meaningless) worker-id role that the queue operations require.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Actually, I moved `assume:worker-id:*` into the workerType scopes.  Bug 1220750 covers doing that automatically, where actual region and instance IDs can be filled in.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

This failed pretty badly.

First, aws-provisioner was accidentally running under jhford's clientId.

Having fixed that, it seems to still be generating 

 docker-worker:middleware:artifact_extractor Error: ext.certificate issuer doesn't have sufficient scopes
 docker-worker:middleware:artifact_extractor Artifact upload public/logs/ failed, Error: Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes, as JSON: {} Error: Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes | Error: ext.certificate issuer doesn't have sufficient scopes

when I remove the * scope from client-id:aws-provisioner.

In this case, the temporary credential's scopes are not satisfied by the client's scopes.  The temp creds have

    "assume:worker-type:aws-provisioner-v1/opt-linux64",
    "assume:worker-id:*"

and I had missed the `assume:worker-id:*` scope in the `client-id:aws-provisioner` role.

Having added that, I'm not seeing more errors.  Leaving open to track that for a bit.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Looking good. I was momentarily fooled by timezone confusion with papertrail, and then by papertrail temporarily losing a few hours' data.  But all good.