1425703 - Give gps access to generate and test AMIs for docker-worker

Status: RESOLVED FIXED

(Taskcluster :: General, enhancement)

Description

[Gregory Szorc [:gps]]

I (gps) want to hack on docker-worker to realize various performance wins and developer ergonomics improvements. Experience from the past few weeks shows that the current process of having to proxy AMI testing through someone on the TC team is annoying and time-consuming to the TC team.

I spoke with Dustin about things and he thinks it is OK for me to get permissions so I can test things with minimal involvement from TC personnel.

The access I need is as follows:

* Access to secrets so I can generate the "base" docker-worker AMI
* Access to TC AWS account so I can upload generated APIs to it
* Any other access needed to test produced AMIs

Just so we're explicit, my intent is to have the ability to produce ad-hoc AMIs and plug them into testing worker types to evaluate them. I don't plan on bulk updating AMIs used by workers. Although if I have that power, I may eventually use it. But if I do, I'll be sure to run things by TC personnel first.

needinfo dustin to weigh in and triage.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Wander, if limited to just testing AMIs, is this something we can set up?  Then there's no need for CoT keys, etc.  I think this would involve:

 - scopes for secrets (which secrets?)
 - scopes to modify workerTypes
 - an AWS account user

Comment 2

[Wander Lairson Costa]

(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> Wander, if limited to just testing AMIs, is this something we can set up? 
> Then there's no need for CoT keys, etc.  I think this would involve:
> 
>  - scopes for secrets (which secrets?)
When we generate the testing docker images locally, the scripts pull pulse and tc credentials from secrets service. I think this is what gps is talking about.

>  - scopes to modify workerTypes
>  - an AWS account user

I will set up permissions and AWS user for gps.

Comment 3

[Wander Lairson Costa]

scopes have been added. dustin is taking care of AWS and secrets repo.

Comment 4

[Wander Lairson Costa]

Can this be closed?

Comment 5

[Gregory Szorc [:gps]]

Yes and no.

I definitely have enough access to unblock most work. However, the scopes required to run *all* the tests doesn't match what's defined in docker-worker and the required scopes don't match what is granted to me. I was actually using a temporary access key under jonas's account to test things a few months ago :/

I would like to see the scope grants worked out to facilitate testing.

But if you want to close this, I understand.

Comment 6

[Wander Lairson Costa]

(In reply to Gregory Szorc [:gps] from comment #5)
> Yes and no.
> 
> I definitely have enough access to unblock most work. However, the scopes
> required to run *all* the tests doesn't match what's defined in
> docker-worker and the required scopes don't match what is granted to me. I
> was actually using a temporary access key under jonas's account to test
> things a few months ago :/
> 
> I would like to see the scope grants worked out to facilitate testing.
> 
> But if you want to close this, I understand.

Could you please list the scopes you need?

Comment 7

[Gregory Szorc [:gps]]

I can't remember. I just know that if you follow the instructions in the README and elsewhere that a few tests fail due to missing scopes. If you want me to burn a few hours to figure this out, needinfo me again. But I can't promise when I'll have time to hack on docker-worker. I'm pretty heads down with other projects right now.

Comment 8

[Wander Lairson Costa]

(In reply to Gregory Szorc [:gps] from comment #7)
> I can't remember. I just know that if you follow the instructions in the
> README and elsewhere that a few tests fail due to missing scopes. If you
> want me to burn a few hours to figure this out, needinfo me again. But I
> can't promise when I'll have time to hack on docker-worker. I'm pretty heads
> down with other projects right now.

Well, when you have a problem with that again, ping me on IRC.