1213790 - Deploy boto config for release automation

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

I'm going to need to deploy a boto-style config to /builds/release-s3.credentials on linux builds slaves. Bug 1207023 will yield the values to put into this format:

[Credentials]
aws_access_key_id = <key>
aws_secret_access_key = <secret>

We'll have different values for production and staging (releng only, not loaners). After looking around I can't find a clear template for this in modules/slave_secrets, so I'm asking for some help.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I think this would look a lot like, for example
  http://hg.mozilla.org/build/puppet/file/21b5392348c3/modules/slave_secrets/manifests/google_api_key.pp
and key off $slave_trustlevel to figure out which creds to install.  But maybe I'm not understanding the wrinkle that makes it *not* look like that.  I didn't see anything about S3 in bug 1207023 that helped.

More to the point, though, as we think about transitioning to TaskCluster, we shouldn't be baking more secrets into our images.  What's the plan here?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Sorry, I meant bug 1210333 for the credentials, although that probably doesn't give you any more information. It's just AWS credentials for an IAM role, combined with our usual staging/production split so we don't do dumb things in staging by mistake.

If heira can do 'foo\nbar\nbaz' then that sounds like a good way to go.

(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> More to the point, though, as we think about transitioning to TaskCluster,
> we shouldn't be baking more secrets into our images.  What's the plan here?

It's a fair point. We explicitly made the choice that we are going to do this in two steps
* migrate the existing release promotion to cope with the whoop-whoop-evacuate-the-netapp project
* redo all of the release automation in taskcluster (aka build/release promotion)

So the lifetime of this extra secret is constrained, but time sensitive to ship in the short term.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Carry on then :)

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: wip

I'm hitting an error using this.

Setup
* releng-puppet2.srv.releng.scl3.mozilla.com
* /etc/puppet/environments/nthomas is prod 21b5392348c3 + this patch
* /etc/hiera/environments/nthomas_secrets.eyaml has release_s3_credentials_staging in it (no prod entry yet)
* |hiera release_s3_credentials_staging environment=nthomas| returns the right info

Over on my slave I get:
# puppet agent --test --environment=nthomas --server=releng-puppet2.srv.releng.scl3.mozilla.com
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item release_s3_credentials_staging in any Hiera data file and no default supplied at /etc/puppet/environments/nthomas/modules/slave_secrets/manifests/release_s3_credentials.pp:22 on node dev-linux64-ec2-nthomas.dev.releng.use1.mozilla.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Can you see my noob error ?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8672913 [details] [diff] [review]
wip

The patch looks good, and I don't see any error in what you've done.  I wonder if Hiera is not finding secrets in environments correctly?

Rather than debug that, perhaps try just adding that secret to the production secrets file -- if that works, then we can just update the wiki to indicate user environment hiera files don't work.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet] Deploy credentials

Adding the secrets to the production file worked so I was able to finish this off. I dropped a comment on the wiki about secrets environments not working.

The main changes are adding $_ensure, since puppet complained about overwriting the value of $ensure, allowing Darwin to get the file too, and docs.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8673442 [details] [diff] [review]
[puppet] Deploy credentials

http://hg.mozilla.org/build/puppet/rev/13d31b45ae4a
http://hg.mozilla.org/build/puppet/rev/5758bc71b80f

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #7)

Oops, that should be
http://hg.mozilla.org/build/puppet/rev/e9ec7af55b36
http://hg.mozilla.org/build/puppet/rev/5758bc71b80f

Also updated https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets#Secrets

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Verified deployed on mac and linux build slaves.