624622 - use slavealloc to ensure slaves have correct SSH keys

Status: RESOLVED DUPLICATE of bug 792836

(Release Engineering :: General, defect, P3)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

puppet doesn't have to know about the keys - just their MD5's (or a prefix of the MD5 if we want to really be safe about it).  Use an Exec of a shell (or Python) script to verify that the md5's match, possibly rm'ing them if they're incorrect.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Good idea, assuming buildbot doesn't start if the keys are invalid.

Comment 2

[[e-mail] (:bhearsum)]

For some background, we used to do some syncing of keys, but stopped after getting very annoyed with soon-to-be-production or production-but-in-staging-temporarily slaves continually trying to use production keys in staging. It would be nice to have a way of dealing with these without disabling Puppet completely.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Having had a look at the keys, I think that the right thing for puppet to do is automatically blow away any non-staging keys on a staging box, and blow away any non-production keys on a production box.  It's up to us to manually get the correct keys on there.  Does that seem reasonable?

Comment 4

[[e-mail] (:bhearsum)]

I don't think that addresses the issue. A sample use case is bringing up a batch of new slaves, which will be considered "production" as far as Puppet is concerned, but they'll spend time on staging masters to sanity check them. In the scenario you describe, we would put the staging keys on them manually and then Puppet would blow them away.

A similar scenario is a slave that's acting up in production, and we pull it into staging temporarily for debugging.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, I see what you mean.  I was under the impression that staging (buildbot) slaves were always connected to staging-puppet.  Is there a good reason to have slaves spanning the two environments like that?  It seems prone to error to me (which is, I suppose, how this bug came up!).

This gets to a larger issue of what staging means, so I'll take that to release@

Comment 6

[[e-mail] (:bhearsum)]

It's more about reducing work/risk when we move slaves into production.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Now that I'm paying attention to slave-related keys, I've seen two bugs where we put prod keys on a try slave - one of them was bug 626230.

Comment 8

[Chris AtLee [:catlee]]

would having puppet talk to slavealloc to figure out what environment a slave is in, and therefore what keys it needs, address your issues Ben?

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Chris AtLee [:catlee] from comment #8)
> would having puppet talk to slavealloc to figure out what environment a
> slave is in, and therefore what keys it needs, address your issues Ben?

Yeah, it should.

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

is it worthwhile to teach slavealloc how to handle keys itself?

Comment 11

[Chris AtLee [:catlee]]

(In reply to John Ford [:jhford] from comment #10)
> is it worthwhile to teach slavealloc how to handle keys itself?

yeah, that's also an option.

Comment 12

[Chris AtLee [:catlee]]

bear said he'd take this on

Comment 13

[[e-mail] (:bhearsum)]

Dustin, looks like this will be fixed by bug 792836?

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

It's a different, weaker approach to it, yes.