use slavealloc to ensure slaves have correct SSH keys

RESOLVED DUPLICATE of bug 792836

Status

P3
normal
RESOLVED DUPLICATE of bug 792836
8 years ago
5 years ago

People

(Reporter: dustin, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [slavealloc][buildslaves])

puppet doesn't have to know about the keys - just their MD5's (or a prefix of the MD5 if we want to really be safe about it).  Use an Exec of a shell (or Python) script to verify that the md5's match, possibly rm'ing them if they're incorrect.
Whiteboard: [puppet][buildslaves]
Good idea, assuming buildbot doesn't start if the keys are invalid.
For some background, we used to do some syncing of keys, but stopped after getting very annoyed with soon-to-be-production or production-but-in-staging-temporarily slaves continually trying to use production keys in staging. It would be nice to have a way of dealing with these without disabling Puppet completely.
Having had a look at the keys, I think that the right thing for puppet to do is automatically blow away any non-staging keys on a staging box, and blow away any non-production keys on a production box.  It's up to us to manually get the correct keys on there.  Does that seem reasonable?
I don't think that addresses the issue. A sample use case is bringing up a batch of new slaves, which will be considered "production" as far as Puppet is concerned, but they'll spend time on staging masters to sanity check them. In the scenario you describe, we would put the staging keys on them manually and then Puppet would blow them away.

A similar scenario is a slave that's acting up in production, and we pull it into staging temporarily for debugging.
Oh, I see what you mean.  I was under the impression that staging (buildbot) slaves were always connected to staging-puppet.  Is there a good reason to have slaves spanning the two environments like that?  It seems prone to error to me (which is, I suppose, how this bug came up!).

This gets to a larger issue of what staging means, so I'll take that to release@
It's more about reducing work/risk when we move slaves into production.
Now that I'm paying attention to slave-related keys, I've seen two bugs where we put prod keys on a try slave - one of them was bug 626230.
Assignee: dustin → nobody
Priority: -- → P3
Blocks: 516808
would having puppet talk to slavealloc to figure out what environment a slave is in, and therefore what keys it needs, address your issues Ben?
(In reply to Chris AtLee [:catlee] from comment #8)
> would having puppet talk to slavealloc to figure out what environment a
> slave is in, and therefore what keys it needs, address your issues Ben?

Yeah, it should.
is it worthwhile to teach slavealloc how to handle keys itself?
(In reply to John Ford [:jhford] from comment #10)
> is it worthwhile to teach slavealloc how to handle keys itself?

yeah, that's also an option.
bear said he'd take this on
Assignee: nobody → bear
Summary: use puppet to ensure slaves have correct SSH keys → use slavealloc to ensure slaves have correct SSH keys
Whiteboard: [puppet][buildslaves] → [slavealloc][buildslaves]
Assignee: bear → nobody
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
Dustin, looks like this will be fixed by bug 792836?
It's a different, weaker approach to it, yes.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 792836
You need to log in before you can comment on or make changes to this bug.