642790 - Nagios check to verify that slaves have all expected keys

Status: RESOLVED DUPLICATE of bug 792836

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

Moving slaves from one silo to another carries manual intervention at this moment.
Ensuring that the right keys are in the right slave would require a verification check.

AFAIK there are three sets we want to ensure to be correct:
- production keys
- staging keys
- try keys

Something like this could help:
 cd ~/.ssh; sha1sum *.pub | sort

Comment 1

[Armen [:armenzg]]

Not sure how to programatically know which slaves are marked for production, which ones for try and which ones for staging.
Maybe the slave-alloc system has a DB somewhere.


Please update this doc once the bug is resolved:
https://wiki.mozilla.org/ReleaseEngineering/How_To/Move_a_Slave_Between_Production_and_Staging#SSH_Keys

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

This is a better job for puppet than nagios, since it's difficult to dynamically reconfigure nagios.

Comment 3

[Armen [:armenzg]]

OK.

I have added the following command into the documentation [1]:
ssh -i ~/.ssh/xrbld_dsa [e-mail]

[1] https://wiki.mozilla.org/ReleaseEngineering/How_To/Adjust_SSH_keys_on_a_slave

Comment 4

[Chris AtLee [:catlee]]

No need to dynamically reconfig nagios, the plugin just needs to be able to determine what kind of slave it's looking at.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

OK, bug 637686 will let us do that.

Comment 6

[Chris AtLee [:catlee]]

POC: https://github.com/catlee/tools/blob/08c99ea0f389f07a16af807ac2afb6029421b6c5/buildfarm/maintenance/check_ssh_keys.py

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I assume that's checking the .pub keys just as a proof of concept?  I don't copy the .pub keys around since they're unused on the slaves.

Aside from that, this looks pretty good.  I assume we'd distribute the script via puppet, which will also be distributing the keys, so it works for me.

Comment 8

[Chris AtLee [:catlee]]

(In reply to comment #7)
> I assume that's checking the .pub keys just as a proof of concept?  I don't
> copy the .pub keys around since they're unused on the slaves.
> 
> Aside from that, this looks pretty good.  I assume we'd distribute the script
> via puppet, which will also be distributing the keys, so it works for me.

Would work with private keys just as well.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Puppet will be verifying this in bug 792836.