1359494 - [MDC1] fix ssl certs on new puppet masters

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[:dhouse]

I got the certs signed with the right root CA now (I believe), but I get this failure when I tried to use the releng-puppet2...mdc1 apt repo from my vm in scl3. I'm not sure if there is a file/config missing at this point or if this is still a certificate problem.

W: The repository 'https://10.49.48.22/repos/apt/Ubuntu-16.04 xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://puppetagain-apt.pvt.build.mozilla.org/repos/apt/custom/mig-agent xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://10.49.48.22/repos/apt/Ubuntu-16.04/dists/xenial/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca.pem CRLfile: none
E: Some index files failed to download. They have been ignored, or old ones used instead.

Comment 1

[Jake Watkins [:dividehex]]

I suspect the intermediate cert might not be loading correctly because of the format.

/var/lib/puppetmaster/ssl/git/ca-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crt

The releng-puppet1.srv.releng.mdc1 certs look fine and the typical curl and openssl s_client -connect output looks fine, but releng-puppet2 is a different story.  The cert chain cannot be verified.

Comment 2

[Jake Watkins [:dividehex]]

The root cause here was that the Master CA Cert was signed properly but it was not put into place properly.  The previous Master Server Cert was still active and needed to be revoked. Needless to say, this was an excellent exercise in rotating puppet ssl certificates.

There are 2 ways to go about correcting this, 1) manually rotate the certs or 2) remove them and the puppet file locks to allow puppet to handle re-establishing the certs.  I opted for the latter.

Started off making sure puppet didn't run while stuff was in transit.

rm -rf /etc/cron.d/puppetcheck

First, since the Master CA cert was signed with the wrong root CA cert, this also meant that the Master Server Cert and all other certs issued from this specific puppetmaster were invalid and needed to be revoked.  Luckily, only the Master Server cert was still in play (you can tell by the V for valid) and therefore I only needed to revoke that one.

[[e-mail] ca]# cat inventory.txt
V	220423205840Z		01	unknown	/CN=releng-puppet2.srv.releng.mdc1.mozilla.com/OU=PuppetMasters
R	220426114654Z	170427132024Z	02	unknown	/CN=log-aggregator1.srv.releng.mdc1.mozilla.com
R	220426132025Z	170428132216Z	03	unknown	/CN=log-aggregator1.srv.releng.mdc1.mozilla.com
R	220427132217Z	170428132701Z	04	unknown	/CN=log-aggregator1.srv.releng.mdc1.mozilla.com
R	220427132701Z	170428132724Z	05	unknown	/CN=log-aggregator1.srv.releng.mdc1.mozilla.com
R	220427132724Z	170428133007Z	06	unknown	/CN=log-aggregator1.srv.releng.mdc1.mozilla.com
R	220427133007Z	170428133430Z	07	unknown	/CN=log-aggregator1.srv.releng.mdc1.mozilla.com
R	220427133430Z	170428141001Z	08	unknown	/CN=log-aggregator1.srv.releng.mdc1.mozilla.com


openssl ca -revoke "certs/01.pem" -config "./openssl.conf"
Using configuration from ./openssl.conf
Revoking Certificate 01.
Data Base Updated


Next, I needed to remove the various bogus certs, keys and crls in /var/lib/puppetmaster/ssl/git. Use 'git rm'. NOTE: It is helpful to run git as puppetsync user, although the cron ssl sync will fix file ownership if run as root.

[[e-mail] git]# sudo -u puppetsync git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	deleted:    ca-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crl
	deleted:    ca-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crt
	deleted:    certdir/203b1008.0
	deleted:    certdir/203b1008.r0
	deleted:    master-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crt

[[e-mail] git]# sudo -u puppetsync git commit -m "Remove bogus releng-puppet2.srv.releng.mdc1 certs"
[master 87006f0] Remove bogus releng-puppet2.srv.releng.mdc1 certs
 5 files changed, 119 deletions(-)
 delete mode 100644 ca-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crl
 delete mode 100644 ca-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crt
 delete mode 120000 certdir/203b1008.0
 delete mode 120000 certdir/203b1008.r0
 delete mode 100644 master-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crt

Next, I removed the master key from pvt.  Make sure not to remove the master-ca.key!

[[e-mail] ssl]# rm pvt/master.key
rm: remove regular file `pvt/master.key'? y

After that, move the valid Master CA Cert into /var/lib/puppetmaster/ssl/git/ca-certs/releng-puppet2.srv.releng.mdc1.mozilla.com.crt

And remove the file lock prevent puppet from running the ssl_setup_root.sh

[[e-mail] ca]# rm -rf /var/lib/puppetmaster/ssl/ca/.setup-root-complete

Finally, manually execute puppet:

[[e-mail] ssl]# puppet agent --test --server=releng-puppet2.srv.releng.scl3.mozilla.com

This will run through the process of committing the Master Ca Cert to git, generating a Master CA CRL and commiting it, generate hashed symlinks in the certdir and commiting, and finally, generating and signing a Master Server cert/key and commiting the cert to git.

Last but not least, restart httpd to pickup the new Master Server cert and Master CA cert.

Testing the ssl connection:

cd /var/lib/puppet/ssl
openssl s_client -connect releng-puppet2.srv.releng.mdc1.mozilla.com:8140 -verify 3 -CAfile certs/ca.pem -cert certs/releng-puppet2.srv.releng.mdc1.mozilla.com.pem -key private_keys/releng-puppet2.srv.releng.mdc1.mozilla.com.pem


Top output should look like this:
verify depth is 3
CONNECTED(00000003)
depth=2 CN = PuppetAgain Base CA, emailAddress = [e-mail], OU = Release Engineering, O = "Mozilla, Inc."
verify return:1
depth=1 CN = CA on releng-puppet1.srv.releng.mdc1.mozilla.com
verify return:1
depth=0 CN = releng-puppet1.srv.releng.mdc1.mozilla.com, OU = PuppetMasters
verify return:1
---
Certificate chain
 0 s:/CN=releng-puppet1.srv.releng.mdc1.mozilla.com/OU=PuppetMasters
   i:/CN=CA on releng-puppet1.srv.releng.mdc1.mozilla.com
 1 s:/CN=CA on releng-puppet1.srv.releng.mdc1.mozilla.com
   i:/CN=PuppetAgain Base CA/emailAddress=[e-mail]/OU=Release Engineering/O=Mozilla, Inc.
---