1444467 - Check and fix any expiring puppet server certs

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Jake Watkins [:dividehex]]

Looks like there is a cert expiring soon. We should find out which cert this is and update it.  I'd also like to check the rest of the puppet certs CA/Masters/Leafs.

Warning: Certificate 'CA on releng-puppet2.srv.releng.scl3.mozilla.com' will expire on 2018-05-01T20:04:20GMT

Comment 1

[Jake Watkins [:dividehex]]

The cert in question here is the Master CA cert of releng-puppet2.srv.releng.scl3.mozilla.com

[[e-mail] ca-certs]# openssl x509 -enddate -noout -in releng-puppet2.srv.releng.scl3.mozilla.com.crt
notAfter=May  1 20:04:20 2018 GMT

Comment 2

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

Because of the large number of alerts that hit the IRC channels regarding the "CA Certs/CRLs expiring soon" I did acknowledge the following:

<nagios-releng-mdc1> 
	releng-puppet1.srv.releng.mdc1.mozilla.com
	releng-puppet2.srv.releng.mdc1.mozilla.com

<nagios-releng-mdc2> 
	releng-puppet1.srv.releng.mdc2.mozilla.com
	releng-puppet2.srv.releng.mdc2.mozilla.com

<nagios-releng> 
	releng-puppet1.srv.releng.scl3.mozilla.com
	releng-puppet2.srv.releng.scl3.mozilla.com
	releng-puppet1.srv.releng.usw2.mozilla.com
	nagios1.private.releng.scl3.mozilla.com

Comment 3

[Jake Watkins [:dividehex]]

The releng-puppet2.srv.releng.scl3 intermediate CA certificate is going to expire on 5/1.  This means all puppet agent certs signed/issued by this CA cert will become invalid as well.  I will attempt to script most of this but here is a generate layout of actions needed:
 
- make sure aws-manager(s) is generating aws agent certs against puppet masters anywhere else but scl3
  - releng-puppet2.srv.releng.scl3 CA has a lot of aws agents certs issued from it
  - including golden aws hosts
 
- disable getcert.cgi on releng-puppet2.srv.releng.scl3
  - this will prevent any new certs from being generated on this hosts CA
 
- revoke all current certs issued by releng-puppet2.srv.releng.scl and issue new certs from a releng-puppet1.srv.releng.mdc1
  - this is accomplished by running puppetize.sh on each host that has a cert currently issues from releng-puppet2.srv.releng.scl3.  You must also get the new cert issued from a different puppet master 
  - eg. 'PUPPET_SERVER=releng-puppet1.srv.releng.mdc1.mozilla.com ./puppetize.sh'
  - This will automatically produce a request to revoke the old cert from releng-puppet2.srv.releng.scl3 and generate a new cert sign by releng-puppet1.srv.releng.mdc1
  - Once the revoke requests have propagated to releng-puppet2.srv.releng.scl3, they will be revoked there.

- verify the releng-puppet2.srv.releng.scl3 CA inventory.txt file has revoked all EXISTING systems
  - There will be abandoned certificate that are still valid.  These are hosts that no longer exist and they can be safely ignored since they will be invalidated when the intermediate CA cert expires anyway

- replace the intermediate CA cert, key, and regen crl for releng-puppet2.srv.releng.scl3
  - generate a new key and csr
  - sign it with the root CA cert
  - put lock file in place on CA dir
  - cp the expiring CA cert, key to a backup location in case of rollback
  - rename expiring CA cert and regenerate its hash link
  - cp new cert, key to proper locations
    - certificate = /var/lib/puppetmaster/ssl/git/ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crt
    - private_key = /var/lib/puppetmaster/ssl/pvt/master-ca.key
  - regenerate CA crl
    - master_ca_crl="/var/lib/puppetmaster/ssl/git/ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crl"
  - generate crt/crl hash links
  - add and commit all these changes to local git repo (to be synced to other puppetmasters)
  - run ssl_git_sync.sh

- revoke and reissue master certificate
  - refer to make_master_cert.sh
  - master_key="/var/lib/puppetmaster/ssl/pvt/master.key"
  - master_cert="/var/lib/puppetmaster/ssl/git/master-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crt"

- enable getcert.cgi
  - log into a host and test generating a new cert with puppetize.sh against the new intermediate CA cert
  - if successful, run puppet on same host

- regenerate root crl, then add and commit it to git
  - This will officially invalidate the expiring intermediate CA cert and all certs signed by it
  - remove old CA cert and hash link from git

- monitor things
  - make sure nagios alerts for expired certs clear up
  - watch puppetmail for getcert.cgi errors
  - watch nagios for puppet freshness check alerts from clients ( an indication the host may have never successfully been issued a new cert )


There may be some gotchas and catch22's here so the instructions will probably change as I go along.

Comment 4

[Jake Watkins [:dividehex]]

Attachment: stop aws instances from puppetizing and using scl3 puppetmasters

Comment 5

[Jake Watkins [:dividehex]]

I think the most sure way to move hosts with certs issued from the expiring CA cert to a new CA is to build a cert renewing mechanism into puppet.  This would ensure any existing agents that are checking into puppet currently don't get missed.  It is safe to assume that if a puppet client already has a valid agent cert then it should have the ability to request a new certificate through the standard getcert.cgi process.  This would be similar to the puppetize.sh sans the forced puppet run and reboot.

Here is a rough estimate of how I see this working:
* puppet renew agent cert script placed on host
* cron job set to exec renew script; renew script quietly exits if deploypass file not present
* puppet exec determines if client should be allowed to renew
 - if CA is about to expire
 - if agent cert is going to expire
 - if current agent cert is issued from puppet CA that is going to be decommed (checked against a list of valid puppetmasters?)
* if puppet determines it's time to allow the host to renew, drop deploypass on disk which allows the renew script to run on its next cron pass
* on renew script success, shred deploypass otherwise alert via puppetmail or papertrail.  Cron will keep attempting in case the problem is server side and gets fixed.

I would much rather have a puppet agents self serve than trying to renew certs via fabric or ansible which means we will need to manually account for all host.  And that is bound to miss stuff.

:rail, :dhouse, sound good?

Comment 6

[Rail Aliiev [:rail]]

Sounds sane to me. Thanks for the detailed plan!

Comment 7

[:dhouse]

+1 I'm on board also.

Comment 8

[Jake Watkins [:dividehex]]

Attachment: draft of puppet cert renew bits

This is a rough draft of a (semi)working certificate renew mechanism.  I just wanted to solicit feedback and get a sanity check before I continue.  If it looks good, I'll refine it and resubmit for official review before landing.

This has only been tested on linux but should work on osx also.  Windows might be a different story.

Comment 9

[:dhouse]

Comment on attachment 8967206 [details] [diff] [review]
draft of puppet cert renew bits

Review of attachment 8967206 [details] [diff] [review]:
-----------------------------------------------------------------

looks great!

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8967206 [details] [diff] [review]
draft of puppet cert renew bits

Sorry for the tardy reply. The approach looks great to me!

Comment 11

[Jake Watkins [:dividehex]]

Attachment: bug_1444467_renew_cert.patch

This will land and begin rolling out to any host that runs puppet periodically (eg. servers)

Once this has taken affect, we can also apply it to puppet::atboot

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8968301 [details] [diff] [review]
bug_1444467_renew_cert.patch

Review of attachment 8968301 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/puppet/manifests/renew_cert.pp
@@ +37,5 @@
> +        $ensure_deploypass = true
> +    }
> +
> +    if $ensure_deploypass {
> +        file { '/var/lib/puppet/deploypass':

Even though renew_cert.sh deletes the file, can you delete it here as well if ensure_deploypass is false. Just being paranoid a bit. :)

Comment 13

[Jake Watkins [:dividehex]]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #12)
> Comment on attachment 8968301 [details] [diff] [review]
> bug_1444467_renew_cert.patch

> Even though renew_cert.sh deletes the file, can you delete it here as well
> if ensure_deploypass is false. Just being paranoid a bit. :)

Thanks Rail!  That is just the paranoia I'm looking looking for.  I'll change this to ensure present or absent on every run.

Comment 14

[Jake Watkins [:dividehex]]

Comment on attachment 8968301 [details] [diff] [review]
bug_1444467_renew_cert.patch

Checked in with suggested changes:

    if $ca_server != $issuer {
        $ensure_deploypass = 'file'
    } elsif $thirty_days_before_exp <= time() {
        $ensure_deploypass = 'file'
    } else {
        $ensure_deploypass = 'absent'
    }

    file { '/var/lib/puppet/deploypass':
        ensure    => $ensure_deploypass,
        mode      => filemode('600'),
        content   => secret('deploy_password'),
        show_diff => false,
        require   => File['/usr/local/bin/renew_cert.sh'];
    }


https://hg.mozilla.org/build/puppet/rev/d3a68c8122676f18f565b23fda06ff24da98f613
https://hg.mozilla.org/build/puppet/rev/749c7f92ad88777c7c642cf011c3fd9918ca89c1

Puppet lint fixes:
https://hg.mozilla.org/build/puppet/rev/c1c52917ea546cbaa84fb44415630f86d64c2902
https://hg.mozilla.org/build/puppet/rev/d1ae5a4c7f832e3e1cc82657cda15bb86420a7a1

Comment 15

[Jake Watkins [:dividehex]]

This deployment went quite well with only few minor issues.

First, these two hosts failed to get certs due to a lock file on the CA dir and both cleared up on the next cron run of the renew_cert.sh script.
Apr 17 10:55:37 pushsnapworker-1.srv.releng.use1.mozilla.com renew-cert-failed: Got incorrect certificates (!?); exiting
Apr 17 10:55:37 balrogworker-6.srv.releng.use1.mozilla.com renew-cert-failed: Got incorrect certificates (!?); exiting

Apr 17 11:10:03 pushsnapworker-1.srv.releng.use1.mozilla.com renew-cert-success: New puppet agent certificate installed on pushsnapworker-1.srv.releng.use1.mozilla.com issued from releng-puppet1.srv.releng.use1.mozilla.com
Apr 17 11:10:08 balrogworker-6.srv.releng.use1.mozilla.com renew-cert-success: New puppet agent certificate installed on balrogworker-6.srv.releng.use1.mozilla.com issued from releng-puppet1.srv.releng.use1.mozilla.com 

Second issue, both aws puppetmasters failed to renew their scripts:
Apr 17 10:45:06 releng-puppet1.srv.releng.use1.mozilla.com renew-cert-failed: /var/lib/puppet/certs.sh is empty; exiting
Apr 17 10:52:06 releng-puppet1.srv.releng.usw2.mozilla.com renew-cert-failed: /var/lib/puppet/certs.sh is empty; exiting

This is because the /etc/hosts file is managed differently than the puppetmasters in the data centers (because of cloud-init).  When resolving it's own host name, /etc/hosts returned the IPv6 localhost (::1) IP.  The getcert.cgi script failed because, one, the script only validates IPv4 addresses, and second, doesn't not include localhost IPs in the allowed network list.

To fix this, we need to add the IPv4 localhost to the allowed network list (already done) and have the renew_cert.sh script make the request by IPv4 address instead of by host name in order to force IPv4.

Comment 16

[Jake Watkins [:dividehex]]

Attachment: lookup IPv4 address and use that for cert request

This looks up the IPv4 address of the puppet server hostname and uses that to make the request to getcert.cgi

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8968652 [details] [diff] [review]
lookup IPv4 address and use that for cert request

Review of attachment 8968652 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

Comment 18

[Jake Watkins [:dividehex]]

Comment on attachment 8968652 [details] [diff] [review]
lookup IPv4 address and use that for cert request

Landed with small additional change:

-password_mgr.add_password(None, 'https://'+puppet_server, 'deploy', deploypass)
+password_mgr.add_password(None, 'https://'+puppet_server_ip, 'deploy', deploypass)


https://hg.mozilla.org/build/puppet/rev/ab7254fcb867f80776b86bf48865a6e1cbbcedfa
https://hg.mozilla.org/build/puppet/rev/c36967b1289e4ecb26e3376b7a626850de048553

Comment 19

[Jake Watkins [:dividehex]]

Attachment: apply to puppet::atboot

Now that the dust has settled, let's roll this out to puppet::atboot hosts.  I assume the golden image ec2 instances are not puppet::atboot clients.  If this is wrong, please correct me.

Comment 20

[Jake Watkins [:dividehex]]

Comment on attachment 8968697 [details] [diff] [review]
apply to puppet::atboot

https://hg.mozilla.org/build/puppet/rev/a59f9c542af03bd9ac126d76482b899bbbeffb61
https://hg.mozilla.org/build/puppet/rev/a34c40dd0bb9934bee1a9ff4110544d6160c82d0

Comment 21

[Jake Watkins [:dividehex]]

This certificate self renewal process has been very successful, most of the active hosts have renewed their certificate and all but a handful have migrated off of releng-puppet2.srv.releng.scl3.  I've gone through the cert inventory on the expiring CA and revoked ~500 valid certs for hosts that no longer existed.  The hosts that do remain are hosts that currently exist but haven't renewed their cert yet.  We need to determine why they haven't renewed off of the expiring CA and if we should manually intervene or revoke and decomm the host/instance.


These are Windows ix hosts in SCL3.  I'm not sure why they are running puppet.  :markco, :q?
V       190318140755Z           1CF0    unknown /CN=t-w732-ix-004.wintest.releng.scl3.mozilla.com
V       190627035851Z           2234    unknown /CN=t-w864-ix-003.wintest.releng.scl3.mozilla.com
V       190902165715Z           2428    unknown /CN=t-w732-ix-003.wintest.releng.scl3.mozilla.com
V       220103203050Z           3A2F    unknown /CN=t-w864-ix-011.wintest.releng.scl3.mozilla.com


These 3 ec2 instances are offline.  Should either terminate them and revoke the certs or turn them on to run puppet, then shut them back down?  :jlund?
V       200208182547Z           2759    unknown /CN=buildbot-master76.bb.releng.use1.mozilla.com
V       200208183202Z           275B    unknown /CN=buildbot-master94.bb.releng.use1.mozilla.com
V       200208202616Z           2764    unknown /CN=buildbot-master74.bb.releng.usw2.mozilla.com


This host seems to be offline.  Is it broken or just shutdown?  If it is broken, we should decomm and revoke.  :jlund?
V       220906131904Z           43FD    unknown /CN=bld-lion-r5-054.build.releng.scl3.mozilla.com


:tomprince, could you either 'hg pull -u' your puppet dev environment to include the self cert renewal or manually run puppetize.sh against a different puppetmaster?
V       230406050904Z           4BDF    unknown /CN=tb-bouncer-dev-1.srv.releng.use1.mozilla.com
V       230411203153Z           4C35    unknown /CN=tb-signing-4.srv.releng.usw2.mozilla.com
V       230411203153Z           4C36    unknown /CN=tb-signing-8.srv.releng.usw2.mozilla.com
V       230411203204Z           4C37    unknown /CN=tb-signing-6.srv.releng.usw2.mozilla.com
V       230411203213Z           4C38    unknown /CN=tb-signing-10.srv.releng.usw2.mozilla.com
V       230412165555Z           4C40    unknown /CN=tb-signing-2.srv.releng.usw2.mozilla.com

Comment 22

[Jake Watkins [:dividehex]]

I looks like Windows golden instances puppetize against a hard coded puppetmaster in /data/repos/Windows/puppettize.vbs.  I've changed the puppet master from releng-puppet2.srv.releng.scl3 to releng-puppet2.srv.releng.mdc1.

Comment 23

[Mark Cornmesser [:markco]]

You can revoke for the ix machines. Puppet is not currently installed on them.

Comment 24

[Jordan Lund (:jlund)]

(In reply to Jake Watkins [:dividehex] from comment #21)

> 
> These 3 ec2 instances are offline.  Should either terminate them and revoke
> the certs or turn them on to run puppet, then shut them back down?  :jlund?
> V       200208182547Z           2759    unknown
> /CN=buildbot-master76.bb.releng.use1.mozilla.com
> V       200208183202Z           275B    unknown
> /CN=buildbot-master94.bb.releng.use1.mozilla.com
> V       200208202616Z           2764    unknown
> /CN=buildbot-master74.bb.releng.usw2.mozilla.com
> 

These were disabled and stopped in bug 1410109. We can terminate them now.

> 
> This host seems to be offline.  Is it broken or just shutdown?  If it is
> broken, we should decomm and revoke.  :jlund?
> V       220906131904Z           43FD    unknown
> /CN=bld-lion-r5-054.build.releng.scl3.mozilla.com
> 
>

Last job was 10 days ago. Given low volume of requests for this pool in Buildbot, we can decomm this.

Jake, would you like Buildduty to:

- terminate the masters and remove them from slavealloc?
- decomm bld-lion-r5-054

Comment 25

[Jake Watkins [:dividehex]]

(In reply to Mark Cornmesser [:markco] from comment #23)
> You can revoke for the ix machines. Puppet is not currently installed on
> them.

Thanks!  These have been revoked:

V       190318140755Z           1CF0    unknown /CN=t-w732-ix-004.wintest.releng.scl3.mozilla.com
V       190627035851Z           2234    unknown /CN=t-w864-ix-003.wintest.releng.scl3.mozilla.com
V       190902165715Z           2428    unknown /CN=t-w732-ix-003.wintest.releng.scl3.mozilla.com
V       220103203050Z           3A2F    unknown /CN=t-w864-ix-011.wintest.releng.scl3.mozilla.com

Comment 26

[Jake Watkins [:dividehex]]

(In reply to Jordan Lund (:jlund) from comment #24)
> (In reply to Jake Watkins [:dividehex] from comment #21)
> 
> Jake, would you like Buildduty to:
> 
> - terminate the masters and remove them from slavealloc?
> - decomm bld-lion-r5-054

Yes, please!  I've revoked the 3 buildbot-masters and the bld-lion host certs.

Comment 27

[Tom Prince [:tomprince]]

You can revoke tb-bouncer-dev-1.srv.releng.use1.mozilla.com; I need to rebuild that anyway.

Comment 28

[Jake Watkins [:dividehex]]

tb-signing hosts have renewed their certificates :-)


Apr 20 09:44:03 ip-10-132-30-190 renew-cert-success: New puppet agent certificate installed on tb-signing-8.srv.releng.usw2.mozilla.com issued from releng-puppet1.srv.releng.usw2.mozilla.com
Apr 20 09:45:03 ip-10-132-30-163 renew-cert-success: New puppet agent certificate installed on tb-signing-6.srv.releng.usw2.mozilla.com issued from releng-puppet1.srv.releng.usw2.mozilla.com
Apr 20 09:50:03 tb-signing-4.srv.releng.usw2.mozilla.com renew-cert-success: New puppet agent certificate installed on tb-signing-4.srv.releng.usw2.mozilla.com issued from releng-puppet1.srv.releng.usw2.mozilla.com
Apr 20 09:51:04 tb-signing-2.srv.releng.usw2.mozilla.com renew-cert-success: New puppet agent certificate installed on tb-signing-2.srv.releng.usw2.mozilla.com issued from releng-puppet1.srv.releng.usw2.mozilla.com
Apr 20 09:53:03 tb-signing-10.srv.releng.usw2.mozilla.com renew-cert-success: New puppet agent certificate installed on tb-signing-10.srv.releng.usw2.mozilla.com issued from releng-puppet1.srv.releng.usw2.mozilla.com


This just leaves tb-bouncer-dev-1.srv.releng.use1.mozilla.com which doesn't seem to have a node def defined in the 'production' branch so it might have never completed puppetizing.

Comment 29

[Jake Watkins [:dividehex]]

(In reply to Tom Prince [:tomprince] from comment #27)
> You can revoke tb-bouncer-dev-1.srv.releng.use1.mozilla.com; I need to
> rebuild that anyway.

Thanks :tomprince, I'll revoke it.

Comment 30

[Jake Watkins [:dividehex]]

clear NI on q

Comment 31

[Jake Watkins [:dividehex]]

I've disabled puppet and /var/lib/puppetmaster/deploy/cgi-bin/getcert.cgi on releng-puppet2.srv.releng.scl3

[[e-mail] ca]# mv /etc/cron.d/puppetcheck ~/
[[e-mail] cgi-bin]# mv getcert.cgi getcert.cgi.disabled

This will prevent any new or existing hosts from generating certs against this CA while the CA cert is updated.

Comment 32

[Jordan Lund (:jlund)]

(In reply to Jake Watkins [:dividehex] from comment #26)
> (In reply to Jordan Lund (:jlund) from comment #24)
> > (In reply to Jake Watkins [:dividehex] from comment #21)
> > 
> > Jake, would you like Buildduty to:
> > 
> > - terminate the masters and remove them from slavealloc?
> > - decomm bld-lion-r5-054
> 
> Yes, please!  I've revoked the 3 buildbot-masters and the bld-lion host
> certs.

Buildduty do not have permission to terminate aws masters.

@apop - could you start a tracking bug for decommisioning Buildbot masters. Then add these three to the list. We can terminate them at a later date but we should be tracking state in a bug

Comment 33

[Adrian Pop]

I have created a new bug, where we can track all the updates for decommissioning the three buildbot masters

Comment 34

[Adrian Pop]

(In reply to Jordan Lund (:jlund) from comment #32)
> (In reply to Jake Watkins [:dividehex] from comment #26)
> > (In reply to Jordan Lund (:jlund) from comment #24)
> > > (In reply to Jake Watkins [:dividehex] from comment #21)
> > > 
> > > Jake, would you like Buildduty to:
> > > 
> > > - terminate the masters and remove them from slavealloc?
> > > - decomm bld-lion-r5-054
> > 
> > Yes, please!  I've revoked the 3 buildbot-masters and the bld-lion host
> > certs.
> 
> Buildduty do not have permission to terminate aws masters.
> 
> @apop - could you start a tracking bug for decommisioning Buildbot masters.
> Then add these three to the list. We can terminate them at a later date but
> we should be tracking state in a bug
I have created a new bug, where we can track all the updates for decommissioning the three buildbot masters
Here is the bug : https://bugzilla.mozilla.org/show_bug.cgi?id=1455741

Comment 35

[Jake Watkins [:dividehex]]

CA cert has been update!  These are approximately the steps to do so:

Stop cron so we aren't doing things to the CA and git repo while things are in transit
service crond stop

Remove master server cert and key
rm /var/lib/puppetmaster/ssl/pvt/master.key
rm /var/lib/puppetmaster/ssl/git/master-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crt

Remove CA master key, cert and crl
rm /var/lib/puppetmaster/ssl/pvt/master-ca.key
rm /var/lib/puppetmaster/ssl/git/ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crt
rm /var/lib/puppetmaster/ssl/git/ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crl

Remove old crt/crl hash links
rm /var/lib/puppetmaster/ssl/git/certdir/e5b961ca.0
rm /var/lib/puppetmaster/ssl/git/certdir/e5b961ca.r0

Reset CA db
rm /var/lib/puppetmaster/ssl/ca/inventory*
rm /var/lib/puppetmaster/ssl/ca/serial*
rm -rf /var/lib/puppetmaster/ssl/ca/certs
touch /var/lib/puppetmaster/ssl/ca/inventory.txt
echo "0001" > /var/lib/puppetmaster/ssl/ca/serial
mkdir /var/lib/puppetmaster/ssl/ca/certs

Move new key/cert in place( these were generated previously and signed by the puppet root crt/key )
mv /tmp/releng-puppet2.srv.releng.scl3.mozilla.com.crt /var/lib/puppetmaster/ssl/git/master-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crt
mv /tmp/releng-puppet2.srv.releng.scl3.mozilla.com.key /var/lib/puppetmaster/ssl/pvt/master-ca.key

Generate a CA master CRL
cd /var/lib/puppetmaster/ssl/ca
openssl ca -gencrl -config ./openssl.conf -out ../git/ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crl

Generate hashlinks
cd /var/lib/puppetmaster/git/certdir
ln -s ../ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crt $(openssl x509 -noout -hash -in ../ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crt).0
ln -s ../ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crl $(openssl crl -noout -hash -in ../ca-certs/releng-puppet2.srv.releng.scl3.mozilla.com.crl).r0

Commit git changes
cd /var/lib/puppetmaster/ssl/git
sudo -u puppetsync git add .
sudo -u puppetsync git commit -m "Remove old CA crt/key/crl and add new ones"

Force a git sync
sudo -u puppetsync /var/lib/puppetmaster/ssl/scripts/ssl_git_sync.sh

Make a new master server cert/key
/var/lib/puppetmaster/ssl/scripts/make_master_cert.sh

Run puppet to enable getcert.cgi and puppet
puppet agent -t

Restart apache just to be sure
/usr/sbin/apachectl restart

Restart cron
service crond start

Comment 36

[Jake Watkins [:dividehex]]

I'm going to call this r/f at this time.

As for the future, it looks like the other SCL3 puppetmaster will expire the same month we should be shutting down SCL3.  We won't have to go through the entire certs swap but we can easily invalidate the SCL3 puppetmasters when the time comes to migrate any hosts with signed certificates from CAs that are doomed to go down with SCL3.

[[e-mail] ca-certs]# for i in $(ls -1 *.crt); do echo $i: $(openssl x509 -noout -enddate -in $i); done
releng-puppet1.srv.releng.mdc1.mozilla.com.crt: notAfter=Apr 24 16:04:46 2020 GMT
releng-puppet1.srv.releng.mdc2.mozilla.com.crt: notAfter=Mar 12 21:34:16 2021 GMT
releng-puppet1.srv.releng.scl3.mozilla.com.crt: notAfter=Aug 20 15:09:52 2018 GMT
releng-puppet1.srv.releng.use1.mozilla.com.crt: notAfter=Mar 7 21:05:09 2020 GMT
releng-puppet1.srv.releng.usw2.mozilla.com.crt: notAfter=Mar 7 21:28:10 2020 GMT
releng-puppet2.srv.releng.mdc1.mozilla.com.crt: notAfter=Apr 24 16:01:55 2020 GMT
releng-puppet2.srv.releng.mdc2.mozilla.com.crt: notAfter=Mar 12 22:44:02 2021 GMT
releng-puppet2.srv.releng.scl3.mozilla.com.crt: notAfter=Apr 2 15:47:23 2021 GMT
root.crt: notAfter=May 20 02:22:50 2022 GMT