1350506 - sea-puppet's puppet setup is horked.

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Edmund Wong (:ewong)]

This looks like bug 1311822. 

(No, I don't recall running puppet cert on sea-puppet)

puppet agent --test on -3 returns:

Error: Could not request certificate: Error 400 on SERVER: this master is not a CA
Exiting; failed to retrieve certificate and waitforcert is disabled

Comment 1

[Edmund Wong (:ewong)]

Following the instructions to that bug, I did:

1) backedup all the stuff in var/lib/puppetmaster/ssl/ca to ../backup.zip
2) removed the last entry in inventory.txt
3) removed the *.pems

rebooted puppetmaster.

rebooted -3.

waiting for it to return

Comment 2

[Edmund Wong (:ewong)]

did |puppet agent --trace --test|  on -3 and got:


Error: Could not request certificate: Error 400 on SERVER: this master is not a CA
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:207:in `is_http_200?'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/certificate/rest.rb:12:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:201:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/ssl/host.rb:207:in `certificate'
/usr/lib/ruby/site_ruby/1.8/puppet/ssl/host.rb:36:in `localhost'
/usr/lib/ruby/site_ruby/1.8/puppet/ssl/validator/default_validator.rb:27:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/ssl/validator.rb:27:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/ssl/validator.rb:27:in `default_validator'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http_pool.rb:35:in `http_instance'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:57:in `network'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:82:in `http_request'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:62:in `http_get'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:96:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:190:in `do_request'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/request.rb:264:in `do_request'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:190:in `do_request'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:90:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/certificate/rest.rb:12:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:201:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/ssl/host.rb:207:in `certificate'
/usr/lib/ruby/site_ruby/1.8/puppet/ssl/host.rb:326:in `wait_for_cert'
/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:478:in `wait_for_certificates'
/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:319:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:384:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:510:in `plugin_hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:384:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:488:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:384:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8
Exiting; failed to retrieve certificate and waitforcert is disabled

Comment 3

[Edmund Wong (:ewong)]

After wrestling with puppet and certificate chaining, I think I've managed to
unhork the puppet infrastructure.

The basic gist of fixing it I got from [1]


1) generate a Rootca self-signed cert
   i) generate rootCA CRL
2) generate a puppetmaster ca csr
   - sign the puppetmaster ca csr with the RootCA cert
   - Generate the puppetmaster CA CRL
     i) copy the ca-cert to /var/lib/puppetmaster/ssl/git/ca-certs
         (and rename to sea-puppet.community.scl3.mozilla.crt)
    ii) copy the puppetmaster's ca and rootca cert and crls to
         /var/lib/puppetmaster/ssl/git/certdir
         - then run the following script in that dir:

         for i in *.crl; do
            h=`openssl crl -hash -noout -in $i`
            fn=$h.r0
            echo "    Linking ${fn} to $i..."
            [ ! -f $fn ] && ln -s $i $fn
         done
         for i in *.pem; do
             h=`openssl x509 -hash -noout -in $i`
             fn=$h.0
             echo "    Linking ${fn} to $i..."
             [ ! -f $fn ] && ln -s $i $fn
         done

3) generate a puppetmaster (leaf) csr
   - sign puppetmaster (leaf) csr with the puppetmaster ca cert
      i) copy this puppetmaster leaf cert to
          /var/lib/puppetmaster/ssl/git/master-certs

4) generate the hosts (2->13 + sea-puppet + sea-master1) csrs
   - sign with the puppetmaster ca's cert
   - for each host:
      i) copy the crt to /var/lib/puppet/ssl/public_keys (rename to pem)
     ii) copy the crt also to /var/lib/puppet/ssl/certs (and rename to pem)
    iii) copy the key to /var/lib/puppet/ssl/private_keys (and rename to pem)
     iv) copy the rootca's crt to /var/lib/puppet/ssl/certs and rename to ca.pem)

5) then run |puppet agent -t| on every host.

[1] - https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Certificate_Chaining

So I'm going to ask Callek for confirmation on whether I did it right.

Comment 4

[Edmund Wong (:ewong)]

I think it's unhorked namely because puppet had overwritten my new mercurial
installation with the old one.

So I need to update the puppet mercurial module.

Comment 5

[Edmund Wong (:ewong)]

Attachment: [puppet] proposed patch

Comment 6

[Edmund Wong (:ewong)]

Attachment: [puppet] proposed patch

Comment 7

[Edmund Wong (:ewong)]

Attachment: [puppet] proposed patch (v3)

realized I hadn't flushed the repo version.

Comment 8

[Edmund Wong (:ewong)]

Attachment: [puppet] proposed patch (v4)

Comment 9

[Edmund Wong (:ewong)]

then on the puppetmaster:

1) copied the mozilla-python27-mercurial-3.9.1-1.el6.x86_64.rpm to 
   /data/repos/yum/releng/public/CentOS/6/x86_64

2) cd /data/repos/yum/releng/public/CentOS/6/x86_64

3) createrepo --update ./

then on each puppet slave (as root)

1) yum clean all
2) puppet agent -t

Comment 10

[Tony Mechelynck [:tonymec]]

FWIW, there has been (as of this writing) one successful L64 trunk hourly build since the outage which had started immediately after buildID=20170420023553.
This new build is at http://ftp.mozilla.org/pub/seamonkey/tinderbox-builds/comm-central-trunk-linux64/1492959533/ ; it has buildID=20170423075853 which is after comment #9 (2 hours after if the build ID is in Mozilla time zone or 9 hours after if in UTC).

Comment 11

[Tony Mechelynck [:tonymec]]

oops, got my arithmetic wrong: it is 5 hours earlier if in UTC.

Comment 12

[Edmund Wong (:ewong)]

fixed.