659005 - migrate scl-production-puppet to a VM

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This host is currently on iX hardware which is already showing signs of HD failure.  Let's migrate it to a KVM instance before sending it back to iX.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I'm working on building a puppet manifest to set this up, so it might take slightly longer than expected

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

This isn't a high priority at the moment, since we're not sending the ix systems back yet, but still needs to be done.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I made some good progress on this today - time is pressing again since we'll be taking this system down to replace its heatsink and fan.

I have the puppet master installed and mostly configured, but I'm missing some of the Apache configuration.  First, I'm not sure how best to generate these files:

        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem

and haven't looked up what they are yet.

Second, I need to adjust the DocumentRoot.  On other systems, this has been done by editing httpd.conf, but surely there's a more puppet-friendly way to do so.

The work so far is here:
  http://hg.mozilla.org/users/dmitchell_mozilla.com/puppet-manifests/rev/ae347bacded5

I went a little overboard with installing the prerequisites for Puppet.  I'll need to experiment to find the ramifications there for existing systems (slaves).  The circular dependencies kind of suck, too - you have to have the right version of ruby installed by hand (although it's hard to run puppet without that), and moreover you have to have the right version of augeas installed by hand.  Furthermore, puppet won't be able to upgrade any of these packages.  Yuck.  Still, in the glorious puppety future that's something we can fix with a well-scripted base image install.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> I have the puppet master installed and mostly configured, but I'm missing
> some of the Apache configuration.  First, I'm not sure how best to generate
> these files:
> 
>         SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
>         SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem

These are the SSL files generated by the Puppet master when it first starts IIRC. I know that they were definitely not generated by hand.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m659005-puppet-manifests-p1-r1.patch

This is functional, and with some careful placement of certificates and editing of /etc/hosts on clients, it can substitute for the existing scl-production-puppet without any need to re-sign certificates.

Once this is landed, I'll try reproducing the host "from scratch", then open a new bug to replace the existing scl-production-puppet with this one.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

This needs to get finished as part of my transition to IT.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I'm re-creating the VM now, so I can make sure that the puppet manifests work from scratch (except the circular dependencies..)

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Here's the from-bare-image install process:

scp [e-mail]:/var/cache/yum/epel/packages/*.rpm .
scp [e-mail]:/var/cache/yum/updates/packages/*.rpm .
rpm -i ruby-libs-1.8.5-5.el5_4.8.x86_64.rpm
rpm -i ruby-1.8.5-5.el5_4.8.x86_64.rpm 
rpm -i augeas-libs-0.7.0-1.el5.x86_64.rpm
rpm -i facter-1.5.7-1.el5.noarch.rpm
rpm -i ruby-augeas-0.3.0-1.el5.x86_64.rpm 
rpm -i ruby-shadow-1.4.1-7.el5.x86_64.rpm
rpm -i puppet-0.24.8-4.el5.noarch.rpm
puppetd --test --server=master-puppet1.build.scl1.mozilla.com --environment dustin
# (straighten out certs)
# (handle circular dep)
rpm -i --upgrade  http://master-puppet1.build.mozilla.org/production/centos5-x86_64/puppet-master/RPMs/augeas-{libs-,}0.8.1-2.el5.x86_64.rpm 
puppetd --test --server=master-puppet1.build.scl1.mozilla.com --environment dustin

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I discovered bug 674811 in this process, but that doesn't seem to block rolling this out.

Remaining "manual" steps for the seamless rollover:

# Copy over the puppet files.  Open question as to a better way to do this.
ssh scl-production-puppet tar -C /builds -cf - . | tar -C /N -xf -z

# check out manifests
hg clone http://hg.mozilla.org/build/puppet-manifests /etc/puppet/manifests

# fake the hostname
hostname scl-production-puppet.build.scl1.mozilla.com
vi /etc/puppet/puppet.conf (edit certname)
vi /etc/httpd/conf.d/puppetmaster.conf

# copy over the SSL stuff from the old server
rm -rf /var/lib/puppet/ssl; ssh scl-production-puppet tar -C /var/lib/puppet -cf - ssl | tar -C /var/lib/puppet -xf -

# and secrets.pp
scp scl-production-puppet:/etc/puppet/manifests/secrets.pp /etc/puppet/manifests

# restart the puppetmaster and Apache to use it
/etc/init.d/puppetmaster restart
/etc/init.d/httpd restart

# test it on a slave - add to /etc/hosts:
10.12.48.20 scl-production-puppet.build.scl1.mozilla.com
puppetd --server scl-production-puppet.build.scl1.mozilla.com --test --noop

# look, ma, no cert problems!

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m659005-puppet-manifests-p1-r2.patch

This rebases against tip and adds

diff --git a/modules/puppet/manifests/master/config.pp b/modules/puppet/manifests/master/config.pp
--- a/modules/puppet/manifests/master/config.pp
+++ b/modules/puppet/manifests/master/config.pp
@@ -21,6 +21,19 @@ class puppet::master::config {
         mode => '644'
     }
 
+    # RPMs don't create these dirs
+    file {
+        "/var/lib/puppet/yaml":
+            ensure => directory,
+            owner => "puppet",
+            group => "puppet",
+            require => Class['puppet::master::install'];
+        [ "/var/lib/puppet/yaml/node", "/var/lib/puppet/yaml/facts" ]:
+            ensure => directory,
+            owner => "puppet",
+            group => "puppet";
+    }
+
     # auth.conf
     file {
         "/etc/puppet/auth.conf":

to the previous patch, and also hopefully bumps it up in the patch queue.  I'll be happy with a review from either of you, but I *would* like to get this deployed sooner rather than later, since it's a finish-up-your-tasks thing I was supposed to have done by Tuesday :)

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 549023 [details] [diff] [review]
m659005-puppet-manifests-p1-r2.patch

Review of attachment 549023 [details] [diff] [review]:
-----------------------------------------------------------------

I think this is going to break releng-mirror01, which inherits "masternode" currently.

::: modules/packages/manifests/apache.pp
@@ +1,1 @@
> +class packages::apache {

Does it make sense to integrate this into the newly created "apache" module rather than put it here?

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to comment #11)
> I think this is going to break releng-mirror01, which inherits "masternode"
> currently.

Ah, yes, the patch bitrotted..

> Does it make sense to integrate this into the newly created "apache" module
> rather than put it here?

Hm, I should integrate them in general, yes.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m659005-puppet-manifests-p1-r3.patch

OK, well, here's the updated patch.  However, this will try to downgrade the puppet version on releng-mirror01, among others, since it is designed such that any puppet client runs 0.24.8.

I tried to fix this by making it install 0.25.5, but there were too many version interdepenencies to straighten out.  Using the EPEL led to
 package {
   "puppet":
      ensure => "0.25.5-1.el5";
 }
trying to install Puppet-2.6.7, which probably makes sense to someone familiar with yum.

In the interests of getting this done, I'd like to just make the switch to the new puppetmaster, without committing anything that would keep that master up to date.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

I was *so* close to doing this today .. then I realizd it's Friday evening and I should probably wait until Monday or so.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

I had enough help around to get me out of any messes, so I went ahead.  After synchronizing puppet-files and the manifests, that just meant swapping hostnames in DNS and DHCP; we now have scl-production-puppet{-old,}.bmo.

The new (KVM) master is now handing out catalogs as if it had been doing this forever.  Things look absolutely fine, but I'll keep my eyes on it for a while longer.  I'll file a bug to turn this host into a w32 slave sometime next week, when I'm confident we won't need it.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Amy also identified this missing line in nrpe.cfg:

# simple check on puppet master
command[check_puppet]=/usr/lib/nagios/plugins/check_procs -c 4:4 -C puppetmasterd