486615 - set-up centralized deployment of software/machine configuration on mac osx build slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Corey Shields [:cshields]]

This is a tracker for the test deployment of puppet to manage the osx build
slaves. Separate tracker bugs are filed for windows and linux.

Comment 1

[Corey Shields [:cshields]]

Puppet is installed on moz2-darwin9-slave08 and works, talking with test-mgmt.  No changes pushed out via manifests yet, but the installation was painless.  Will start writing and testing manifests from here, pending bug 491852 for testing any big changes.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

When moz2-darwin-slave08 comes back to the buildbot pool, we should put the macOSX-10.5.2-mini-ref-image onto it. That has the right version of XCode and so on, while the one taking in bug 491852 doesn't.

Comment 3

[Corey Shields [:cshields]]

So, the image on slave08 seems pretty well up to date with the refence image doc.  What should we use to test puppet on this node?  (it is already talking and working well with the puppet master)

Is there anything in the hopper to add to the image soon that we could test?  Or should we try to bring this up to date with the right version of XCode mentioned above?

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> So, the image on slave08 seems pretty well up to date with the refence image
> doc.  What should we use to test puppet on this node?  (it is already talking
> and working well with the puppet master)
> 
> Is there anything in the hopper to add to the image soon that we could test? 
> Or should we try to bring this up to date with the right version of XCode
> mentioned above?

Definitely want to do the Xcode update. For Mac, we really need to be able to take a from-scratch install and get it up to date. Is that something you can work on with this machine?

Comment 5

[[e-mail] (:bhearsum)]

Alright, so here's where we are on this:
In all of my tests against a mini just imaged with the ref image the Puppet manifests work 100% fine. There's still some issues with fresh 10.5.2 installs. Time is getting tighter, and there are some upcoming Mac platform changes wanted, so I want to get this rolled out sooner rather than later. Here's what I think we should do:
* Merge in the currently working Mac manifests with the checked in ones.
* Schedule time to, and deploy Puppet on the existing Mac slaves, doing staging minis plus one xserve first, followed by production.
* In parallel or afterwards, get the manifests working with a fresh 10.5.2 install. (Corey, I'm tagging you for this if you have time.)

Comment 6

[Corey Shields [:cshields]]

yeah, I've got time to work on the fresh installs..  (Ben knows this, I'm laying it out for everyone else) The problems we have on the fresh installs occur on the first run of puppet..  So, subsequent runs seem to iron out all of the failures.  This means to test each change made to puppet to see if the problem is fixed or not, requires a fresh install of the OS.  Its a bit of a painful process for the person reinstalling all of the time (Ben).

Comment 7

[[e-mail] (:bhearsum)]

Quick brain dump of deployment instructions:
wget install script
sudo ./puppet-deploy.sh
# then logon to Puppet server and accept the certificate with puppetca --sign
$slave.build.mozilla.org (or $slave.mozilla.org for try)
# this will do the initial setup and get puppet starting at boot
sudo puppetd --test --server staging-puppet.build.mozilla.org
# this will get Puppet running as a daemon
sudo launchctl load -w /Library/LaunchDaemons/com.reductivelabs.puppet.plist

I'll be cleaning those up and posting the deployment script soon.

Comment 8

[[e-mail] (:bhearsum)]

We're done here! Big thanks to Corey for all his work on this, and to John Ford for rolling it out.