486614 - set-up centralized deployment of software/machine configuration on linux build slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Corey Shields [:cshields]]

This is a tracker for the test deployment of puppet to manage the linux build slaves. Separate tracker bugs are filed for windows and osx.

Comment 1

[Corey Shields [:cshields]]

Attachment: possible manifest layout and organization (not all-inclusive of course)

This is how I would layout and organize puppet manifests.  The idea with puppet is to try and reuse as much code as possible, and so the smaller these packages can be broken down the more possible that becomes.  There is a lot of importing of packages involved, you start with a list of a few top level packages depending on what the system is. 

Take for example a linux build slave, named slave-02, puppet would be instructed that slave-02 imports base.pp, centos.pp, and buildslave.pp  which would import the rest of the necessary manifests from there.

Comment 2

[Corey Shields [:cshields]]

Current state of the Linux side:

test-mgmt.build.mozilla.org is setup with puppet as a puppet server, and test-linslave.build.mozilla.org is setup as a puppet client.  I have a few manifest files begun but mostly just enough to be sure that the above pair are working together.

Next steps will involve building and testing manifests that fit the configuration of the build slaves.

Comment 3

[Corey Shields [:cshields]]

Quick update:

I've got configs written to go through Version 3 of the reference image as outlined in https://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0

(in other words, a base centos install would be a full version 3 image after a puppet run)  I hope to get Ben in to look at it soon, but given the all-hands this may have to be next week.

Comment 4

[Corey Shields [:cshields]]

Ok, most all of the reference image is recreated in puppet manifests for the Linux slaves.  I've done this mostly because the image I was given had very little of the reference image steps in it (as opposed to the osx slave which seems quite up to date).

We have made use of an nfs share to store the package files.  This means that to install a package the client can just source it from the share rather than having to copy it over the network first.  This also cuts down half of the code in the puppet manifests having to deal with copying these files.

I have left configuration files on the puppet server and continue to use the file-copy mechanisms to handle those.  

I have a bit of cleanup from the NFS migration and another file to migrate yet, and then this can be considered done and moved into staging.

Comment 5

[[e-mail] (:bhearsum)]

I've been working Corey and fiddling with this in staging over the past week. This is shaping up quite well. I've been dumping the manifest files into http://hg.mozilla.org/users/bhearsum_mozilla.com/puppet-manifests. So far, with some tweaks to what Corey initially gave me I was able to:
* Setup staging-puppet as the master for staging machines
* Setup puppetd on moz2-linux-slave04 and have it sync up against staging-puppet without any errors
* Create a manifest file that installs the Python 'virtualenv' package.

Great progress here, mostly due to all of Corey's hard work.

Comment 6

[[e-mail] (:bhearsum)]

I deployed puppet on the Linux staging slaves today: moz2-linux-slave03, 04 and 17. I still need to deploy on try-linux-slave05, that may need firewall changes though. I'll look into that today.

Comment 7

[[e-mail] (:bhearsum)]

I landed all of the distrib files in the new mofo module, puppet-files, and created a checkout of it on the NFS share we're using.

Still need to import the manifests from my user repository to the real one.

Comment 8

[[e-mail] (:bhearsum)]

I just pushed some initial content to http://hg.mozilla.org/build/puppet-manifests and updated the checkout on staging-puppet.

Comment 9

[[e-mail] (:bhearsum)]

Going to steal this bug for the last few bits - thanks for all your hard work on this Corey.

Comment 10

[[e-mail] (:bhearsum)]

All the deps are done, this bug is FIXED!