996629 - Setup Puppet for Mozmill CI OS X machines in qa.scl3.mozilla.com

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that the Puppet infrastructure has been setup for us already a while ago, we want to get started in getting it used. Therefore the second step (after Ubuntu on bug 973535) will be to integrate the OS X machines. As of now we have all OS X flavors: 10.6, 10.7, 10.8, and 10.9.

Dustin, is there anything special I would have to obey for OS X? Or are those similar steps as for Ubuntu?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

It's a little easier, really, since most of the software is installed with OS X.

We may need to do some funny business to support installing OS X, though - that doesn't use the normal Linux kickstart process.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Is that something we can actually get done this quarter or is there any blocker I should be aware of, which could put our goal at risk? I assume Windows will be harder to do as OS X?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

We could certainly image a few machines this quarter.  Setting up the ability to re-image them easily might be more tricky.  I'm not sure if we'd want to use DeployStudio or JAMF, and I'm not sure if we'd want to try to set up flows to the systems used to do so in releng, or build independent infra for it.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so in regards of the type of Puppet support for this quarter, should be updates to various tools be fine? I mean when a new release of Flash or Java comes out, that we can easily deploy it via Puppet? I think that should be the very first thing we would like to have. I wouldn't worry about completely re-imaging a Mac Mini. Such a thing we could do later.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Assuming that those are easily deployed from a DMG, yes, that should be pretty easy.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Yes, via DMG and updating some application preferences like disabling automatic updates. So good to know! That's all what we most likely need for the beginning. Thanks Dustin!

What do you think, shall we start with OS X or Ubuntu? What's better to get started?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Since Ubuntu's not out yet, let's start with OS X.  Let me see what folks think is the best installation process.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so I talked with Dustin on IRC and here are the steps we have to do (if I'm not wrongly got something):

1. We need a dedicated Mac machine which acts as deploy machine. I have requested it via bug 997230
2. Install DeployStudio (http://www.deploystudio.com/Home.html - unclear which version is recommended)
3. Install Puppet and Facter: http://puppetagain.pub.build.mozilla.org/data/repos/DMGs/
4. Download http://hg.mozilla.org/build/puppet/file/tip/modules/puppet/files/puppetize.sh to /Users/root to one of the Minis
5. Run 'puppet agent --test --noop' to check if there are errors and what would be installed

Dustin, can you please check if all that is what we have to do?

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Well, those are two sets of instructions mushed together.  Both can be done in parallel.

Set up DeployStudio
1. We need a dedicated Mac machine which acts as deploy machine. I have requested it via bug 997230
2. Install DeployStudio (http://www.deploystudio.com/Home.html - unclear which version is recommended)

Puppetize a QA Mac
1. Install Puppet and Facter: http://puppetagain.pub.build.mozilla.org/data/repos/DMGs/
2. Download http://hg.mozilla.org/build/puppet/file/tip/modules/puppet/files/puppetize.sh to /Users/root to one of the Minis
3. Run ./puppetize.sh
4. Run 'puppet agent --test --noop' to check if there are errors and what would be installed

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sure, sorry for munging those together. So while we wait for our deploystudio instance, could you give us access to one we could already use?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

For testing purposes I will use a Mac Mini from staging. Here the steps:

1. Do the Puppetize steps from above for the mini
2. Add a new node for this host at https://github.com/mozilla/build-puppet/blob/master/manifests/qa-nodes.pp (https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain)
3. Include appropriate toplevel class (toplevel::base) and get it working
4. Figure out a possible slave class for custom setup steps like toplevel:mm-slave

It has been suggested to check user environments for easier testing: https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> 2. Download
> http://hg.mozilla.org/build/puppet/file/tip/modules/puppet/files/puppetize.
> sh to /Users/root to one of the Minis

That means we have to get the root user account activated. How to do this see:
http://support.apple.com/kb/ht1528

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Henrik Skupin (:whimboo) from comment #12)
> That means we have to get the root user account activated. How to do this
> see:
> http://support.apple.com/kb/ht1528

Actually this is quite complicated to do. Something easier is this: https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/man8/dsenableroot.8.html

> dsenableroot -r %password%

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dustin, I don't know what to do exactly when setting the new password for deployment on puppetmaster. The documentation on that is not that good enough. Would you mind to create a new password given that we both don't have the old one anymore? Thanks.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

I changed the password.  See https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets for background on how secrets are handled.  Basically, you first generate the crypt'd password with htpasswd, and then encrypt it with eyaml, then paste the resulting block into /etc/hiera/secrets.eyaml.  Then run 'puppet agent --test' on the affected hosts (in this case the puppetmaster itself) to put the result into action.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Dustin J. Mitchell [:dustin] from comment #15)
> crypt'd password with htpasswd, and then encrypt it with eyaml, then paste
> the resulting block into /etc/hiera/secrets.eyaml.  Then run 'puppet agent
> --test' on the affected hosts (in this case the puppetmaster itself) to put
> the result into action.

Thanks. But that is a step which I cannot do myself due to limited permissions on /etc/hiera. So thanks for updating.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Dustin J. Mitchell [:dustin] from comment #15)
> the resulting block into /etc/hiera/secrets.eyaml.  Then run 'puppet agent
> --test' on the affected hosts (in this case the puppetmaster itself) to put
> the result into action.

This actually fails for me with:

Info: Creating a new SSL key for puppetmaster1.qa.scl3.mozilla.com
Error: Could not request certificate: Error 400 on SERVER: this master is not a CA
Exiting; failed to retrieve certificate and waitforcert is disabled

I assume that can only be run as root.


Further I tried to run ./puppetize.sh on the OS X host (10.22.73.78) as root, but it also fails with:

Contacting puppet server puppet
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1207, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1174, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error Tunnel connection failed: 403 Forbidden>
Failed to get certificates; re-trying after delay

Might those two problems be related? Maybe you want to have a look at?

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Pretty much everything about puppet happens as root - editing /etc/hiera/secrets.eyaml, running puppet, and so on.

The tunnel connection problem was because the request was using the proxy configuration in the environment variables.  An unqualified 'puppet' doesn't match any of the exceptions.  Bug 997711 should clear that up.  I've run
  no_proxy=$no_proxy,puppet ./puppetize.sh
on mm-osx-107.qa.scl3.mozilla.com now.  I also put
  ssldir = /var/lib/puppet/ssl
in /etc/puppet/puppet.conf, since this is not the default value on OS X.  You can now run
  puppet agent --test --noop
to your heart's content.  You'll probably want to add --environment=hskupin to point it to your environment, etc.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I updated https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service for the proxy exclusion list for puppet and repos.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dustin, can you please tell me where the puppet repo is located on the puppetmaster? I would like to change it to use our new qa fork. Thanks.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Change the config in manifests/qa-config.pp.  I *think* that's sufficient, but I haven't changed production repos in a while.  You can also edit .hg/hgrc in /etc/puppet/production just to be sure.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Taskcluster is the future and Puppet is no longer on our list. So marking this bug as wontfix.