Closed Bug 996629 Opened 8 years ago Closed 6 years ago

Setup Puppet for Mozmill CI OS X machines in qa.scl3.mozilla.com

Categories

(Mozilla QA Graveyard :: Infrastructure, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: whimboo, Assigned: whimboo)

References

()

Details

Given that the Puppet infrastructure has been setup for us already a while ago, we want to get started in getting it used. Therefore the second step (after Ubuntu on bug 973535) will be to integrate the OS X machines. As of now we have all OS X flavors: 10.6, 10.7, 10.8, and 10.9.

Dustin, is there anything special I would have to obey for OS X? Or are those similar steps as for Ubuntu?
It's a little easier, really, since most of the software is installed with OS X.

We may need to do some funny business to support installing OS X, though - that doesn't use the normal Linux kickstart process.
Is that something we can actually get done this quarter or is there any blocker I should be aware of, which could put our goal at risk? I assume Windows will be harder to do as OS X?
We could certainly image a few machines this quarter.  Setting up the ability to re-image them easily might be more tricky.  I'm not sure if we'd want to use DeployStudio or JAMF, and I'm not sure if we'd want to try to set up flows to the systems used to do so in releng, or build independent infra for it.
Ok, so in regards of the type of Puppet support for this quarter, should be updates to various tools be fine? I mean when a new release of Flash or Java comes out, that we can easily deploy it via Puppet? I think that should be the very first thing we would like to have. I wouldn't worry about completely re-imaging a Mac Mini. Such a thing we could do later.
Assuming that those are easily deployed from a DMG, yes, that should be pretty easy.
Yes, via DMG and updating some application preferences like disabling automatic updates. So good to know! That's all what we most likely need for the beginning. Thanks Dustin!

What do you think, shall we start with OS X or Ubuntu? What's better to get started?
Since Ubuntu's not out yet, let's start with OS X.  Let me see what folks think is the best installation process.
Ok, so I talked with Dustin on IRC and here are the steps we have to do (if I'm not wrongly got something):

1. We need a dedicated Mac machine which acts as deploy machine. I have requested it via bug 997230
2. Install DeployStudio (http://www.deploystudio.com/Home.html - unclear which version is recommended)
3. Install Puppet and Facter: http://puppetagain.pub.build.mozilla.org/data/repos/DMGs/
4. Download http://hg.mozilla.org/build/puppet/file/tip/modules/puppet/files/puppetize.sh to /Users/root to one of the Minis
5. Run 'puppet agent --test --noop' to check if there are errors and what would be installed

Dustin, can you please check if all that is what we have to do?
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Well, those are two sets of instructions mushed together.  Both can be done in parallel.

Set up DeployStudio
1. We need a dedicated Mac machine which acts as deploy machine. I have requested it via bug 997230
2. Install DeployStudio (http://www.deploystudio.com/Home.html - unclear which version is recommended)

Puppetize a QA Mac
1. Install Puppet and Facter: http://puppetagain.pub.build.mozilla.org/data/repos/DMGs/
2. Download http://hg.mozilla.org/build/puppet/file/tip/modules/puppet/files/puppetize.sh to /Users/root to one of the Minis
3. Run ./puppetize.sh
4. Run 'puppet agent --test --noop' to check if there are errors and what would be installed
Sure, sorry for munging those together. So while we wait for our deploystudio instance, could you give us access to one we could already use?
For testing purposes I will use a Mac Mini from staging. Here the steps:

1. Do the Puppetize steps from above for the mini
2. Add a new node for this host at https://github.com/mozilla/build-puppet/blob/master/manifests/qa-nodes.pp (https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain)
3. Include appropriate toplevel class (toplevel::base) and get it working
4. Figure out a possible slave class for custom setup steps like toplevel:mm-slave

It has been suggested to check user environments for easier testing: https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment
(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> 2. Download
> http://hg.mozilla.org/build/puppet/file/tip/modules/puppet/files/puppetize.
> sh to /Users/root to one of the Minis

That means we have to get the root user account activated. How to do this see:
http://support.apple.com/kb/ht1528
Depends on: 997385
(In reply to Henrik Skupin (:whimboo) from comment #12)
> That means we have to get the root user account activated. How to do this
> see:
> http://support.apple.com/kb/ht1528

Actually this is quite complicated to do. Something easier is this: https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/man8/dsenableroot.8.html

> dsenableroot -r %password%
Dustin, I don't know what to do exactly when setting the new password for deployment on puppetmaster. The documentation on that is not that good enough. Would you mind to create a new password given that we both don't have the old one anymore? Thanks.
Flags: needinfo?(dustin)
I changed the password.  See https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets for background on how secrets are handled.  Basically, you first generate the crypt'd password with htpasswd, and then encrypt it with eyaml, then paste the resulting block into /etc/hiera/secrets.eyaml.  Then run 'puppet agent --test' on the affected hosts (in this case the puppetmaster itself) to put the result into action.
(In reply to Dustin J. Mitchell [:dustin] from comment #15)
> crypt'd password with htpasswd, and then encrypt it with eyaml, then paste
> the resulting block into /etc/hiera/secrets.eyaml.  Then run 'puppet agent
> --test' on the affected hosts (in this case the puppetmaster itself) to put
> the result into action.

Thanks. But that is a step which I cannot do myself due to limited permissions on /etc/hiera. So thanks for updating.
(In reply to Dustin J. Mitchell [:dustin] from comment #15)
> the resulting block into /etc/hiera/secrets.eyaml.  Then run 'puppet agent
> --test' on the affected hosts (in this case the puppetmaster itself) to put
> the result into action.

This actually fails for me with:

Info: Creating a new SSL key for puppetmaster1.qa.scl3.mozilla.com
Error: Could not request certificate: Error 400 on SERVER: this master is not a CA
Exiting; failed to retrieve certificate and waitforcert is disabled

I assume that can only be run as root.


Further I tried to run ./puppetize.sh on the OS X host (10.22.73.78) as root, but it also fails with:

Contacting puppet server puppet
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1207, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1174, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error Tunnel connection failed: 403 Forbidden>
Failed to get certificates; re-trying after delay

Might those two problems be related? Maybe you want to have a look at?
Pretty much everything about puppet happens as root - editing /etc/hiera/secrets.eyaml, running puppet, and so on.

The tunnel connection problem was because the request was using the proxy configuration in the environment variables.  An unqualified 'puppet' doesn't match any of the exceptions.  Bug 997711 should clear that up.  I've run
  no_proxy=$no_proxy,puppet ./puppetize.sh
on mm-osx-107.qa.scl3.mozilla.com now.  I also put
  ssldir = /var/lib/puppet/ssl
in /etc/puppet/puppet.conf, since this is not the default value on OS X.  You can now run
  puppet agent --test --noop
to your heart's content.  You'll probably want to add --environment=hskupin to point it to your environment, etc.
Flags: needinfo?(dustin)
I updated https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service for the proxy exclusion list for puppet and repos.
Blocks: 997738
Depends on: 997721
Dustin, can you please tell me where the puppet repo is located on the puppetmaster? I would like to change it to use our new qa fork. Thanks.
Change the config in manifests/qa-config.pp.  I *think* that's sufficient, but I haven't changed production repos in a while.  You can also edit .hg/hgrc in /etc/puppet/production just to be sure.
Depends on: 1033352
Depends on: 1053977
Depends on: 1058559
Depends on: 1059141
Depends on: 1087284
Taskcluster is the future and Puppet is no longer on our list. So marking this bug as wontfix.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
QA Contact: hskupin
Resolution: --- → WONTFIX
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.