973535 - Setup Puppet for Mozmill CI Ubuntu machines in qa.scl3.mozilla.com

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that the Puppet infrastructure has been setup for us already a while ago, we want to get started in getting it used. Therefore the first step will be to integrate the Ubuntu machines. As of now we have Ubuntu 12.04 and 13.10 as 32bit and 64bit.

Dustin, would all dependencies be fixed for those two releases? Especially the clones of the Ubuntu 13.10 distribution and packages?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

We have Ubuntu 12.04 repositories in place already.  We do not have the 13.10 repositories mirrored, though it shouldn't be terribly difficult to add that mirror on your puppetmaster.

I'd recommend starting with 12.04.  Once that's working, 13.10 should fall into place relatively easily.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> We have Ubuntu 12.04 repositories in place already.  We do not have the
> 13.10 repositories mirrored, though it shouldn't be terribly difficult to
> add that mirror on your puppetmaster.

Most likely we will no longer need 13.10, but 14.04 instead. Given that this is the next LTS release of Ubuntu we will get rid of all of our current 12.04 and 13.10 machines, and only have 14.04. Given that we would need those builds soon, could you plan on mirroring it once it has been released? How long does this process usually take until we have a working mirror for a new version? Could there be some pitfalls for new releases with PuppetAgain?
 
> I'd recommend starting with 12.04.  Once that's working, 13.10 should fall
> into place relatively easily.

Sounds good. It's one of our goals this quarter to get Ubuntu and OS X puppetized, so we will get started soon.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Mirroring takes as long as an rsync does - so a matter of hours.  For 12.04, we mirrored everything, including the restricted and universe sections.  It's not totally clear that was necessary, and the result is 107G, well over half of the space required for the puppetmaster.  I believe that "universe" is the bulk of that.  Could we start by just mirroring main and restricted, and then perhaps cherry-picking files from universe as necessary?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sure the only additional software we really need for now is Adobe Flash and Java. Otherwise our system is mainly blank. For details please see:

https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service#QAAutomationESXService-Installation

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Great!  Can you ping me when 14 is released, then?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sure, will do. Probably we will work with Adrian or someone else first to get it running in the usual way. Once we are fine and don't see any problems, we could go ahead with Puppet for that version. I think this bug can become a tracking bug for all the work necessary. So dependent bugs will be filed then.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As a note, for testing puppet on Ubuntu we already have mm-ub-1204-32-temp.qa.scl3.mozilla.com (10.22.73.138) setup.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dustin, when I run 'puppet agent --test --noop' on that machine, I get the following failure:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: web_proxy does not support Ubuntu at /etc/puppet/production/modules/web_proxy/manifests/init.pp:18 on node mm-ub-1204-32-temp.qa.scl3.mozilla.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

That file has

        case $operatingsystem {
            CentOS: {
                file {
                    "/etc/environment":
                        content => template("${module_name}/environment.erb");
                }
            }
            default: {
                fail("${module_name} does not support ${operatingsystem}")
            }
        }

so, no case for Ubuntu.  Adding support on Ubuntu is now bug 997721.  As I mentioned in irc, Releng doesn't use the Mozilla proxies (we get a special legacy exception, lucky us), so we haven't had to deal with this particular issue.

Comment 10

[Andrei Eftimie]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> Great!  Can you ping me when 14 is released, then?

Ubuntu 14.04 is out:
http://releases.ubuntu.com/14.04/

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

I'm mirroring trusty to relabs now.  If that works OK and isn't too huge. I'll copy that over to the moco puppetmasters and we'll go from there.  I'll need to add a patch so that precise continues to look at the precise repo, with trusty looking at trusty.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Good news!  Without the universe section, it's only 30GB.  I'll put that on releng-puppet2 now, and it will mirror to the QA master from there.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dustin, is all the base work to support Ubuntu 14.04 done now? If yes we might want to re-assign this bug to myself.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

It's not, since I'm on PTO.  Jake (copied) is waiting on it too.

Comment 15

[Avi Halachmi (:avih)]

Bug 1014789 suggests that we regressed some 70% on svg opacity.

I'm not quite sure what this bug does, but I hope this is a technical issue rather than a real regression. Could someone please explain in a sentence or two what this bug fixes/changes, and if you think it could cause a real regression?

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Avi, this bug is about setting up some new hosts -- not a code change.  I suspect you've got the wrong bug.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

sorry, typo'd the bug originally

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dustin, we have to add your nodes as a new slave type. Given that we have slaves for mozmill-ci and tps-ci, I wonder who we should subclass them. I assume 'build' is currently used by RelEng for the buildbot builders. Shall we use 'qa-mozmill' and 'qa-tps', or without the QA prefix?

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are no longer interested on Puppet given that we make use of taskcluster now.