886226 - Add a Vagrantfile for the Linux64 build environment

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

I think it would be cool if the tree contained a Vagrantfile and provisioning script to recreate the CentOS 6.2 Linux64 build environment, complete with mock_mozilla.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Add a Vagrantfile for creating the Linux64 build environment

And here is my first stab at it. Once I publish the CentOS 6.2 image I
created (will need to do it tomorrow once I have a faster internet
connection), you should be able to:

  cd config/bootstrap-linux64
  vagrant up
  # Wait a few minutes for the VM to initialize, download package
  # updates, populate mock_mozilla, etc.
  vagrant ssh
  mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/me/build
--unpriv --shell /bin/bash
  ./mach build

I currently mount the cloned source directory as /builds/slave/me/build,
which is fine as the regular ssh user. However, it gets unmounted in the
mock environment. I need to do some homework.

For this to get checked into the tree, I'd like to have the Vagrant .box
file published somewhere somewhat official. Unless people counts, I may
start asking around for hosting. It should be a one-time thing,
methinks.

Comment 2

[Gregory Szorc [:gps]]

needinfo? me to remind me to publish the CentOS box image so others can play around.

Comment 3

[Justin Wood (:Callek)]

I recall :dustin telling someone (you) this past week about how |puppetagain.pub.build.mozilla.org| is *not* meant for anyone to actually use as a repo source. So I don't think we want to actually publish this without involving mozilla IT to consider putting a .pub. repo somewhere other than on our real puppetmaster.

Comment 4

[(not currently active) Ted Mielczarek]

FYI: http://ftp.mozilla.org/pub/mozilla.org/mozilla/VMs/

is where we put the (old) copy of the Linux refplatform VM.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

:gps, "should be a one-time thing" is almost certainly incorrect - the build hosts change frequently, so this snapshot would need to have some way to stay up to date.

Internally, we don't use snapshots at all - we use a Base CentOS 6.2 image, and install/configure everything from there with puppet, on each host.  We've considered using snapshots for efficiency, but haven't seen the need yet.  If you do this with Vagrant, please take the approach that the snapshot is just an optimization of the canonical base-image-plus-puppet process.

The puppet implementation in question is PuppetAgain
  https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain
Among its goals is to make the configuration of releng systems transparent, so this suggestion seems quite appropriate.

Ideally this would get linked in with the puppetagain infrastructure, so that new hosts are automatically kept up to date.  Catlee has played with using 'puppet apply' for that purpose, but I tend to distrust that tool since, without a fileserver.conf, it doesn't properly account for non-modules puppet:/// resources.

Unfortunately, the data attached to PuppetAgain is a fundamental part of its configuration - and it's 260+GB.  Not really something we can expect a developer to set up on their laptop.  The current "public" exposure of the PuppetAgain data is http://puppetagain.pub.build.mozilla.org, but that has a few problems:
 - it doesn't expose any private data, some of which is needed for construction of builders
 - it is single-hosted on a production puppet master, so high load could cause production failures
 - its URI structure does not match that of a puppetmaster's internally exposed HTTP service
We could fix the second and third points without too much trouble, given a coherent plan that requires it.

One option would be to ship a puppetmaster .box which uses tricks like FUSE and HTTP proxies to make the data available without copying all of it locally.  The corresponding Vagrantfile could take care of setting up SSL certificates and secrets, with an $org of 'vagrant'.  Then devs would first set up a puppetmaster, then install a base host of whatever OS they desired (CentOS, Ubuntu, Darwin..) and puppetize it against the local puppetmaster.

We could, in theory, host a single puppetmaster that is publicly accessible, with well-known secrets.  But given that it's in Ruby, I don't expect CVE-2013-3567 to be the last of its sort, so this option is far too risky from an infrasec perspective.

I'm sure there are other options I just haven't thought of!

All that said, I'll throw it back to you folks to say, basically, "patches accepted" (but please do it right) to this fundamentally open project.  I'd love to have folks outside of releng/relops paying attention to it.

Comment 6

[Gregory Szorc [:gps]]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> Internally, we don't use snapshots at all - we use a Base CentOS 6.2 image,
> and install/configure everything from there with puppet, on each host. 
> We've considered using snapshots for efficiency, but haven't seen the need
> yet.  If you do this with Vagrant, please take the approach that the
> snapshot is just an optimization of the canonical base-image-plus-puppet
> process.

I agree that using puppet + configs from buildbot-configs and buildbotcustom would be ideal. I did try going down this road. However, it is next to impossible for someone "on the outside" to get puppet to run. You would need to set up a puppet master or stub out tons of files not publicly available. This is a massive undertaking!

My initial goal with the effort in this bug is to allow developers to create an environment that is kinda/sorta/mostly like the official build environment. Does it have to be exact (signing keys and all)? No. I care mostly that the compilation toolchain and system libraries are the same. That way, developers can reasonably reproduce the official Linux build environment *without* checking out a slave from Release Engineering.

I concede that the current implementation does a pretty poor job of replicating the base CentOS operating system install (because we don't use puppet). I welcome discussions on how we can rectify this. It likely culminates with RelEng publishing a base image (virtualbox/vagrant/etc) on an ongoing basis or facilitating easier Puppet client running (like you mentioned).

That being said, I'm going to challenge the assertion that base OS parity is important! Since we are using a mock/chroot environment to build Firefox, my thinking is that as long as the mock/chroot environment is good, it doesn't really matter what's going on in the underlying OS/kernel. I would argue you could tar up this mock environment, throw it on *any* 64-bit Linux machine, chroot in, and the results would be good enough. And, this is effectively what I plan to do with a soon-to-be-published Docker image. Now, there are surely exceptions. Are any large enough to abort this effort? I don't the answer to that.

If others feel replicating just the mock environment (and not the entire base OS) is sufficient, then I argue we should focus our efforts on that. IMO it wouldn't be too much work to create an automation job that creates the mock environment, throws it in an archive, and publishes it on a public server. If we had this, we could change the build jobs to just download and use this archive (instead of recreating it from scratch every time). I would think this would result in some nice perf wins. e.g. bug 851294.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

It's not actually a major undertaking.  I suspect that you were looking at the old puppet (http://hg.mozilla.org/build/puppet-manifests) rather than puppetagain.

You're right about mock, and Docker's a good choice for that (although since it only runs on Ubuntu, somewhat limiting).  But that doesn't help for OS X or Windows.

Comment 8

[Chris AtLee [:catlee]]

I have a Vagrantfile that works for Centos6 + mock with puppet. Will try and get that posted somewhere for wider consumption.

Comment 9

[Gregory Szorc [:gps]]

CentOS 6.2 64-bit base Vagrant box: https://people.mozilla.com/~gszorc/centos62-64.box

Archive with mock_mozilla root filesystem: https://people.mozilla.com/~gszorc/mock_mozilla.tar.bz2

I'm able to |docker import| the mock_mozilla archive. I'm assuming a build will complete without error.

Comment 10

[Gregory Szorc [:gps]]

https://github.com/mitchellh/packer also looks interesting. Operates a level up from Vagrant so the images it creates can be used by multiple consumers. If RelEng used package to produce their machine images, downstream consumers could use one of many supported hosting environments, including Vagrant and EC2. Neato.

Comment 11

[Gregory Szorc [:gps]]

I'm not actively working on this.

catlee was looking at bug 896023 earlier today. If our goal is verifiable builds, that bug is more important. If our goal is to more easily enable new developers to set up a Firefox development environment, we can move forward on this bug (at the cost of results aren't deterministic nor verifiable). I think our bootstrap.py script and MozillaBuild adequately fulfill the need of setting up development environments. So, I'm fine with holding off on a Vagrantfile until it can be used to create something more official.

Comment 12

[Chris AtLee [:catlee]]

I suggest we WONTFIX this in favour of the Docker images being used by Taskcluster.

Comment 13

[Gregory Szorc [:gps]]

I agree.