720470 - puppet should be able to configure a 10.7 build slave

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

We should be able to use puppet to configure a Rev5 mac mini as a build slave for our build farm.  It needs to run 10.7.  For now, lets go with 10.7.2 as that is what is current now.  As per bug 720027, lets install Xcode 4.1 for now.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: some brief ideas.

Some initial ideas, using existing 10.7 manifests to configure the bulk of the build slave image.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

I don't think that the Xcode pkg file can be installed by /usr/sbin/installer, which is what the pkgdmg provider uses.  My searches show that this is a common issue with 10.7 and newer apple pkg files.  I am not getting an error message, just that the package install is taking a lot longer than it does normally and hasn't yet completed.  I don't know if this is down to being run on a VM or a bug in the installer.

I am going to let it run overnight to see if it completes by morning.  Chances are that we'll have to bake a version of Xcode into the base image we use.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

interestingly, since comment 2 was posted, the machine is still not finished the installation of xcode.  In top the installd process is taking up ~95% of the cpu with a system load average exceeding 4.

While the DMG is useful for reproducibility, it doesn't look like its going to work for deployment.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

built some of the dependencies with

curl -LO http://ftp.gnu.org/gnu/autoconf/autoconf-2.13.tar.gz
tar zxf autoconf-2.13.tar.gz
cd autoconf-2.13
./configure --prefix=/usr/local  --program-suffix=213
# make sure that jhford's destdir patch is imported
make
make install DESTDIR=installroot
sh create-dmg.sh installroot/usr/local/ autoconf-2.13 autoconf213 /usr/
cd ..

curl -LO http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
tar zxf yasm-1.1.0.tar.gz
cd yasm-1.1.0
./configure --prefix=/usr/local
make
make install DESTDIR=installroot
sh create-dmg.sh installroot/usr/local/ yasm-1.1.0 yasm110 /usr/

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Another interesting thing about the xcode dmg is that it is a wrapper for what's in the App Store, i.e. the dmg+pkg intalls "/applications/install xcode", which still has to be run.

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

built hg

curl -LO http://mercurial.selenic.com/release/mercurial-2.0.2.tar.gz
tar zxf mercurial-2.0.2.tar.gz
cd mercurial-2.0.2
python setup.py install --root installroot
sh create-dmg.sh installroot/. mercurial-2.0.2 mercurial202 /

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

built 7z, hereafter known by p7zip.

curl -LO "http://downloads.sourceforge.net/project/p7zip/p7zip/9.20.1/p7zip_9.20.1_src_all.tar.bz2"
tar jxf p7zip_9.20.1_src_all.tar.bz2
cd p7zip_9.20.1
cp makefile.macosx_64bits makefile.machine
make all_test
sed -e "s/^DEST_DIR=$/DEST_DIR=installroot/" < install.sh | sh
# Not sure why i needed to run as root, but got tons of permissions errors otherwise
sudo sh create-dmg.sh installroot/usr/local/ p7zip-9.20.1 7z9201 /usr/

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

the disable-screensaver script seems to need a change to work on the new hardware.  My guess is that this is related to it being server hardware instead of normal hardware.

This is the change needed for this hardware

< uuid=$(ioreg -rd1 -c IOPlatformExpertDevice | grep -E '(UUID)' | sed -e "s/.*\"\(.*\)\"$/\1/")
---
> uuid=$(ioreg -rd1 -c IOPlatformExpertDevice | grep -E '(UUID)' | sed -e "s/.*\".*-\(.*\)\"$/\1/")

Comment 9

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

jhford asked me about libIDL, libIDL is no longer a build dependency as of mozilla9.

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

I found that the python-2.6.4 dmg that we use for 10.6 did not work on 10.6.  I also found that building 2.6.4 didn't work at all.  Instead of upgrading to the latest 2.6 series of python, I've built Python 2.7.2.  The python that we have running on the 10.6 build slaves is a framework based python.  I am not entirely sure why we do that, but I've replicated that for 10.7.

This is how I built python 2.7.2:

curl -LO http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2
cd Python-2.7.2
./configure --enable-framework=/tools/python-2.7.2 --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk/ --prefix=/tools/python-2.7.2 --with-universal-archs=intel
make -j8
make install DESTDIR=$PWD/installroot
sh create-dmg.sh installroot/. python-2.7.2 python27 /

With this, there are a couple things left to figure out.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: a more complete work in progress patch

This able to sync and start buildbot.  Some things unresolved:

1) nagios.  I haven't really looked into nagios much yet
2) in os/osx.pp, there are two setuid-as-root programs "chown_root" and "chown_revert".  What are they for?  Do we still need them?  Are they really what they look like -- a setuid-as-root copy of chown?
3) repackaging-tools.pp refers to "repackaging-tools-10.6.dmg" and "repackaging-tools-10.5.dmg".  What are they?  Do we still need this?  It looked like it is related to BYOB repacking, not L10N repacking.  The bug that introduced this resource was bug 569675.

Those issues aside, I hope there isn't anything that people find particularly egregious in this.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John Ford [:jhford] from comment #11)
> Created attachment 592370 [details] [diff] [review]
> a more complete work in progress patch
> 
> This able to sync and start buildbot.  Some things unresolved:
> 
> 1) nagios.  I haven't really looked into nagios much yet

I think I have this working, I'll need someone from Relops to help me verify that it is working.

> 2) in os/osx.pp, there are two setuid-as-root programs "chown_root" and
> "chown_revert".  What are they for?  Do we still need them?  Are they really
> what they look like -- a setuid-as-root copy of chown?

According to https://developer.mozilla.org/en/Build_and_Install#Mac_OS_X, these are used for packaging universal XR sdks.  Do we still need this though?

> 3) repackaging-tools.pp refers to "repackaging-tools-10.6.dmg" and
> "repackaging-tools-10.5.dmg".  What are they?  Do we still need this?  It
> looked like it is related to BYOB repacking, not L10N repacking.  The bug
> that introduced this resource was bug 569675.
> 
> Those issues aside, I hope there isn't anything that people find
> particularly egregious in this.

I have a copy of this working.  I think my Mercurial-2.0.2 dmg is not useful because it uses system python, not the /tools python.  The pythonpath env-var is set to the /tools interpreter.  I'll rebuild mercurial

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

building ccache with:

curl -LO http://samba.org/ftp/ccache/ccache-3.1.7.tar.bz2
tar jxf ccache-3.1.7.tar.bz2
cd ccache-3.1.7
./configure --prefix=/usr/local
make
make install DESTDIR=installroot
sh create-dmg.sh installroot/usr/local/ yasm-1.1.0 yasm110 /usr/

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John Ford [:jhford] from comment #13)
> sh create-dmg.sh installroot/usr/local/ yasm-1.1.0 yasm110 /usr/

sh create-dmg.sh installroot/usr/local/ ccache-3.1.7 ccache317 /usr/

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

Built buildbot master so we can do sendchanges from the slaves.

curl -LO http://buildbot.googlecode.com/files/buildbot-0.8.5.tar.gz
tar zxf buildbot-0.8.5.tar.gz
cd buildbot-0.8.5
python setup.py install --root installroot
sh create-dmg.sh installroot/. buildbot-master-0.8.5 buildbotmaster085 /

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

I am seeing the following failures in sample dep builds on the test slave:

1) clobberer fails -- this is because the host is showing up as being outside of the build-network.  When I go to the clobberer page I get prompted for ldap credentials.  This is expected behaviour for clobberer

2) graph server fails to post -- this is because the builder faux-names aren't in the staging db yet.  This should be fixed once the sql from bug 722494 is run on the staging graph server

3) buildbot sendchanges fail -- this is because I am sendchanging to a master that isn't running

4) leak test builds fail alive test -- there is a failure in subprocess.Popen about a file not found.  This usually a binary not being found.

5) shark builds fail to package -- this is because CHUD isn't installed on these slaves.  my understanding is that CHUD isn't lion compatible.

6) xulrunner doesn't compile, complaining about the SDK it is using

7) nightly run #0 skipped a bunch of steps -- queuing up a new build to see if it works the second time.

Comment 17

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John Ford [:jhford] from comment #16)
> 2) graph server fails to post -- this is because the builder faux-names
> aren't in the staging db yet.  This should be fixed once the sql from bug
> 722494 is run on the staging graph server

This is working now that the sql has been run on the staging graphserver

> 3) buildbot sendchanges fail -- this is because I am sendchanging to a
> master that isn't running

I pointed my sendchange target to a real test master and it is working now.

> 4) leak test builds fail alive test -- there is a failure in
> subprocess.Popen about a file not found.  This usually a binary not being
> found.

Two issues I've found so far.  One is that the build is creating an app bundle with the name of "FirefoxFirefoxDebug.app" and the other is that there is a failing assertion on 'compMgr'.  See the logs at the bottom of this comment [1].

> 7) nightly run #0 skipped a bunch of steps -- queuing up a new build to see
> if it works the second time.

This is now working.  I am going to check at some point today to see whether nightly updates work.  I also need to verify nightly L10N.


[1]
args: ['/builds/slave/m-beta-osxlion-dbg/build/obj-firefox/dist/FirefoxDebug.app/Contents/MacOS/firefox-bin', '-foreground', '-no-remote', '-profile', '/builds/slave/m-beta-osxlion-dbg/build/obj-firefox/_leaktest/leakprofile/', 'http://localhost:8888/bloatcycle.html', '-register']
INFO | automation.py | Application pid: 70626
WARNING: NS_ENSURE_TRUE(compMgr) failed: file /builds/slave/m-beta-osxlion-dbg/build/obj-firefox/xpcom/build/nsComponentManagerUtils.cpp, line 90
nsStringStats
 => mAllocCount:           2197
 => mReallocCount:          387
 => mFreeCount:            2178  --  LEAKED 19 !!!
 => mShareCount:           7477
 => mAdoptCount:             89
 => mAdoptFreeCount:         89
INFO | automation.py | Application ran for: 0:00:00.144318
INFO | automation.py | Reading PID log: /var/folders/cg/ckl2jpw51bq2x_pm_zy5lf4r0000gs/T/tmpHyrQJKpidlog

Comment 18

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John Ford [:jhford] from comment #16)
> 4) leak test builds fail alive test -- there is a failure in
> subprocess.Popen about a file not found.  This usually a binary not being
> found.

These started working, not sure what was wrong or what fixed it.

> 6) xulrunner doesn't compile, complaining about the SDK it is using

Stopped complaining about SDK, started complaining about missing chown_root and chown_revert.  Deployed chown_root and chown_revert to expected locations.


I have a test nightly up at http://johnford.info/files/2012-02-02-03-02-01-mozilla-central/ and will be putting the latest debug build in a similar location.

Comment 19

[John Ford [:jhford] CET/CEST Berlin Time]

Still having issues with 10.7 xulrunner permissions.  I am rebuilding chown_revert and chown_root on 10.7.

clang -arch i386 -arch x86_64 chown_revert.c -o chown_revert
clang -arch i386 -arch x86_64 chown_root.c -o chown_root

Comment 20

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: puppet manifest changes

These should be all the puppet changes needed to bring a 10.7 slave up from nothing.  I have some notes on what I did

General:
-instead of using operating system release, which maps to kernel version, I switched all switches that I use to be general to major release versions.  This is a facter variable available to all of our slaves.
-package building instructions should be in this bug
-using 1280x1024x32 resolution for now so edid boxes aren't required

base/nodes.pp, staging.pp, staging-fileserver.conf - self explanatory

classes/base.pp
-change from including osx.pp to including osx_lion.pp for 10.7 slaves only

modules/builsslave:
-added a more verbose error message to runslave.py
-added comment on corner case to cleanup.pp
-changed install.pp to use slaveType puppet variable to be explicit about whether a slave is build or test instead of relying on magic from kernel version

modules/nagios:
-use pkgdmg provider instead of install_rpm for 10.7

modules/packages:
-set 10.7 wget ssl support files to non-macports location
-reworked devtools.pp to do things appropriately for 10.7
-installed individual dependencies of the repackaging tools instead of opaque dmg for 10.7

os/osx_lion.pp
-new file with description of a 10.7 build slave
-based heavily on talos_osx_rev4 and osx classes

Comment 21

[Chris Cooper [:coop] (he/him)]

Comment on attachment 599345 [details] [diff] [review]
puppet manifest changes

Review of attachment 599345 [details] [diff] [review]:
-----------------------------------------------------------------

If you have explanations for the nits, then r+.

::: modules/packages/manifests/repackaging-tools.pp
@@ +32,3 @@
>                      }
>                      file {
>                          "/usr/local/bin/7z":

No 7za install+check required for 10.7?

::: os/talos_osx_rev4.pp
@@ +1,1 @@
> +# osx_lion.pp

Are you using talos_osx_rev4.pp as the basis for this new file (osx_lion.pp), or is talos_osx_rev4.pp really going away? It's unclear to me from the diff, but maybe it's just too late here.

Comment 22

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris Cooper [:coop] from comment #21)
> Comment on attachment 599345 [details] [diff] [review]
> puppet manifest changes
> 
> Review of attachment 599345 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> If you have explanations for the nits, then r+.
> 
> ::: modules/packages/manifests/repackaging-tools.pp
> @@ +32,3 @@
> >                      }
> >                      file {
> >                          "/usr/local/bin/7z":
> 
> No 7za install+check required for 10.7?

7za is built and installed from the p7zip package in osx_lion.pp.  It's a generally useful tool, and likely the easiest way to support lzma compression for tests/symbols so it seems like it'd be generally useful to all mac build slaves.

> ::: os/talos_osx_rev4.pp
> @@ +1,1 @@
> > +# osx_lion.pp
> 
> Are you using talos_osx_rev4.pp as the basis for this new file
> (osx_lion.pp), or is talos_osx_rev4.pp really going away? It's unclear to me
> from the diff, but maybe it's just too late here.

Bugzilla is messing up my patches.  I just downloaded this patch then diffed it to my local copy of the patch and got a 100+ line diff between the BZ version and my local copy of the file I uploaded.  I'll play around with this to figure out if I can reproduce reliably.

Anyway, the diff header for that file is the broken bit, the contents should still be the valid differences between talos_osx_rev4.pp and osx_lion.pp.  The header I have locally is:

diff --git a/os/talos_osx_rev4.pp b/os/osx_lion.pp
copy from os/talos_osx_rev4.pp
copy to os/osx_lion.pp
--- a/os/talos_osx_rev4.pp
+++ b/os/osx_lion.pp

Comment 23

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 599345 [details] [diff] [review]
puppet manifest changes

updated patch landed

http://hg.mozilla.org/build/puppet-manifests/rev/2610cd62615f

Comment 24

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: list of files used

this is a list of the files that I tested with in /N/staging/darwin11-x86_64/build/

Comment 25

[John Ford [:jhford] CET/CEST Berlin Time]

this has been done for a while and has been tested using the bld-lion-001-080 deployment as a final test.  future puppet work for darwin11 slaves will be done as follow up bugs.

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to John Ford [:jhford] -- If you expect a reply from me, use needsinfo? instead of CC from comment #4)
> # make sure that jhford's destdir patch is imported

Can you point to a copy of this patch?