548950 - deploy ccache on mac builders

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Chris AtLee [:catlee]]

since the slave prioritization patch landed, the xserves are doing the majority of builds, so we should put ccache on them to eke out another few minutes of build time.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

building locally with
wget http://samba.org/ftp/ccache/ccache-3.1.6.tar.bz2
tar jxf ccache-3.1.6.tar.bz2
cd ccache-3.1.6
./configure --prefix=/usr/local
make
make install DESTDIR=root
wget http://hg.mozilla.org/build/puppet-manifests/raw-file/aa75d6e91838/create-dmg.sh
sh create-dmg.sh root/usr/local ccache-3.1.6 ccache3 /usr/

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: dmg for sanity checks

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: manifest changes that should let us deploy ccache

need to run this through staging

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: working patch

moz2-darwin10-slave04:~ root# puppetd -v --test --server staging-puppet.build.mozilla.org
info: Caching catalog at /var/puppet/state/localconfig.yaml
notice: Starting catalog run
notice: //Node[build]/base/osx/Package[ccache-3.1.6.dmg]/ensure: created
notice: //Node[build]/base/osx/buildslave::cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully
notice: //Node[build]/base/osx/Exec[disable-indexing]/returns: executed successfully
notice: //Node[build]/base/osx/Exec[remove-index]/returns: executed successfully
info: //Node[build]/base/osx/buildslave::cleanup/Tidy[/Users/cltbld/Library/Application Support/Firefox/console.log]/ensure: Tidy target does not exist; ignoring
notice: Finished catalog run in 21.19 seconds
moz2-darwin10-slave04:~ root# ccache -V
ccache version 3.1.6

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

from production machine:
moz2-darwin10-slave13:~ cltbld$ ccache -V
ccache version 3.1.6

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

The darwin9 (10.5) slaves are failing to run puppet because they can't mount the dmg (nothing in darwin9-i386 for them). Nor should they because it's a 64bit binary, but the manifest is not version specific.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 555197 [details] [diff] [review]
working patch

Original landing:
http://hg.mozilla.org/build/puppet-manifests/rev/8fbb0b07dd3a

Backout:
http://hg.mozilla.org/build/puppet-manifests/rev/5a4295cae1de

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Possible solutions are:
* compile cacche as for i386 as well as x86_64 (separately, or universal)
* not bother using ccache on 10.5 machines and put the puppet change in the 10.2.0 section further up in os/osx.pp

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Nick Thomas [:nthomas] from comment #8)
> Possible solutions are:
> * compile cacche as for i386 as well as x86_64 (separately, or universal)
> * not bother using ccache on 10.5 machines and put the puppet change in the
> 10.2.0 section further up in os/osx.pp

Lets go with option 2 as the ccache I built isn't actually able to run on 10.5 because I built it against the 10.6 SDK.  The 10.5 pool has great wait times and with bug 674647, 10.5 becomes much less important.

Fwiw, 10.5 is capable of running 64bit binaries.  The problem with the ccache binary is that it is built against the 10.6 SDK instead of the 10.5 SDK.
moz2-darwin9-slave08:~ cltbld$ arch -i386 ./a.out
sizeof(int*): 4
moz2-darwin9-slave08:~ cltbld$ arch -x86_64 ./a.out
sizeof(int*): 8

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: patch that only deploys to 10.6 machines

The patch as submitted uses the same dmg file (ccache-3.1.6.dmg) as the last patch.  The log below has 3.1.6-2 to trick puppet into reinstalling an already installed package.

moz2-darwin10-slave04:~ cltbld$ sudo puppetd -v --test --server staging-puppet.build.mozilla.org
info: Caching catalog at /var/puppet/state/localconfig.yaml
notice: Starting catalog run
info: //Node[build]/base/osx/buildslave::cleanup/Tidy[/Users/cltbld/Library/Application Support/Firefox/console.log]/ensure: Tidy target does not exist; ignoring
notice: //Node[build]/base/osx/Exec[remove-index]/returns: executed successfully
notice: //Node[build]/base/osx/Exec[disable-indexing]/returns: executed successfully
notice: //Node[build]/base/osx/Package[ccache-3.1.6-2.dmg]/ensure: created
notice: //Node[build]/base/osx/buildslave::cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully
notice: Finished catalog run in 32.60 seconds

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 555463 [details] [diff] [review]
patch that only deploys to 10.6 machines

Looks fine from a deployment point of view. 

What are you planning to do about turning ccache on ? Something like
* creating /builds/ccache, and initializing the cache (eg like modules/packages/manifests/moz-rpms.pp does on linux)
* setting enable_ccache for macosx64 platform, and setting the env vars in mozilla/config.py
* updating the mozconfigs

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

Yep, that looks right.  I am going to deal with enabling ccache as part of the rev4/5 builder bug.  If I recall correctly, catlee's benchmarks showed that rev2 hardware wasn't made much faster by using ccache.  rev4/5 have much better hardware and should benefit from ccache being enabled.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

This is going to be fixed by bug 720470.  Marking this bug as resolved.