744135 - puppet should manage ccache size on r5 osx builders

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

we already do on linux/linux64:
http://hg.mozilla.org/build/puppet-manifests/file/d6e1d385b230/modules/packages/manifests/moz-rpms.pp#l36

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: patch

Not sure where the DMGs for PuppetAgain should reside, guessed puppet:///repos/DMGs/ would be a good place

Comment 2

[Justin Wood (:Callek)]

Comment on attachment 634383 [details] [diff] [review]
patch

feedback- for the mere fact that it doesn't do what this bug is filed for (I can certainly review for 10.8 machines though. -- and I can see how you could have got mixed up)

This bug is about the r5 builders, which are defined in the (old) puppet at http://hg.mozilla.org/build/puppet-manifests and is not on the same puppetmaster as our "PuppetAgain" work.

Also this code for 10.8 of course does not manage the ccache size, and I don't *think* we have the current plan to need ccache on our 10.8 machines.

Comment 3

[Kim Moir [:kmoir] ET]

Yes, I misread the intent of the bug.  The ccache dmg is installed on Lion machines by the modules in the puppet-manifests repo.  However, this isn't needed for 10.8 machines because they are just test machines,  not builders.

Comment 4

[Justin Wood (:Callek)]

Comment on attachment 634383 [details] [diff] [review]
patch

Review of attachment 634383 [details] [diff] [review]:
-----------------------------------------------------------------

Kim, is this patch still relevant? I note again that this bug was relating to the old puppet system/repo. But if it is relevant for PuppetAgain, I'll hand it over to dustin

Comment 5

[Kim Moir [:kmoir] ET]

No I don't think this patch is still relevant.  I'll make a new one.

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: patch

I've tested this on the staging server with a r5 mac slave and it works.  The linux machines currently have their ccache maximum set to 5G.

notice: //Node[build]/base/osx_lion/Exec[increasemaxccachesize]/returns: executed successfully
notice: Finished catalog run in 6.59 seconds
r5-mini-006:~ cltbld$  /usr/local/bin/ccache -s
cache directory                     /builds/ccache
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
files in cache                     19182
cache size                         929.4 Mbytes
max cache size                       5.0 Gbytes
r5-mini-006:~ cltbld$ env | grep CCACHE
CCACHE_DIR=/builds/ccache

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 644343 [details] [diff] [review]
patch

Review of attachment 644343 [details] [diff] [review]:
-----------------------------------------------------------------

::: os/osx_lion.pp
@@ +204,5 @@
> +               command=> "/usr/local/bin/ccache -M 5G",
> +               environment => ["CCACHE_DIR=/builds/ccache", "CCACHE_COMPRESS=1"],
> +               user => "cltbld",
> +               group => "admin",
> +               require => [ Package["ccache-3.1.7.dmg"], File["/builds/ccache"]];

nit: looks like there's some indentation misalignment here?

this should be at least 10G. These machines have 1T of space, so no need to skimp out here.

r+ with fixed indentation and MOAR SPACE!

Comment 8

[Kim Moir [:kmoir] ET]

Updated patch http://hg.mozilla.org/build/puppet-manifests/rev/cdaaf3ac8b02 landed and puppet servers updated.

Comment 9

[Armen [:armenzg]]

Attachment: fix ccache issue

For some reason this was causing even production machines to output an error.
I don't know why would the machines still initialize buildbot if puppet would fail:

You can see it on the log:
bld-lion-r5-041:~ root# /usr/bin/puppetd --onetime --no-daemonize --logdest console   --server scl3-production-puppet.srv.releng.scl3.mozilla.com
notice: Starting catalog run
notice: //Node[build]/base/osx_lion/buildslave::cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[set-time-server]/returns: executed successfully
notice: //Node[build]/ntp/File[/etc/ntp.conf]/checksum: checksum changed '{md5}454a1f30f9c7ec4222bdc2c83abdaec3' to '{md5}c086d264d534ecb48ad1221839c37583'
notice: //Node[build]/ntp/File[/etc/ntp.conf]/content: content changed '{md5}c086d264d534ecb48ad1221839c37583' to '{md5}454a1f30f9c7ec4222bdc2c83abdaec3'
notice: //Node[build]/ntp/Exec[restart-ntp]: Triggering 'refresh' from 2 dependencies
notice: //Node[build]/base/osx_lion/Exec[turn-off-sleep]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[remove-index]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[turn-on-auto-power]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disallow-sleep-button]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disable-indexing]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[verify-xcode]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[restart-on-panic]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disable-updater]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[turn-on-wol]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[clean-saved-state]/returns: executed successfully
err: //Node[build]/base/osx_lion/Exec[setmaxccachesize]/returns: change from notrun to 0 failed: /usr/local/bin/ccache -M 10G returned  instead of 0 at /etc/puppet/manifests/os/osx_lion.pp:210
notice: //Node[build]/base/osx_lion/Exec[disable-bluetooth]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disable-screensaver]/returns: executed successfully
notice: Finished catalog run in 8.06 seconds
bld-lion-r5-041:~ root# /usr/bin/puppetd --onetime --no-daemonize --logdest console   --server scl3-production-puppet.srv.releng.scl3.mozilla.com
notice: Starting catalog run
notice: //Node[build]/base/osx_lion/Exec[set-time-server]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disable-bluetooth]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disable-screensaver]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disable-updater]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[turn-off-sleep]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disable-indexing]/returns: executed successfully
notice: //Node[build]/base/osx_lion/buildslave::cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[restart-on-panic]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[turn-on-wol]/returns: executed successfully
notice: //Node[build]/ntp/File[/etc/ntp.conf]/checksum: checksum changed '{md5}454a1f30f9c7ec4222bdc2c83abdaec3' to '{md5}c086d264d534ecb48ad1221839c37583'
notice: //Node[build]/ntp/File[/etc/ntp.conf]/content: content changed '{md5}c086d264d534ecb48ad1221839c37583' to '{md5}454a1f30f9c7ec4222bdc2c83abdaec3'
notice: //Node[build]/ntp/Exec[restart-ntp]: Triggering 'refresh' from 2 dependencies
notice: //Node[build]/base/osx_lion/Exec[turn-on-auto-power]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[verify-xcode]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[setmaxccachesize]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[disallow-sleep-button]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[remove-index]/returns: executed successfully
notice: //Node[build]/base/osx_lion/Exec[clean-saved-state]/returns: executed successfully
notice: Finished catalog run in 11.53 seconds

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Puppet's exit statuses weren't very good back then.