588150 - Upgrade Linux buildbots to ccache 3.0

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement, P5)

Description

[Justin Lebar (not reading bugmail)]

ccache 3.0 is a pretty big improvement over previous versions.  Direct mode (plus CCACHE_BASEDIR, if necessary), has greatly sped up my builds.

Comment 1

[Syed Albiz [:salbiz]]

Ran a couple of experiments to establish the kinds of performance gains we
might see from this upgrade. 
Steps:
1)hg clone http://hg.mozilla.org/{mozilla-central|tracemonkey}
2)Build the ipc/chromium directory 20 times and measured the average time taken
to build with the current ccache (or in the case of macos, plain gcc), and then
built the same directory with ccache v3.0.1 taken from the ccache website
tarball. 

2a)Set env variable CCACHE_BASEDIR=(parent_of_build_objdir), prime ccache by
building mozilla-central, then compile tracemonkey

2b)Before all build runs, purge the disk cache, so the build runs off a cold
disk cache.

The results from these runs were:
On Darwin:
plain gcc: 95.15s, (Hot disk cache)ccachev3: 5.24, (Cold cache)ccachev3: 19.33s
tracemonkey(ccache primed on mozilla-central) 5.73s

On Linux:
ccache2.4: 14.15s, (Hot disk cache)ccachev3: 3.57, (Cold cache)ccachev3: 8.26s
tracemonkey(ccache primed on mozilla-central) 7.34s

Based on these rough results, we will probably see some nice gains all around
by upgrading to ccache 3

Comment 2

[Syed Albiz [:salbiz]]

Attachment: [untested]patch_buildbot_configs to add ccache to mac, set CCACHE_COMPRESS

To make this change, the following things need to happen:

-Patch buildbotconfigs (see attached patch) to enable ccache on macs and set a couple of environment variables. In particular, since cache compression is no longer enabled by default, CCACHE_COMPRESS needs to be set.

-Patch puppet-manifests to deploy the new rpm/dmg ccache packages to buildslaves (already tested, patch forthcoming)

-Patch buildbotcustom to set environment variable CCACHE_BASEDIR to the base directory of the build so that ccache strips out absoulte paths containing platform/branch specific directory names from its hash.

Comment 3

[Syed Albiz [:salbiz]]

Attachment: [untested]patch_buildbotcustom_set_CCACHE_BASEDIR

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 474047 [details] [diff] [review]
[untested]patch_buildbot_configs to add ccache to mac, set CCACHE_COMPRESS

Looks good; I wouldn't worry about updating mozilla2, mozilla2-staging.

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 474049 [details] [diff] [review]
[untested]patch_buildbotcustom_set_CCACHE_BASEDIR

So you'll need to do something like

env = self.env.copy()
env['CCACHE_BASEDIR'] = ...
self.addStep(..., env=env)

Comment 6

[Syed Albiz [:salbiz]]

Attachment: [tested]patch_puppet_manifests

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 474051 [details] [diff] [review]
[tested]patch_puppet_manifests

I think we need something on mac to initialize the ccache size to a non-default value.

Comment 8

[Syed Albiz [:salbiz]]

Attachment: [untested]patch_buildbotcustom_set_CCACHE_BASEDIR_r2

Comment 9

[Syed Albiz [:salbiz]]

Attachment: [untested]patch_buildbot-configs

Based on our discussion, moved logic to set CCACHE_BASEDIR into config.py, and backed out changes in mozilla2 & mozilla2-staging

Comment 10

[Chris AtLee [:catlee]]

Comment on attachment 474074 [details] [diff] [review]
[untested]patch_buildbot-configs

You'll need to add 'from buildbot.process.properties import WithProperties' at the top there, but other than that looks good.

Comment 11

[Syed Albiz [:salbiz]]

Attachment: [untested]patch_puppet_manifests

added an 'exec' stanza to os/osx.pp to ensure that macs initialize the cache to 2G (same as linux).

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Please clean up any /home/ccache directories you've created for testing, if that's all done. moz2-linux-slave03 is down to 180MB free on /.

Comment 13

[Syed Albiz [:salbiz]]

Done, df -h now shows 1.2G free on /.
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             8.3G  6.7G  1.2G  85% /

Comment 14

[Syed Albiz [:salbiz]]

Attachment: [untested]patch_buildbot-configs

Comment 15

[Syed Albiz [:salbiz]]

Comment on attachment 474083 [details] [diff] [review]
[untested]patch_puppet_manifests

Noticed a couple of things about how manifests are structured, that might be worth changing the patch to suit. 

Right now I've lumped ccache into the devtools class, where it probably shouldn't be, since it doesn't live in /tools. One place for it might be os/osx.pp, or I was thinking it might be better to follow the same pattern as linux and create a class like moz-dmgs.pp and put the package and any initializing there, keeping things a bit more consistent.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 474083 [details] [diff] [review]
[untested]patch_puppet_manifests

This looks mostly fine. You'll need to use ensure => latest in moz-rpms.pp, though.

Comment 17

[Syed Albiz [:salbiz]]

Attachment: [untested]buildbot-configs_changes

moving CCACHE_BASEDIR declaration to buildbot-configs

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 474083 [details] [diff] [review]
[untested]patch_puppet_manifests

Let's get the ccache package definition and init_ccache stuff close together for osx.

Comment 19

[Syed Albiz [:salbiz]]

Attachment: [tested]puppet_manifests_patch

After a *lot* of testing, I'm going to have to give up on making ccache fast for macs. NFS mounts even on 3.1 (with several fixes to improve NFS perf) are still unbearably slow, and the new direct cache hits from branch to branch are too few to barely make a dent in the overhead caused by hitting the disk for cache misses. I've uploaded the tested version of the patch that adds the new version to linux boxes, since the handful of direct hits does make a bit of difference there.

The only way I can see of improving the direct (fast) ccache hit rate would be to optimize the Makefiles and compilation process to use as few absolute paths/macros as possible, avoid switching out command-line arguments, etc, as these things all cause ccache to fall back to preprocessor (slow) mode.

Comment 20

[Syed Albiz [:salbiz]]

Attachment: [tested]patch_buildbot-configs

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 481682 [details] [diff] [review]
[tested]puppet_manifests_patch

lgtm

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 481683 [details] [diff] [review]
[tested]patch_buildbot-configs

Hmm, why isn't ccache used in non-debug mac builds?

Is ccache already installed on the Macs?

Don't use ccache in release builds, because we're paranoid like that.

Comment 23

[Justin Lebar (not reading bugmail)]

(In reply to comment #22)
> Don't use ccache in release builds, because we're paranoid like that.

While I think it's totally sensible not to use ccache for nightly builds, for regular m-c or try release builds, ccache should be a huge win, no?

Comment 24

[Justin Lebar (not reading bugmail)]

Oh, scratch that comment.  Release means release means release.  I see.

Comment 25

[Syed Albiz [:salbiz]]

Attachment: set BASEDIR in PLATFORM_VARS [tested]

yeah, a mistake on my part, since the ccache-on-macs experiment didn't go so well, I should've pruned the mozconfig changes from that patch. This one just keeps the env changes that adds the CCACHE_BASEDIR variable to the environments.

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 481947 [details] [diff] [review]
set BASEDIR in PLATFORM_VARS [tested]

Why are you dropping the enable_ccache around line 460?

Comment 27

[Syed Albiz [:salbiz]]

Attachment: setting BASEDIR for ccache

oops, yeah. I must have thought we weren't doing ccache builds for debug for some reason. Removed offending line from patch.

Comment 28

[Justin Lebar (not reading bugmail)]

I was reading through the man page for the tip of ccache trunk, and I came across this (not sure if it's in the 3.0 or 3.1 man page):

SHARING A CACHE ON NFS
       It is possible to put the cache directory on an NFS filesystem (or similar filesystems), but keep in mind that:

       ·   Having the cache on NFS may slow down compilation. Make sure to do some benchmarking to see if it’s worth it.

       ·   ccache hasn’t been tested very thoroughly on NFS.

       A tip is to set CCACHE_TEMPDIR to a directory on the local host to avoid NFS traffic for temporary files.

Have we tried CCACHE_TEMPDIR?  Also, do we even know that sharing caches over NFS is the right thing to do?

Comment 29

[[e-mail] (:bhearsum)]

When we used NFS mounts to distribute files for Puppet, they were pretty problematic, primarily because it's tricky to make it work across multiple colos. I think Syed did some testing on them, but I'll let him speak to that.

Comment 30

[Chris AtLee [:catlee]]

The goal with using a shared cache over NFS is so that slave B can make use of slave A's compilation results.

But yeah, doesn't look like NFS handles this particularly well.  I wonder if rsyncing ccache directories around is safe...

Comment 31

[Syed Albiz [:salbiz]]

(In reply to comment #28)
> Have we tried CCACHE_TEMPDIR?  Also, do we even know that sharing caches over
> NFS is the right thing to do?

I did do some tests with a TEMPDIR, and it doesn't really seem to provide much benefit. We still have to hit the NFS mount for every cache miss, which is where that bottleneck really hurts.

Comment 32

[Syed Albiz [:salbiz]]

No further developments, so I think this is ready to land, setting CCACHE_BASEDIR for all builds using ccache 3.x, which should net us a handful of direct ccache hits and increase throughput.

Comment 33

[Chris AtLee [:catlee]]

Comment on attachment 482640 [details] [diff] [review]
setting BASEDIR for ccache

changeset:   3314:45560f8b0c15

Comment 34

[Rail Aliiev [:rail]]

This patch should break release l10n repacks. 
It uses readBranchConfig from http://hg.mozilla.org/build/tools/file/aac57c95e2ed/lib/python/release/info.py#l62, which will fail executing config.py:

from buildbot.process.properties import WithProperties

At least I've got a failure testing another patch using the same approach.

Comment 35

[[e-mail] (:bhearsum)]

Comment on attachment 481682 [details] [diff] [review]
[tested]puppet_manifests_patch

changeset:   244:0d7eb12a4d2b

Comment 36

[[e-mail] (:bhearsum)]

Slaves are picking this up OK and seem to be building fine too.