488412 - Investigate using ccache for linux / mac builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris AtLee [:catlee]]

ccache has the potential to really speed up our linux and mac builds.  We should try and use it if we can.

Ideally we'd use a shared cache between all the slaves of the same platform.

Comment 1

[Chris AtLee [:catlee]]

Currently using the shared drive from bug 472185 on staging.

Comment 2

[Chris AtLee [:catlee]]

First depend tracemonkey build to complete after cache was populated completed in 30 minutes.

Comment 3

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

There's a msvc ccache patch at http://idea.opensuse.org/content/ideas/cross-compile-openoffice.org-for-win32-on-linux -- not sure if it's gotten into upstream or not.

Comment 4

[Boris Zbarsky [:bzbarsky]]

Ideally we should make sure that all our srcdirs are in the same absolute paths on all our machines (because gcc bakes the source file location into the object file as an absolute path, so that different srcdirs miss each other in the ccache).

Comment 5

[Chris AtLee [:catlee]]

All our build machines have identical source directories (/builds/slave/... on linux)

This just means that objects won't be sharable across different build types; e.g. a mozilla-1.9.1 build won't benefit from any work done by previous mozilla-central builds.

ccache includes the current working directory in the hash, so we won't be pulling in objects that have incorrect source directories baked in.

Comment 6

[Chris AtLee [:catlee]]

(In reply to comment #5)
> ccache includes the current working directory in the hash, so we won't be
> pulling in objects that have incorrect source directories baked in.

Turns out this is an optional feature.  What are the consequences of having different source directories present in the debugging information?

Comment 7

[Boris Zbarsky [:bzbarsky]]

> This just means that objects won't be sharable across different build types;

Yeah, that's fine.  We change enough interfaces that this wouldn't be much of a win anyway, if any.

Comment 8

[Chris AtLee [:catlee]]

Initial tests with ccache on an NFS drive aren't promising.

Builds were taking around the same time to complete (some were slower, some were faster), so I ran a test of a simple C++ file compiled 20 times.

The shared NFS drive is mounted using noatime,rsize=32768,wsize=32768,noacl

g++ only: 6.5 seconds
ccache with NFS cache: 5.5 seconds
ccache with local cache: 1.7 seconds

I'm going to try out local caches to see what effect it has on build times.

Comment 9

[Chris AtLee [:catlee]]

Using a local cache directory has yielded better results.

Average build time for mozilla-1.9.1-linux (clobber build) w/o ccache: 3151 seconds
Average build time w/ ccache: 1634 seconds
The ccache builds were of _exactly_ the same code, so this probably is a best case scenario.

Average build time for mozilla-1.9.1-linux (depend build) w/o ccache: 3108 seconds.
Average build time w/ ccache: 2671 seconds

Some of the depend builds were starting from an empty build directory, and others were not.  They were run on a variety of changesets.

It looks like this has an overall performance benefit.

It's not clear why the full clobber builds are significantly faster than the depend builds.  This needs further investigation.

Comment 10

[(not currently active) Ted Mielczarek]

(In reply to comment #9)
> It's not clear why the full clobber builds are significantly faster than the
> depend builds.  This needs further investigation.

That's definitely interesting. Just clobber all the time? :) (But seriously, ccache probably removes most of the need for depend builds, and clobbering every time would actually be pretty nice.)

Comment 11

[Chris AtLee [:catlee]]

So my first attempt at using ccache on mac wasn't too exciting:

Here are the times for just the compile step of a nightly build:

Nightly build w/o ccache: 1 hrs, 30 mins, 23 secs
Nightly build w/ ccache (1st run): 1 hrs, 42 mins, 22 secs
Nightly build w/ ccache (2nd run): 1 hrs, 32 mins, 21 secs

ccache _is_ being called.  On the 2nd run I got 5196 cache hits, and 80 misses.

Comment 12

[Chris AtLee [:catlee]]

This is definitely a win on Linux when using a local cache.  I just completed more testing and a full rebuild with ccache takes just over 20 minutes if everything has been cached already.  If the cache is empty, then a full rebuild takes 52 minutes, which is just a few minutes longer than it would have taken if ccache wasn't used.

I recommend we deploy this to our linux build and linux try slaves.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: ccache-win32

Using the patch at http://idea.opensuse.org/content/ideas/cross-compile-openoffice.org-for-win32-on-linux I have built a version of CCache for windows.  I had to check out the CVS trunk as the patch would not apply cleanly to the stable version.  I have not tested this beyond running it in a plain cmd.exe instance and it runs.  Included is the exe and required cygwin1.dll.

Comment 14

[Chris AtLee [:catlee]]

Back in the pool for now.

The only thing left to do is look at if ccache on an xserve gives any benefit...

Comment 15

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 16

[(not currently active) Ted Mielczarek]

catlee: you were looking at this recently, weren't you?

Comment 17

[Chris AtLee [:catlee]]

sure, I can grab this again :)  we've got another xserve being recycled into the build pool, it's good timing.

Comment 18

[Chris AtLee [:catlee]]

times for an intel-only build on bm-xserve06:

non-ccache build: 32m51.159
first ccache build: 35m24.024
second ccache build: 26m5.296

Not as much of a gain as I had hoped, but still good.