496901 - Deploy ccache on all linux build machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris AtLee [:catlee]]

Investigation from bug 488412 showed that using ccache on Linux is a big win.

We should deploy ccache on all our linux build machines, the try machines.

The version of ccache I used for testing was Debian's ccache, version 2.4-16.  The notable difference from the upstream version is support for compressed object files, which means the cache can store far more objects.

After ccache is installed, the maximum cache size should be set to 2GB:

CCACHE_DIR=/builds/ccache ccache -M 2G

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

Staging slaves are already using ccache - moz2-linux-slave03, 04 and 17 successfully.  

Chris Atlee is going to prepare the rpm for deploying to the other linux slaves and document the process.

Comment 2

[Chris AtLee [:catlee]]

Attachment: Changes to env required

Here's what I was using to test on the linux slaves.

Strictly only the change to the linux env is necessary at this point.  It can safely be landed without impacting production slaves, since the determination of whether ccache is used or is in the mozconfig file.

The additional steps in the build factory zero out the ccache stats, and print them out afterwards.  Pretty handy for debugging!  It might be nice to leave them in somehow, but obviously they should only be run if ccache is used for that build.

Comment 3

[Chris AtLee [:catlee]]

Attachment: Changes to buildbot-configs for ccache

And here's what I was using for buildbot-configs changes.

Again, only the linux side of things is required.  In particular, ccache cannot be enabled for the code coverage build.

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

So John Ford is going to prepare the rpm to use on the linux slaves, and then hopefully roll this out as well.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Now that we have puppet going on all these build slaves, could we use puppet to deploy instead of building an rpm and manually deploying?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> Now that we have puppet going on all these build slaves, could we use puppet to
> deploy instead of building an rpm and manually deploying?

Yes, that's what we've been doing for all new packages.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #6)
> (In reply to comment #5)
> > Now that we have puppet going on all these build slaves, could we use puppet to
> > deploy instead of building an rpm and manually deploying?
> 
> Yes, that's what we've been doing for all new packages.

Cool, thanks for the confirm.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: source rpm

Source RPM for rebuilding

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: binary rpm

binary rpm build with gcc4.1.1

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

rpms are created.  I don't know how to deploy using puppet but would be willing to learn.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: ccache for puppet

This is the puppet manifests change needed to install ccache.  I have this in a separate class to distinguish upstream centos/red hat packages from our own custom packages.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: custom-ccache

Updated patch, cache locations set only for linux

Comment 13

[Chris AtLee [:catlee]]

Comment on attachment 402628 [details] [diff] [review]
custom-ccache

We don't want to run the 'ccache -z' / 'ccache -s' commands on all platforms.

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

that patch is just a new version of the already attached patch.  Would it be ok to wrap that in

if 'linux' in self.platform:
  run-ccache-commands

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 402623 [details] [diff] [review]
ccache for puppet

Please put moz-rpms inside of centos5/ - they are platform specific. (We should fix the other RPMs to be inside of centos5/ too, but that's another bug). Looks fine otherwise.

One small request: Can you drop instructions on how to create the source and binary RPMs into this bug?

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: configs-ccache

Changes to buildbot-configs that are needed to roll this out

Comment 17

[John Ford [:jhford] CET/CEST Berlin Time]

bhearsum, sure i can change that.

To create the binary RPM, you need to run "rpmbuild -ba <the-src.rpm>".  To create the SRPM you would run "rpmbuild -bs ccache.spec" where the spec file is a file inside the SRPM.  To make this specfile, i downloaded the Fedora EPEL package (yumdownloader --source ccache), created a user RPM build tree (rpmdev-setuptree iirc), installed the SRPM (rpm -Uvh <origninal-src.rpm>), made the changes needed to the specfile then built the package (rpmbuild -ba ccache.spec).  I did find that when building the RPM i had to disable a python specific rpm postprocessing script.  I can't remember which one it was but it will give an error when you try to build.  To disable that i added exit 0 after the #! in the script that was erroring out.

Comment 18

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: custom-ccache-v2

I have wrapped the ccache command in a platform check.  All linux slaves should have ccache installed once this is deployed.

Comment 19

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: ccache for puppet 2

Changed the location of the package

Comment 20

[Chris AtLee [:catlee]]

Comment on attachment 402661 [details] [diff] [review]
custom-ccache-v2

Looks basically good, except that this will run ccache on linux64 bit machines as well (since 'linux' in 'linux64' is True).

If it's your intent to deploy ccache to the 64-bit linux machines, great.  Otherwise, change this to "if self.platform == 'linux'"

Comment 21

[John Ford [:jhford] CET/CEST Berlin Time]

Yep, that was intentional.  The buildbot-configs patch does have the configuration changes needed for Linux 64

Comment 22

[[e-mail] (:bhearsum)]

Do we need a different RPM for the 64-bit machines?

Comment 23

[John Ford [:jhford] CET/CEST Berlin Time]

i was under the impression that the 64bit machines were running 32bit operating systems and we are cross compiling Firefox

Comment 24

[Chris AtLee [:catlee]]

you are mistaken!

[cltbld@moz2-linux64-slave01 ~]$ uname -a
Linux moz2-linux64-slave01.build.mozilla.org 2.6.18-53.1.14.el5 #1 SMP Wed Mar 5 11:37:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

[cltbld@moz2-linux64-slave01 ~]$ file /usr/bin/gcc
/usr/bin/gcc: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped

Comment 25

[John Ford [:jhford] CET/CEST Berlin Time]

If that is the case, the reference platform doc is incorrectly instructing us to download the i386 cd images over at https://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0_64-bit 

I can build a 64bit version of the RPM tomorrow.  Do we manage that one vm with puppet? (I'd assume not)

Comment 26

[[e-mail] (:bhearsum)]

They're not managed with Puppet yet, no.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 402665 [details] [diff] [review]
ccache for puppet 2

I think you mean /N/centos5/moz-rpms

Comment 28

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: puppet for ccache 3

Yes, i did mean that.

Comment 29

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: binary rpm for 64-Bit

Binary rpm created on CentOS v5.3 64-bit.

Comment 30

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 402651 [details] [diff] [review]
configs-ccache

These are the buildbot configs required for ccache on linux and linux64.  I have it set up to use ccache on all linux builds that aren't release builds.

Comment 31

[Chris AtLee [:catlee]]

Comment on attachment 402651 [details] [diff] [review]
configs-ccache

We can't use ccache for code coverage builds.

Comment 32

[Chris AtLee [:catlee]]

What's left to do on this bug?

Comment 33

[Chris AtLee [:catlee]]

Comment on attachment 402661 [details] [diff] [review]
custom-ccache-v2

I think if we change this to test if platform == 'linux', we can ignore the linux64 stuff for now.

Comment 34

[Chris AtLee [:catlee]]

Attachment: print out ccache stats in builds

Comment 35

[Chris AtLee [:catlee]]

Attachment: buildbot-configs for ccache

Comment 36

[Chris AtLee [:catlee]]

Attachment: puppet manifests for ccache

Comment 37

[[e-mail] (:bhearsum)]

Comment on attachment 425989 [details] [diff] [review]
print out ccache stats in builds

This'll cause burning for other consumers of this factory that don't have ccache installed. It'd be better to control this with another arg to MercurialBuildFactory IMHO.

Comment 38

[Chris AtLee [:catlee]]

Attachment: puppet manifests for ccache

Comment 39

[[e-mail] (:bhearsum)]

Comment on attachment 426006 [details] [diff] [review]
puppet manifests for ccache

Looks good to me

Comment 40

[[e-mail] (:bhearsum)]

Comment on attachment 425990 [details] [diff] [review]
buildbot-configs for ccache

This part is fine, but as you mentioned on, it will need to be updated the buildbotcustom changes that are coming.

Comment 41

[Chris AtLee [:catlee]]

Attachment: print out ccache stats in builds, v2

added an enable_ccache parameter to MercurialBuildFactory that triggers the ccache steps.

Comment 42

[Chris AtLee [:catlee]]

Attachment: buildbot-configs for ccache, v2

turn on ccache for linux, linux-debug only.

Comment 43

[Chris AtLee [:catlee]]

Comment on attachment 426006 [details] [diff] [review]
puppet manifests for ccache

changeset:   92:b60a779ee769

Comment 44

[Chris AtLee [:catlee]]

Comment on attachment 426006 [details] [diff] [review]
puppet manifests for ccache

So the creation of /builds/ccache failed on some of the new ix slaves, because /builds isn't owned by cltbld.

Should that be fixed, or should I fix the package to ensure that /builds/ccache is owned and writable by cltbld?

Comment 45

[Chris AtLee [:catlee]]

Attachment: Update ccache manifest to work on new /builds

Comment 46

[Chris AtLee [:catlee]]

Comment on attachment 426627 [details] [diff] [review]
Update ccache manifest to work on new /builds

changeset:   95:eda9f5661e31

Comment 47

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 48

[Chris AtLee [:catlee]]

Comment on attachment 426374 [details] [diff] [review]
print out ccache stats in builds, v2

changeset:   609:0b3ad6bd5a2b

Comment 49

[Chris AtLee [:catlee]]

Comment on attachment 426376 [details] [diff] [review]
buildbot-configs for ccache, v2

changeset:   2081:17cd4970c46c

Comment 50

[Chris AtLee [:catlee]]

Still todo: 64-bit linux

Will ignore try slaves for now; should be trivial once bug 520227 is done.

Comment 51

[[e-mail] (:bhearsum)]

(In reply to comment #50)
> Still todo: 64-bit linux
> 
> Will ignore try slaves for now; should be trivial once bug 520227 is done.

The try slaves sync up to puppet already so I'd be surprised if they don't have ccache yet.

Comment 52

[Chris AtLee [:catlee]]

(In reply to comment #51)
> (In reply to comment #50)
> > Still todo: 64-bit linux
> > 
> > Will ignore try slaves for now; should be trivial once bug 520227 is done.
> 
> The try slaves sync up to puppet already so I'd be surprised if they don't have
> ccache yet.

Yeah, I meant more in terms of updating the try mozconfigs, factories, etc.

Comment 53

[Chris AtLee [:catlee]]

Moved the 64-bit linux stuff to its own bug (551786).