940788 - (sccache) implement AWS S3-backed ccache for efficient cluster compilation of Firefox

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(dormant account)]

At the moment our build machines do not share state. In order to minimize compilation time one needs to write a fancy scheduler that schedules builds on machines with trees that are the closest ancestor. This is silly.

S3 is a fast key/blob store. We should key the store by hashes of dependencies of objects being built. This should work for object files, libraries and other build system artifacts. ccache direct mode appears to implement this for .o files, should be a good starting point to look into this.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Initial shared cache implementation

I'm looking for a rubberstamp here. Maybe you can find someone more appropriate.

A few notes:
- This comes in the form of a patch because i don't think splinter handles non patches, but this is not, in fact, something to apply to some existing tree. This will come in a tarball along with the required components (dnspython, which, boto) distributed with tooltool.
- There are plenty of flags it doesn't bail on, because my intent hasn't been to create a generic tool that works everywhere. I was more focused on getting something that works for our needs, and this does. We can (and will) adjust later.
- The Timer class is kind of awful, but meh. That's something that's meant to go away, but I want to keep it in the initial deployment so that we have stats in case something goes wrong.

Comment 2

[Mike Hommey [:glandium]]

Attachment: tarball to deploy on tooltool

sha512: af25ecf03b65795d21f011939984b130db167a4efc4f306700f373854f9d7ae664662cb7812c3d737eace7f3735324daa6eb540b5e42f90189b0d9a8dd5f4c9f

I don't know how tooltool works wrt setup scripts. Does it allow to use several setup scripts, or would i need to merge a setup script with the one for gcc?

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8373848 [details]
tarball to deploy on tooltool

uploaded:
size: 150816
sha512: af25ecf03b65795d21f011939984b130db167a4efc4f306700f373854f9d7ae664662cb7812c3d737eace7f3735324daa6eb540b5e42f90189b0d9a8dd5f4c9f


(In reply to Mike Hommey [:glandium] from comment #2)
> I don't know how tooltool works wrt setup scripts. Does it allow to use
> several setup scripts, or would i need to merge a setup script with the one
> for gcc?

It can only run one. You'll need to merge it with the gcc script and I can upload the new one.

Comment 4

[Mike Hommey [:glandium]]

Attachment: setup script to deploy on tooltool

So here's a setup script for gcc+sccache.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8374488 [details]
setup script to deploy on tooltool

size: 81
sha512: 59002eae04fa3534df3bbb0026dff50d3872313551514f9ccdf33080c8b8e34a6295b3b6f2c078b3ddef099023897a42adb65d34c9364f84dac5b8e7d022bf39

Comment 6

[Mike Hommey [:glandium]]

Attachment: Use sccache on try for builds happening on AWS that currently use ccache

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 8374653 [details] [diff] [review]
Use sccache on try for builds happening on AWS that currently use ccache

Review of attachment 8374653 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/mozconfig.cache
@@ +4,5 @@
> +
> +# Setup for build cache
> +
> +bucket=
> +if test -f "$topsrcdir/sccache/sccache.py"; then

Since both of these files are in-repo, is this check necessary? If sccache didn't exist, I would assume that code such as this wouldn't be present either.

Looks fine otherwise.

Comment 8

[Mike Hommey [:glandium]]

(In reply to Ben Hearsum [:bhearsum] from comment #7)
> Since both of these files are in-repo, is this check necessary? If sccache
> didn't exist, I would assume that code such as this wouldn't be present
> either.

sccache/sccache.py is not in-repo, it's in tooltool. And it's not in the tooltool manifest for everything that's built on try spot/ec2 instances.

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 8373845 [details] [diff] [review]
Initial shared cache implementation

Review of attachment 8373845 [details] [diff] [review]:
-----------------------------------------------------------------

Looks sane.

I'd encourage you to add far more comments / docs for this code if we ever want to maintain it as a real tool.

::: sccache/sccache.py
@@ +189,5 @@
> +        return self._status
> +
> +    @property
> +    def name(self):
> +        return self._name

small style nit: don't need to use properties for simple getters and setters. Use self.name, self.status as regular attributes. If you need extra behaviour controlling their access, then move them behind properties.

@@ +208,5 @@
> +                from cStringIO import StringIO
> +                import shutil
> +                with gzip.GzipFile(mode='r',
> +                        fileobj=StringIO(self._data)) as obj:
> +                    shutil.copyfileobj(obj, out)

This method is really opaque - can you add comments on what it's doing?

Comment 10

[Mike Hommey [:glandium]]

https://hg.mozilla.org/mozilla-central/rev/a62bde1d6efe

Comment 11

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #10)
> https://hg.mozilla.org/mozilla-central/rev/a62bde1d6efe

Note this only enabled on try. I'll file a followup bug to enable on other branches after a period of testing on try.

Comment 12

[Rail Aliiev [:rail]]

Make sure that bug 966301 is resolved before we enable this on non-try branches. bld-linux64-ec2 slaves has no roles assigned yet.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Hopeful fix for comm-central check-sync-dirs bustage.
https://hg.mozilla.org/comm-central/rev/1eead52e8a51

https://tbpl.mozilla.org/php/getParsedLog.php?id=34623308&tree=Thunderbird-Trunk

Not sure what other changes might be needed to the build configs there, but this hopefully at least takes care of the immediate problem.