Bug 940788 (sccache)

implement AWS S3-backed ccache for efficient cluster compilation of Firefox

RESOLVED FIXED in mozilla30

Status

defect
RESOLVED FIXED
6 years ago
Last year

People

(Reporter: taras.mozilla, Assigned: glandium)

Tracking

unspecified
mozilla30
x86_64
Windows 8
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments)

Reporter

Description

6 years ago
At the moment our build machines do not share state. In order to minimize compilation time one needs to write a fancy scheduler that schedules builds on machines with trees that are the closest ancestor. This is silly.

S3 is a fast key/blob store. We should key the store by hashes of dependencies of objects being built. This should work for object files, libraries and other build system artifacts. ccache direct mode appears to implement this for .o files, should be a good starting point to look into this.
Reporter

Updated

6 years ago
Blocks: 940789
Assignee

Updated

6 years ago
Depends on: 963304
Depends on: 959991
Depends on: 966048
Assignee

Updated

6 years ago
Depends on: 966152
Assignee

Updated

5 years ago
Alias: sccache
Assignee

Comment 1

5 years ago
I'm looking for a rubberstamp here. Maybe you can find someone more appropriate.

A few notes:
- This comes in the form of a patch because i don't think splinter handles non patches, but this is not, in fact, something to apply to some existing tree. This will come in a tarball along with the required components (dnspython, which, boto) distributed with tooltool.
- There are plenty of flags it doesn't bail on, because my intent hasn't been to create a generic tool that works everywhere. I was more focused on getting something that works for our needs, and this does. We can (and will) adjust later.
- The Timer class is kind of awful, but meh. That's something that's meant to go away, but I want to keep it in the initial deployment so that we have stats in case something goes wrong.
Attachment #8373845 - Flags: review?(catlee)
Assignee

Comment 2

5 years ago
sha512: af25ecf03b65795d21f011939984b130db167a4efc4f306700f373854f9d7ae664662cb7812c3d737eace7f3735324daa6eb540b5e42f90189b0d9a8dd5f4c9f

I don't know how tooltool works wrt setup scripts. Does it allow to use several setup scripts, or would i need to merge a setup script with the one for gcc?
Attachment #8373848 - Flags: checkin?(rail)
Comment on attachment 8373848 [details]
tarball to deploy on tooltool

uploaded:
size: 150816
sha512: af25ecf03b65795d21f011939984b130db167a4efc4f306700f373854f9d7ae664662cb7812c3d737eace7f3735324daa6eb540b5e42f90189b0d9a8dd5f4c9f


(In reply to Mike Hommey [:glandium] from comment #2)
> I don't know how tooltool works wrt setup scripts. Does it allow to use
> several setup scripts, or would i need to merge a setup script with the one
> for gcc?

It can only run one. You'll need to merge it with the gcc script and I can upload the new one.
Attachment #8373848 - Flags: checkin?(rail) → checkin+
Assignee

Comment 4

5 years ago
So here's a setup script for gcc+sccache.
Attachment #8374488 - Flags: checkin?(rail)
Comment on attachment 8374488 [details]
setup script to deploy on tooltool

size: 81
sha512: 59002eae04fa3534df3bbb0026dff50d3872313551514f9ccdf33080c8b8e34a6295b3b6f2c078b3ddef099023897a42adb65d34c9364f84dac5b8e7d022bf39
Attachment #8374488 - Flags: checkin?(rail) → checkin+
Assignee

Updated

5 years ago
Depends on: 971555
Assignee

Updated

5 years ago
Depends on: 971586
Comment on attachment 8374653 [details] [diff] [review]
Use sccache on try for builds happening on AWS that currently use ccache

Review of attachment 8374653 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/mozconfig.cache
@@ +4,5 @@
> +
> +# Setup for build cache
> +
> +bucket=
> +if test -f "$topsrcdir/sccache/sccache.py"; then

Since both of these files are in-repo, is this check necessary? If sccache didn't exist, I would assume that code such as this wouldn't be present either.

Looks fine otherwise.
Assignee

Comment 8

5 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #7)
> Since both of these files are in-repo, is this check necessary? If sccache
> didn't exist, I would assume that code such as this wouldn't be present
> either.

sccache/sccache.py is not in-repo, it's in tooltool. And it's not in the tooltool manifest for everything that's built on try spot/ec2 instances.
Attachment #8374653 - Flags: review?(bhearsum) → review+
Comment on attachment 8373845 [details] [diff] [review]
Initial shared cache implementation

Review of attachment 8373845 [details] [diff] [review]:
-----------------------------------------------------------------

Looks sane.

I'd encourage you to add far more comments / docs for this code if we ever want to maintain it as a real tool.

::: sccache/sccache.py
@@ +189,5 @@
> +        return self._status
> +
> +    @property
> +    def name(self):
> +        return self._name

small style nit: don't need to use properties for simple getters and setters. Use self.name, self.status as regular attributes. If you need extra behaviour controlling their access, then move them behind properties.

@@ +208,5 @@
> +                from cStringIO import StringIO
> +                import shutil
> +                with gzip.GzipFile(mode='r',
> +                        fileobj=StringIO(self._data)) as obj:
> +                    shutil.copyfileobj(obj, out)

This method is really opaque - can you add comments on what it's doing?
Attachment #8373845 - Flags: review?(catlee) → review+
Assignee

Comment 10

5 years ago
https://hg.mozilla.org/mozilla-central/rev/a62bde1d6efe
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla30
Assignee

Comment 11

5 years ago
(In reply to Mike Hommey [:glandium] from comment #10)
> https://hg.mozilla.org/mozilla-central/rev/a62bde1d6efe

Note this only enabled on try. I'll file a followup bug to enable on other branches after a period of testing on try.
Make sure that bug 966301 is resolved before we enable this on non-try branches. bld-linux64-ec2 slaves has no roles assigned yet.
Depends on: 966301
Hopeful fix for comm-central check-sync-dirs bustage.
https://hg.mozilla.org/comm-central/rev/1eead52e8a51

https://tbpl.mozilla.org/php/getParsedLog.php?id=34623308&tree=Thunderbird-Trunk

Not sure what other changes might be needed to the build configs there, but this hopefully at least takes care of the immediate problem.
Assignee

Updated

5 years ago
Blocks: sccache-win
Assignee

Updated

5 years ago
Blocks: 988722
Assignee

Updated

5 years ago
Blocks: 1008015

Updated

Last year
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.