Closed Bug 940788 (sccache) Opened 7 years ago Closed 7 years ago
implement AWS S3-backed ccache for efficient cluster compilation of Firefox
12.16 KB, patch
|Details | Diff | Splinter Review|
147.28 KB, application/x-xz-compressed-tar
81 bytes, application/x-shellscript
15.23 KB, patch
|Details | Diff | Splinter Review|
At the moment our build machines do not share state. In order to minimize compilation time one needs to write a fancy scheduler that schedules builds on machines with trees that are the closest ancestor. This is silly. S3 is a fast key/blob store. We should key the store by hashes of dependencies of objects being built. This should work for object files, libraries and other build system artifacts. ccache direct mode appears to implement this for .o files, should be a good starting point to look into this.
I'm looking for a rubberstamp here. Maybe you can find someone more appropriate. A few notes: - This comes in the form of a patch because i don't think splinter handles non patches, but this is not, in fact, something to apply to some existing tree. This will come in a tarball along with the required components (dnspython, which, boto) distributed with tooltool. - There are plenty of flags it doesn't bail on, because my intent hasn't been to create a generic tool that works everywhere. I was more focused on getting something that works for our needs, and this does. We can (and will) adjust later. - The Timer class is kind of awful, but meh. That's something that's meant to go away, but I want to keep it in the initial deployment so that we have stats in case something goes wrong.
Attachment #8373845 - Flags: review?(catlee)
sha512: af25ecf03b65795d21f011939984b130db167a4efc4f306700f373854f9d7ae664662cb7812c3d737eace7f3735324daa6eb540b5e42f90189b0d9a8dd5f4c9f I don't know how tooltool works wrt setup scripts. Does it allow to use several setup scripts, or would i need to merge a setup script with the one for gcc?
Attachment #8373848 - Flags: checkin?(rail)
Comment on attachment 8373848 [details] tarball to deploy on tooltool uploaded: size: 150816 sha512: af25ecf03b65795d21f011939984b130db167a4efc4f306700f373854f9d7ae664662cb7812c3d737eace7f3735324daa6eb540b5e42f90189b0d9a8dd5f4c9f (In reply to Mike Hommey [:glandium] from comment #2) > I don't know how tooltool works wrt setup scripts. Does it allow to use > several setup scripts, or would i need to merge a setup script with the one > for gcc? It can only run one. You'll need to merge it with the gcc script and I can upload the new one.
Attachment #8373848 - Flags: checkin?(rail) → checkin+
So here's a setup script for gcc+sccache.
Attachment #8374488 - Flags: checkin?(rail)
Comment on attachment 8374488 [details] setup script to deploy on tooltool size: 81 sha512: 59002eae04fa3534df3bbb0026dff50d3872313551514f9ccdf33080c8b8e34a6295b3b6f2c078b3ddef099023897a42adb65d34c9364f84dac5b8e7d022bf39
Attachment #8374488 - Flags: checkin?(rail) → checkin+
Comment on attachment 8374653 [details] [diff] [review] Use sccache on try for builds happening on AWS that currently use ccache Review of attachment 8374653 [details] [diff] [review]: ----------------------------------------------------------------- ::: build/mozconfig.cache @@ +4,5 @@ > + > +# Setup for build cache > + > +bucket= > +if test -f "$topsrcdir/sccache/sccache.py"; then Since both of these files are in-repo, is this check necessary? If sccache didn't exist, I would assume that code such as this wouldn't be present either. Looks fine otherwise.
(In reply to Ben Hearsum [:bhearsum] from comment #7) > Since both of these files are in-repo, is this check necessary? If sccache > didn't exist, I would assume that code such as this wouldn't be present > either. sccache/sccache.py is not in-repo, it's in tooltool. And it's not in the tooltool manifest for everything that's built on try spot/ec2 instances.
Attachment #8374653 - Flags: review?(bhearsum) → review+
Comment on attachment 8373845 [details] [diff] [review] Initial shared cache implementation Review of attachment 8373845 [details] [diff] [review]: ----------------------------------------------------------------- Looks sane. I'd encourage you to add far more comments / docs for this code if we ever want to maintain it as a real tool. ::: sccache/sccache.py @@ +189,5 @@ > + return self._status > + > + @property > + def name(self): > + return self._name small style nit: don't need to use properties for simple getters and setters. Use self.name, self.status as regular attributes. If you need extra behaviour controlling their access, then move them behind properties. @@ +208,5 @@ > + from cStringIO import StringIO > + import shutil > + with gzip.GzipFile(mode='r', > + fileobj=StringIO(self._data)) as obj: > + shutil.copyfileobj(obj, out) This method is really opaque - can you add comments on what it's doing?
Attachment #8373845 - Flags: review?(catlee) → review+
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla30
(In reply to Mike Hommey [:glandium] from comment #10) > https://hg.mozilla.org/mozilla-central/rev/a62bde1d6efe Note this only enabled on try. I'll file a followup bug to enable on other branches after a period of testing on try.
Make sure that bug 966301 is resolved before we enable this on non-try branches. bld-linux64-ec2 slaves has no roles assigned yet.
Depends on: 966301
Hopeful fix for comm-central check-sync-dirs bustage. https://hg.mozilla.org/comm-central/rev/1eead52e8a51 https://tbpl.mozilla.org/php/getParsedLog.php?id=34623308&tree=Thunderbird-Trunk Not sure what other changes might be needed to the build configs there, but this hopefully at least takes care of the immediate problem.
You need to log in before you can comment on or make changes to this bug.