Closed
Bug 1286934
Opened 9 years ago
Closed 9 years ago
Switch automation build jobs to use sccache2
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(firefox53 fixed)
RESOLVED
FIXED
mozilla53
Tracking | Status | |
---|---|---|
firefox53 | --- | fixed |
People
(Reporter: ted, Assigned: ted)
References
Details
Attachments
(1 file)
I've got sccache2 far enough along that I'm going to start testing it on try. I plan to ensure that everything works OK as well as run comparative builds from the same base changeset to compare build times and cache hit rates.
If that all looks good we'll be able to land my patch to switch automation to use sccache2!
Assignee | ||
Comment 1•9 years ago
|
||
Comment 2•9 years ago
|
||
Before deploying, you should add support for SCCACHE_RECACHE. That's the most immediate thing that I found missing, I still need to go through most of the code (I've read it a little already).
Comment 3•9 years ago
|
||
I should mention why: I've had to use it a couple times in the past because of bugs in sccache causing bad things in the cache and breaking builds permanently on try. The last occurrence was related to MACOSX_DEPLOYMENT_TARGET.
Assignee | ||
Comment 4•9 years ago
|
||
Assignee | ||
Comment 5•9 years ago
|
||
Assignee | ||
Comment 6•9 years ago
|
||
Assignee | ||
Comment 7•9 years ago
|
||
Assignee | ||
Comment 8•9 years ago
|
||
Assignee | ||
Comment 9•9 years ago
|
||
Assignee | ||
Comment 10•9 years ago
|
||
Assignee | ||
Comment 11•9 years ago
|
||
Assignee | ||
Comment 12•9 years ago
|
||
Assignee | ||
Comment 13•9 years ago
|
||
Comment 14•9 years ago
|
||
Hi ted, what is the latest for this work?
Assignee | ||
Comment 15•9 years ago
|
||
Assignee | ||
Comment 16•9 years ago
|
||
Assignee | ||
Comment 17•9 years ago
|
||
Assignee | ||
Comment 18•9 years ago
|
||
Assignee | ||
Comment 19•9 years ago
|
||
Assignee | ||
Comment 20•9 years ago
|
||
Assignee | ||
Comment 21•9 years ago
|
||
Assignee | ||
Comment 22•9 years ago
|
||
Assignee | ||
Comment 23•9 years ago
|
||
Assignee | ||
Comment 24•9 years ago
|
||
Assignee | ||
Comment 25•9 years ago
|
||
I was stuck on those burning OS X jobs and I finally realized the problem--the mac builders are the only non-EC2 builders, so they don't have IAM credentials, and the code I'm using to locate AWS credentials (borrowed from rusoto) defaults to looking in `~/.aws/credentials`, but the credentials that sccache.py is using are located in `~/.boto`. I've got a fix locally, I'll push it to try shortly and that should fix those builds, which should mean everything is green.
Assignee | ||
Comment 26•9 years ago
|
||
Assignee | ||
Comment 27•9 years ago
|
||
Assignee | ||
Comment 28•9 years ago
|
||
Assignee | ||
Comment 29•9 years ago
|
||
Assignee | ||
Comment 30•9 years ago
|
||
Assignee | ||
Comment 31•9 years ago
|
||
Assignee | ||
Comment 32•9 years ago
|
||
Assignee | ||
Comment 33•9 years ago
|
||
So after a lot of fiddling (obviously) here's a try job with the original sccache using SCCACHE_RECACHE=1 to force compiling everything instead of reading from the cache:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1c7b7676fb5c3b5883cc2c4566785376c0aafe98
and here's a try run with sccache2 also using SCCACHE_RECACHE=1:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0aca4756d9675c899c5b1a04a89220b49ead79fa
They're built atop the same base revision.
Assignee | ||
Comment 34•9 years ago
|
||
Assignee | ||
Comment 35•9 years ago
|
||
Assignee | ||
Comment 36•9 years ago
|
||
Assignee | ||
Comment 37•9 years ago
|
||
Assignee | ||
Comment 38•9 years ago
|
||
Assignee | ||
Comment 39•9 years ago
|
||
Here's a comparison of two try pushes, similar to comment 33, except I removed the SCCACHE_RECACHE, so this is comparing builds that should be mostly cache hits:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4f7d1609a34d485119d81f1b3e93bb4294fbabc1&newProject=try&newRevision=139e47cbd01b681ee39bdd1338de7d3b83c77871&framework=2&showOnlyImportant=0
I also rebased on top of gps' patches to split the build metrics out between buildbot and taskcluster, as well as by instance type, which helps.
Assignee | ||
Comment 40•9 years ago
|
||
and here's a comparison of a new set of builds (with the same base changeset as comment 39) but with SCCACHE_RECACHE, so it's comparing builds with all cache misses:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=72314f1af81d3ab4404e001d1cf585a1d89bdb58&newProject=try&newRevision=0dff60781e6967d074830232d940a2354885145a&framework=2&showOnlyImportant=0
Assignee | ||
Comment 41•9 years ago
|
||
Assignee | ||
Comment 42•9 years ago
|
||
Assignee | ||
Comment 43•9 years ago
|
||
FYI I summarized the build time comparisons I did on try (comment 39 and comment 40):
https://docs.google.com/document/d/1HWTMnStwFzUVNYAh6K7MwwOShspkDc_w8xCruyEqTWg/edit
I was mostly just trying to ensure that I didn't regress anything, but it looks like sccache2 will actually give us noticeable build time improvements on many platforms.
Assignee | ||
Comment 44•9 years ago
|
||
Comment hidden (mozreview-request) |
Assignee | ||
Comment 46•9 years ago
|
||
This still depends on landing bug 1295937, but that's close enough to landing that I thought I'd get this patch up for review. Most of the useful info should be in the commit message, but here's a few other things:
I'm currently building the binaries using this script on my local Windows/Linux/Mac machines: https://github.com/luser/sccache2/blob/master/scripts/build-release.sh . I'd like to move to something better but I haven't quite figured that out yet. Taskcluster doesn't have support for mac workers yet, so to build binaries in TC I'd have to cross-compile them from Linux (which is feasible).
I'm using another script to upload the resulting binaries to tooltool:
https://github.com/luser/sccache2/blob/master/scripts/upload-tooltool.sh
...and a third script to take those resulting tooltool manifests and merge them into the in-tree manifests:
https://github.com/luser/sccache2/blob/master/scripts/update-gecko-manifests.py
This whole process is kinda crappy. I would love if we could fix bug 1313111 and just have this all work in the taskgraph.
Depends on: 1295937
Comment 47•9 years ago
|
||
mozreview-review |
Comment on attachment 8811801 [details]
bug 1286934 - Switch to using sccache2.
https://reviewboard.mozilla.org/r/93758/#review94068
This looks pretty straightforward!
We should get the scripts for building sccache in the tree. Even if they aren't hooked up to the task graph, it is better than them sitting in some random repo elsewhere. I guess you can toss them in `build/build-sccache` or some such and we can refactor things later.
Attachment #8811801 -
Flags: review?(gps) → review+
Assignee | ||
Comment 48•9 years ago
|
||
I'm fine with putting them wherever, but the *build* is pretty simple. It's the tooltool bits that are the PITA. :-/
Comment 49•9 years ago
|
||
mozreview-review |
Comment on attachment 8811801 [details]
bug 1286934 - Switch to using sccache2.
https://reviewboard.mozilla.org/r/93758/#review93952
::: browser/config/tooltool-manifests/linux32/releng.manifest:36
(Diff revision 1)
> },
> {
> -"size": 167175,
> -"digest": "0b71a936edf5bd70cf274aaa5d7abc8f77fe8e7b5593a208f805cc9436fac646b9c4f0b43c2b10de63ff3da671497d35536077ecbc72dba7f8159a38b580f831",
> "algorithm": "sha512",
> -"filename": "sccache.tar.bz2",
> +"visibility": "public",
in-tree manifests don't need visibility
::: build/sccache.mk:14
(Diff revision 1)
> BASE_DIR = $(MOZ_OBJDIR)/$(firstword $(MOZ_BUILD_PROJECTS))
> endif
>
> preflight_all:
> # Terminate any sccache server that might still be around
> - -python2.7 $(TOPSRCDIR)/sccache/sccache.py > /dev/null 2>&1
> + -$(TOPSRCDIR)/sccache2/sccache --stop-server > /dev/null 2>&1
Should need .exe on Windows, right?
::: build/sccache.mk:14
(Diff revision 1)
> BASE_DIR = $(MOZ_OBJDIR)/$(firstword $(MOZ_BUILD_PROJECTS))
> endif
>
> preflight_all:
> # Terminate any sccache server that might still be around
> - -python2.7 $(TOPSRCDIR)/sccache/sccache.py > /dev/null 2>&1
> + -$(TOPSRCDIR)/sccache2/sccache --stop-server > /dev/null 2>&1
Is the redirection to /dev/null still necessary?
Assignee | ||
Comment 50•9 years ago
|
||
mozreview-review-reply |
Comment on attachment 8811801 [details]
bug 1286934 - Switch to using sccache2.
https://reviewboard.mozilla.org/r/93758/#review93952
> in-tree manifests don't need visibility
Turns out if you don't specify --visibility=public tooltool.py will refuse to upload things, which is why this is here. I'm not motivated enough to try to fix that just for the sake of making the manifests look nicer.
> Should need .exe on Windows, right?
This works because the msys shell will find it with or without the exe. (The change in mozconfig.cache is because we were passing the path to `test -e`.
> Is the redirection to /dev/null still necessary?
Probably not, but I stuck an "echo stats to the log" bit earlier in the build, so it'd be redundant. I haven't yet made sccache2 save its stats to disk, and with the 5 minute inactivity timeout the server would shut itself down in the lull between the compilation phase finishing and the time the build hits `postflight_all`, so the stats were getting lost.
Longer-term I might want to integrate special sccache handling into mach or something.
Assignee | ||
Comment 51•9 years ago
|
||
Assignee | ||
Comment 52•9 years ago
|
||
Assignee | ||
Comment 53•9 years ago
|
||
Assignee | ||
Comment 54•9 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/683e59dc30940172227d80afa8b4e0d9a2cb1bb3
bug 1286934 - Switch to using sccache2. r=gps
Assignee | ||
Comment 55•9 years ago
|
||
FYI, I moved the sccache2 repo to mozilla/sccache:
https://github.com/mozilla/sccache
Comment 56•9 years ago
|
||
bugherder |
Status: NEW → RESOLVED
Closed: 9 years ago
status-firefox53:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
Updated•7 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•