Update sccache version

RESOLVED FIXED in Firefox 55

Status

()

Core
Build Config
RESOLVED FIXED
7 months ago
5 months ago

People

(Reporter: ted, Assigned: ted)

Tracking

unspecified
mozilla55
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox55 fixed)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Assignee)

Description

7 months ago
We've been using the same version of sccache in CI for quite a while. It has had significant changes made to it, including a near-complete rewrite of the networking layer to use tokio.

Let's get it updated to the latest version to make sure that everything still works well. Rust CI has been using the newer versions and aside from a few issues which they've since fixed it seems to be working well.
(Assignee)

Comment 1

7 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb403c84a373
(Assignee)

Comment 2

6 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7b2e833d54bad91ac4a349e9df0f95919de003fd
(Assignee)

Comment 3

6 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=74cecc2cdbc30b70e86c31dd1ed6bdd1d2c4bbc2
(Assignee)

Comment 4

6 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2a1ae00b863a5f09ab3eb0872b6a1b7f7f9fc26
(Assignee)

Comment 5

6 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf459fd4eb49fc9cfcfaaee8500ab52b7924a4d6
(Assignee)

Comment 6

6 months ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=bf459fd4eb49fc9cfcfaaee8500ab52b7924a4d6

OK, this is finally green. A summary of things since that first try push.
1) The Windows builds were busted, it turns out that I had actually completely broken Windows support with this sccache change:
https://github.com/mozilla/sccache/commit/c71af2559741905545218b53e593c73426766c90 (because the serde crate had a bug), but failed to notice because the tests that would have caught it aren't running in sccache CI. I fixed that by updating to a newer serde: https://github.com/mozilla/sccache/commit/371119c4d6f3f55ddf2437694130e24326430cbd

2) I had noticed on a different try push that sccache wasn't caching any compiles on the taskcluster mac builds. This turned out to be due to not properly handling some of the compiler options we were passing. I fixed that here: https://github.com/mozilla/sccache/commit/49523b0d8c9998ebe9de21ea208d3ba0e5317a2 . However, that exposed another breakage on the mac builds, because they're building with --enable-clang-plugin, and there are tests in build/clang-plugin/tests that build with -fsyntax-only which sccache didn't have any support for (it makes the compiler not produce object files). I made sccache not try to cache those compiles: https://github.com/mozilla/sccache/commit/ec10cdb2ddeb3dde9891bea1fa095e504a60a28a .

3) I had broken some things in a refactoring commit: https://github.com/mozilla/sccache/commit/5092b94f29ec6b8b63c09b1d4860073f59bd02a4 . Thankfully a sccache contributor fixed it for me: https://github.com/mozilla/sccache/commit/335f191205f41970d3c79494071bd22cb20023d3 .

4) I had merged a PR from acrichto to add support for MSVC's -showIncludes: https://github.com/mozilla/sccache/commit/334359c396f35df12eb5be94ff25fcbc01915d08 but it didn't produce quite the same output as running MSVC, which broke our configure check for the -showIncludes prefix. Alex fixed that: https://github.com/mozilla/sccache/commit/f22a7d82521cd14de2183046270a6300a0352bf8 .
Comment hidden (mozreview-request)

Comment 8

6 months ago
mozreview-review
Comment on attachment 8857418 [details]
bug 1354048 - update sccache to rev 7a3847276d05deb564cb84a16b8c551b690aaa3e.

https://reviewboard.mozilla.org/r/129416/#review132010

Is there still something wrong on Windows? I checked 2 buildbot builds and 2 TC builds, and all of them have cache write errors:

13:58:08     INFO -  Compile requests              3115
13:58:08     INFO -  Compile requests executed     3111
13:58:08     INFO -  Cache hits                       0
13:58:08     INFO -  Cache misses                  3111
13:58:08     INFO -  Cache timeouts                   0
13:58:08     INFO -  Forced recaches                  0
13:58:08     INFO -  Cache write errors            3111

Linux/OSX/Android all show lots of cache hits and no errors.

::: browser/config/tooltool-manifests/macosx64/clang.manifest:19
(Diff revision 1)
>      "algorithm": "sha512",
>      "filename": "rustc.tar.bz2",
>      "unpack": true
>    },
>    {
> -    "version": "sccache rev b21198a7183a2fe226ff49348b1c0b51bae9f4f8",
> +    "version": "sccache rev 335f191205f41970d3c79494071bd22cb20023d3",

Is there a reason OSX needs a different version than the rest?
Attachment #8857418 - Flags: review?(mshal)
(Assignee)

Comment 9

6 months ago
mozreview-review-reply
Comment on attachment 8857418 [details]
bug 1354048 - update sccache to rev 7a3847276d05deb564cb84a16b8c551b690aaa3e.

https://reviewboard.mozilla.org/r/129416/#review132010

Hm, I apparently missed that! I will take a look.

> Is there a reason OSX needs a different version than the rest?

Oops, I guess I didn't update my repo to the same changeset on OS X. It's not harmful, since the only additional commit is the `-showIncludes` fix for MSVC, but I'll rebuild so we're using the same rev everywhere.
(Assignee)

Updated

6 months ago
Blocks: 1355961
(Assignee)

Comment 10

6 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=26f6a6b9b829484d8bb9e08782ace623bbca96bd
(Assignee)

Comment 11

6 months ago
mozreview-review-reply
Comment on attachment 8857418 [details]
bug 1354048 - update sccache to rev 7a3847276d05deb564cb84a16b8c551b690aaa3e.

https://reviewboard.mozilla.org/r/129416/#review132010

I figured it out! The tokio refactor had changed the HTTP library we used to fetch S3 credentials from the EC2 IAM URL, and the new library doesn't transparently handle redirects. We only hit this on Windows because on Linux we're using the Taskcluster proxy to get credentials, and native OS X builds aren't in EC2. I fixed this the quick way by fixing the URL in question so it doesn't hit a redirect:
https://github.com/mozilla/sccache/commit/7a3847276d05deb564cb84a16b8c551b690aaa3e
Comment hidden (mozreview-request)
(Assignee)

Comment 13

6 months ago
Try push, looking pretty green:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d109feb0dcb375bf010c33de6acb45ce26d757cf

Comment 14

6 months ago
mozreview-review
Comment on attachment 8857418 [details]
bug 1354048 - update sccache to rev 7a3847276d05deb564cb84a16b8c551b690aaa3e.

https://reviewboard.mozilla.org/r/129416/#review133036

I think this is fine to land. I did still see some odd things in some of the stats, like a few (1-2) cache write errors in the Windows builds. I also tried pushing the same try push to see if some of the ones with lots of cache misses would get hits, and they did for the most part (https://treeherder.mozilla.org/#/jobs?repo=try&revision=9733b74ae8336f3f04af36f07c50df6b1781306a). The only exception is the TC OSX cross build - in your push, there were 0 hits and 2764 misses, whereas in my push there were 38 hits and 2927 non-cacheable calls. I'm not sure what to make of that (why would they be non-cacheable in mine but cacheable in and missed in yours?)

But I don't think that's worth blocking this over - we'll have a better idea of how things are working once we have the metrics in place for a while.
Attachment #8857418 - Flags: review?(mshal) → review+
(Assignee)

Comment 15

6 months ago
Yeah, I saw 1-2 cache write errors in a few builds, I think that's just transient S3 network failures. The OS X cross builds are interesting--they were not being cached at all with the previous version of sccache. I made some changes to try to change that, but it's possible that something still isn't working right. In any event, if they're not caching anything that's not a regression.
(Assignee)

Comment 16

6 months ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/8c8aba039a48898e87b64be661c2b353d3197bb6
bug 1354048 - update sccache to rev 7a3847276d05deb564cb84a16b8c551b690aaa3e. r=mshal

Comment 17

6 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/8c8aba039a48
Status: NEW → RESOLVED
Last Resolved: 6 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Blocks: 1361256
Depends on: 1357622
You need to log in before you can comment on or make changes to this bug.