1373334 - Look into why sccache for rustc is not a big win for non-stylo Firefox builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

In bug 1357825 comment 15 I compared build times of builds using sccache only for C++ compilation vs. builds using sccache for C++ and Rust compilation:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=48a41d0d35beab8649f7a8f52b1bb41f5a1ecec5&newProject=try&newRevision=d38783e43fc74a70b595f32b43a50cfcc182178e&framework=2&filter=build%20times&showOnlyImportant=0

There wasn't a big difference in build times on non-stylo builds. stylo builds are obviously building much more Rust code, but we enabled building webrender by default a while ago so we're building a fair number of crates as part of the standard build, so I would have expected *some* build time win. It seems to be only ~15 seconds faster on linux64-opt, and possibly 10 seconds *slower* on linux32-opt.

I grepped cargo's output out of the build logs and it looks like we build 139 crates in a normal Linux Firefox build, and 208 in a stylo build. This is the diff between the list of crates from a normal Firefox build and a stylo build (additions are only in stylo):
https://gist.github.com/3c488a6d6a25338fd599c2f9000b7c65

It's pretty hard to diagnose this from CI builds because we just don't have enough information about what cargo is doing internally. I opened a cargo issue about getting fine-grained build timings out: https://github.com/rust-lang/cargo/issues/4164 . That would certainly help us see where time is being spent.

When I was testing sccache with Servo, I noticed that even in a build with 100% cache hits we wasted a lot of time running build scripts. Generally these were -sys crates that were building some C++ code, so they'd spend a bunch of time invoking some other build system. Even using sccache as ccache in these cases we still do a lot of wasted work, since we'd wind up getting cache hits for the -sys crate itself, so we wouldn't even *use* the C++ libraries the build script built. It might behoove us to figure out a way to allow sccache to cache cargo's build script invocations.

Comment 1

[Mike Hommey [:glandium]]

Bug 1380458 makes a significant difference.

Comment 2

[(not currently active) Ted Mielczarek]

I commented there, should have commented here instead:
That sure helped! Nice drop around July 14th: https://mzl.la/2vf2HCP

Most noticeable on the Windows builds. The osx-cross data is bimodal because of the static analysis builds, we should turn them off (bug 1355118).

Comment 3

[Chris Peterson [:cpeterson]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> I commented there, should have commented here instead:
> That sure helped! Nice drop around July 14th: https://mzl.la/2vf2HCP

Ted, can we resolve this bug as fixed then? Or are there other sccache issues you would like to investigate?

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to Chris Peterson [:cpeterson] from comment #3)
> Ted, can we resolve this bug as fixed then? Or are there other sccache
> issues you would like to investigate?

I'll do another pair of try pushes to compare build times with and without sccache-for-rust and see what it looks like after those fixes. One of the pushes will be stock m-c, one will be with my patch from bug 1357825 backed out.

Comment 5

[Chris Peterson [:cpeterson]]

Is this bug still relevant now that we are now building Stylo support (pref'd off) by default for automation and developer builds? A "non-stylo Firefox build" is no longer the default configuration.

Comment 6

[(not currently active) Ted Mielczarek]

Probably not. I think glandium fixed the underlying issues anyway.