1373460 - Enable sccache for all builds in automation

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

Currently, the in-tree mozconfigs must include build/mozconfig.cache in order to have sccache enabled. There appears to be a divide between "trunk" builds and release builds: sccache is enabled for most things that run on mozilla-central but isn't enabled for beta, release, etc configurations (with the exception of macosx64 on beta, which I suspect is a bug). This means that beta and release builds are several minutes slower on average compared to nightly builds.

I suspect that the reason for not enabling compiler caching on release builds stems from historical concern of the build system's robustness. I'd like to think that build system improvements in the past few years (including better sandboxing of builds in TaskCluster) coupled with the transition from ccache to sccache mean that we now trust our build system caching and can therefore enable caching by default in automation.

ted: do you feel comfortable a) enabling sccache by default in automation b) globally enabling it for all builds (where it is supported)?

Comment 1

[Mike Hommey [:glandium]]

I don't think that build caching should be enabled on any build that reaches users. That goes against the whole chain-of-trust, and, in a future where sccache gets more lenient about source locations and things like that, will kill build determinism for those builds.

Combine that with release promotion, and you can't have nice things...

Comment 2

[Gregory Szorc [:gps]]

Playing devil's advocate, if all compiler inputs are hashed with a cryptographically strong hasher, what's the threat model? Cache poisoning?

Regarding source locations, those should be constant across Firefox CI, right. Even Tor's deterministic build does so in a VM where the paths are constant, no?

Regardless of what we do here, if the use of sccache is really a security/determinism issue, we should be enforcing it in taskgraph or in configure, not at the mozconfig/mozharness layer (which is fragile and as I said in the initial comment is already inconsistent for macosx64 beta).

Comment 3

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #2)
> Playing devil's advocate, if all compiler inputs are hashed with a
> cryptographically strong hasher, what's the threat model? Cache poisoning?

Sounds like it.

Note, I'm not sure what chain-of-trust is trying to protect from, but it tries to identify everything involved in the build, and having compiler input sent to a black box, and getting back unidentified data sounds like it goes against its model. Ask aki, I guess.

> Regarding source locations, those should be constant across Firefox CI,
> right.

Not until buildbot dies.

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to Mike Hommey [:glandium] from comment #1)
> I don't think that build caching should be enabled on any build that reaches
> users. That goes against the whole chain-of-trust, and, in a future where
> sccache gets more lenient about source locations and things like that, will
> kill build determinism for those builds.

I don't think I'm likely to fix the alternate source paths stuff at this point, given that we're getting very close to a taskcluster-only future.

(In reply to Gregory Szorc [:gps] from comment #0)
> ted: do you feel comfortable a) enabling sccache by default in automation b)
> globally enabling it for all builds (where it is supported)?

Enabling it by default in automation should be generally OK, since our build machines should all have the necessary configuration to make it work anyway. However, I don't think it will provide any net benefit for PGO builds, which are what we ship on everything-but-OS X anyway, so maybe we can just skip it for Nightly/Release builds and still feel fairly confident?

(In reply to Gregory Szorc [:gps] from comment #2)
> Playing devil's advocate, if all compiler inputs are hashed with a
> cryptographically strong hasher, what's the threat model? Cache poisoning?

I think it ought to be safe, but it's kind of hard to reason about as well. Maybe if we get to a point where our Nightly+Release builds are built in a reproducible manner this would be easier to justify since we ought to be able to verify that we're producing the exact same bits?
 
> Regardless of what we do here, if the use of sccache is really a
> security/determinism issue, we should be enforcing it in taskgraph or in
> configure, not at the mozconfig/mozharness layer (which is fragile and as I
> said in the initial comment is already inconsistent for macosx64 beta).

This I agree with. When enabling sccache for Taskcluster builds I did put a `needs-sccache` attribute on tasks:
https://dxr.mozilla.org/mozilla-central/rev/fe809f57bf2287bb937c3422ed03a63740b3448b/taskcluster/taskgraph/transforms/task.py#151
https://dxr.mozilla.org/mozilla-central/rev/fe809f57bf2287bb937c3422ed03a63740b3448b/taskcluster/taskgraph/transforms/task.py#586
https://dxr.mozilla.org/mozilla-central/rev/fe809f57bf2287bb937c3422ed03a63740b3448b/taskcluster/taskgraph/transforms/job/__init__.py#62
https://dxr.mozilla.org/mozilla-central/rev/fe809f57bf2287bb937c3422ed03a63740b3448b/taskcluster/taskgraph/transforms/build.py#23
https://dxr.mozilla.org/mozilla-central/rev/fe809f57bf2287bb937c3422ed03a63740b3448b/taskcluster/taskgraph/transforms/job/mozharness.py#233

I'd love to have that be authoritative and basically get rid of mozconfig.cache.