Enable sccache on more builds

RESOLVED FIXED in Firefox 56

Status

enhancement
RESOLVED FIXED
2 years ago
Last year

People

(Reporter: glandium, Assigned: glandium)

Tracking

(Blocks 1 bug)

unspecified
mozilla56
Dependency tree / graph

Firefox Tracking Flags

(firefox56 fixed)

Details

Attachments

(4 attachments)

This is kind of related to bug 1373460, but it feels like that bug is broader. I'm merely concerned that, most notably, PGO and ASAN builds are not using sccache. For PGO, while it didn't matter in the past, because sccache doesn't do anything (although, actually, it probably caches the first pass), it would make a bigger difference now that a lot of rust code is built that can be cached (and is not PGOed anyways).
Note this patch queue leaves out two problems with PGO builds:
- Stats are wrong on Windows builds because the sccache server shuts down during xul.dll linking -> we'll need https://github.com/mozilla/sccache/pull/155
- Stats are doubly wrong because preflight/postflight run before and after each build pass, which means we're shutting down the sccache server between the two passes. The --show-stats is also run at the end of both passes. I'm leaving this for a followup.
(In reply to Mike Hommey [:glandium] from comment #5)
> Note this patch queue leaves out two problems with PGO builds:
> - Stats are wrong on Windows builds because the sccache server shuts down
> during xul.dll linking -> we'll need
> https://github.com/mozilla/sccache/pull/155

I noticed this for our native OS X builds a while ago and filed bug 1359880, but didn't get around to fixing it. My plan was to just make the sccache server persist stats to disk.
Comment on attachment 8888180 [details]
Bug 1382502 - Add sccache to asan/msan/fuzzing tooltool manifest.

https://reviewboard.mozilla.org/r/159108/#review166500
Attachment #8888180 - Flags: review?(ted) → review+
Comment on attachment 8888181 [details]
Bug 1382502 - Include mozconfig.cache in most cases.

https://reviewboard.mozilla.org/r/159110/#review166502

::: build/mozconfig.cache:111
(Diff revision 2)
>  
>  if test -z "$bucket"; then
>      case "$platform" in
>      win*) : ;;
>      *)
> -        ac_add_options --with-ccache
> +        export CCACHE=ccache

I think we should honestly just get rid of this at this point. ccache probably isn't a net win for our builders.
Attachment #8888181 - Flags: review?(ted) → review+
Comment on attachment 8888182 [details]
Bug 1382502 - Enable sccache on PGO builds.

https://reviewboard.mozilla.org/r/159112/#review166506
Attachment #8888182 - Flags: review?(ted) → review+
Comment on attachment 8888183 [details]
Bug 1382502 - Remove ${var}_IS_SET variables.

https://reviewboard.mozilla.org/r/159114/#review166508
Attachment #8888183 - Flags: review?(ted) → review+
Comment on attachment 8888181 [details]
Bug 1382502 - Include mozconfig.cache in most cases.

https://reviewboard.mozilla.org/r/159110/#review166502

> I think we should honestly just get rid of this at this point. ccache probably isn't a net win for our builders.

I'll file a separate bug.
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 00031c3b8755 -d ba130cb1f427: rebasing 409197:00031c3b8755 "Bug 1382502 - Add sccache to asan/msan/fuzzing tooltool manifest. r=ted"
merging browser/config/tooltool-manifests/linux64/asan.manifest
warning: conflicts while merging browser/config/tooltool-manifests/linux64/asan.manifest! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/7c81b19ddec1
Add sccache to asan/msan/fuzzing tooltool manifest. r=ted
https://hg.mozilla.org/integration/autoland/rev/78f62bcbe088
Include mozconfig.cache in most cases. r=ted
https://hg.mozilla.org/integration/autoland/rev/ed4688fd119a
Enable sccache on PGO builds. r=ted
https://hg.mozilla.org/integration/autoland/rev/692beddc13e3
Remove ${var}_IS_SET variables. r=ted
Blocks: 1384340
This introduced lots of compiler warnings. Can you take a look over them?

== Change summary for alert #8273 (as of July 25 2017 23:04 UTC) ==

Regressions:

3242%  compiler warnings summary linux64 asan asan debug     328.00 -> 10,961.42
2850%  compiler warnings summary linux64 asan asan opt       297.50 -> 8,777.42
2826%  compiler warnings summary linux64 asan asan-fuzzing   299.17 -> 8,752.83
  5%  compiler warnings summary linux32 pgo                 1,407.33 -> 1,472.33
  4%  compiler warnings summary linux64 pgo                 1,450.67 -> 1,515.83

Improvements:

 46%  build times summary linux64 pgo taskcluster-m4.4xlarge     3,925.92 -> 2,101.03
 31%  build times summary linux32 pgo taskcluster-m4.4xlarge     3,505.67 -> 2,415.71
  7%  compiler warnings summary windows2012-32 pgo               141.00 -> 131.00
  6%  compiler warnings summary windows2012-64 pgo               160.08 -> 150.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=8273
Flags: needinfo?(mh+mozilla)
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #22)
>  46%  build times summary linux64 pgo taskcluster-m4.4xlarge     3,925.92 ->
> 2,101.03
>  31%  build times summary linux32 pgo taskcluster-m4.4xlarge     3,505.67 ->

Wow, that's much more improvement than I would have expected on PGO builds!
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #22)
> This introduced lots of compiler warnings. Can you take a look over them?
> 
> == Change summary for alert #8273 (as of July 25 2017 23:04 UTC) ==
> 
> Regressions:
> 
> 3242%  compiler warnings summary linux64 asan asan debug     328.00 ->
> 10,961.42
> 2850%  compiler warnings summary linux64 asan asan opt       297.50 ->
> 8,777.42
> 2826%  compiler warnings summary linux64 asan asan-fuzzing   299.17 ->
> 8,752.83
>   5%  compiler warnings summary linux32 pgo                 1,407.33 ->
> 1,472.33
>   4%  compiler warnings summary linux64 pgo                 1,450.67 ->
> 1,515.83

That's the result of using sccache, which has some issues with this. Considering these shortcomings are not limited to these builds, I'm inclined to ignore them. Bug 1384436 should also make things better.
Flags: needinfo?(mh+mozilla)
> Bug 1384436 should also make things better.

bwarf, if made things worse. Ted, I'm going to address the things raised in https://github.com/mozilla/sccache/issues/117 and will probably address https://github.com/mozilla/sccache/issues/163 at the same time.
Possibly related to this recent change?
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(mh+mozilla)
Depends on: 1386028
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.