Closed Bug 1408836 Opened 3 years ago Closed 3 years ago

5.63 - 14.8% build times (windows2012-32, windows2012-32-rusttests, windows2012-64) regression on push 95dff4968a191f64341700d28fa954d898cd8367 (Sat Oct 14 2017)

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(firefox-esr52 unaffected, firefox56 unaffected, firefox57 unaffected, firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox-esr52 --- unaffected
firefox56 --- unaffected
firefox57 --- unaffected
firefox58 --- fixed

People

(Reporter: igoldan, Assigned: ted)

References

Details

(Keywords: regression)

Attachments

(1 file)

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=95dff4968a191f64341700d28fa954d898cd8367

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 15%  build times summary windows2012-64 opt static-analysis taskcluster-c4.4xlarge     2,627.71 -> 3,016.69
 13%  build times summary windows2012-32 opt static-analysis taskcluster-c4.4xlarge     2,549.04 -> 2,886.88
 13%  build times summary windows2012-32-rusttests opt taskcluster-c4.4xlarge           2,350.66 -> 2,646.14
 12%  build times summary windows2012-32 opt taskcluster-c4.4xlarge                     2,670.84 -> 2,982.84
 10%  build times summary windows2012-64 opt taskcluster-c4.4xlarge                     2,722.51 -> 2,990.57
  7%  build times summary windows2012-64 pgo taskcluster-c4.4xlarge                     5,310.73 -> 5,676.06
  6%  build times summary windows2012-32 pgo taskcluster-c4.4xlarge                     5,024.18 -> 5,306.97

Improvements:

100%  sccache hit rate summary windows2012-64 opt static-analysis taskcluster-c4.4xlarge     0.79 -> 0.00
100%  sccache hit rate summary windows2012-32 debug static-analysis taskcluster-c4.4xlarge   0.76 -> 0.00
 91%  sccache hit rate summary windows2012-64 debug static-analysis taskcluster-c4.4xlarge   0.85 -> 0.08
 84%  sccache hit rate summary windows2012-32 opt static-analysis taskcluster-c4.4xlarge     0.84 -> 0.14
  6%  sccache hit rate summary windows2012-32-rusttests opt taskcluster-c4.4xlarge           0.99 -> 0.92
  6%  sccache hit rate summary windows2012-32 opt taskcluster-c4.4xlarge                     0.98 -> 0.92
  6%  sccache hit rate summary windows2012-64 opt taskcluster-c4.4xlarge                     0.99 -> 0.93
  6%  sccache hit rate summary windows2012-64 debug taskcluster-c4.4xlarge                   0.97 -> 0.92


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=10018

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics
Component: Untriaged → Build Config
Product: Firefox → Core
:rillian FYI Updating Rust to 1.20.0 increased the build times on Windows platform.
Flags: needinfo?(giles)
No longer blocks: 1402299
It looks like there are a lot of sccache misses on rust (quick glance suggests 100% miss) since the upgrade.
Flags: needinfo?(ted)
(In reply to Mike Hommey [:glandium] from comment #2)
> It looks like there are a lot of sccache misses on rust (quick glance
> suggests 100% miss) since the upgrade.

Indeed it does. Thanks for the heads-up.

Rust 1.21.0 is on autoland now. Hopefully that will be an improvement.
Flags: needinfo?(giles)
Looking at the couple of builds we got before the trees closed, builds with rust 1.21 had the same 100% miss rate.
I found time to look at this today, and testing with sccache and a simple Rust crate showed that I was getting zero cache hits on Windows. Digging in a little bit showed that the root cause was that sccache includes all environment variables starting with CARGO_ in the cache key, and since cargo gained jobserver support it passes an env var like `CARGO_MAKEFLAGS=--jobserver-fds=__rust_jobserver_semaphore_992383377 --jobserver-auth=__rust_jobserver_semaphore_992383377` which is very non-cacheable.

I'm going to write a small patch to filter out `CARGO_MAKEFLAGS` from sccache's hash key generation, and also write an sccache test that tests Rust compilation via cargo.
Assignee: nobody → ted
Flags: needinfo?(ted)
I landed my sccache fix (along with a test) and released 0.2.2 on crates.io:
https://github.com/mozilla/sccache/commit/2d9bdc1e8fb1fa5a531da2930670bbfcb66f78a0
Blocks: 1409386
Comment on attachment 8922148 [details]
bug 1408836 - update sccache to 0.2.2 to fix a Rust caching regression from Rust 1.20.

https://reviewboard.mozilla.org/r/193148/#review198388

::: taskcluster/scripts/misc/build-sccache.sh:5
(Diff revision 1)
>  #!/bin/bash
>  set -x -e -v
>  
> +# 0.2.2
>  SCCACHE_REVISION=df04fa530d6b7d79fef8c848879d47dcc4d95b32

You're not changing the sha1.
Attachment #8922148 - Flags: review?(mh+mozilla)
Comment on attachment 8922148 [details]
bug 1408836 - update sccache to 0.2.2 to fix a Rust caching regression from Rust 1.20.

https://reviewboard.mozilla.org/r/193148/#review198394
Attachment #8922148 - Flags: review?(mh+mozilla) → review+
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/c3fb1be6aca0
update sccache to 0.2.2 to fix a Rust caching regression from Rust 1.20. r=glandium
https://hg.mozilla.org/mozilla-central/rev/c3fb1be6aca0
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Depends on: 1409048
No longer depends on: 1409048
Duplicate of this bug: 1409048
I noticed this small improvement:

== Change summary for alert #10184 (as of Thu, 26 Oct 2017 00:30:39 GMT) ==

Improvements:

  5%  build times summary windows2012-32 opt static-analysis taskcluster-c4.4xlarge     2,685.00 -> 2,555.24

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=10184
However, most of the regressions remained.
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.