1522771 - 16.3 - 88.63% build times / sccache hit rate (windows2012-32, windows2012-32-noopt, windows2012-64, windows2012-64-noopt) regression on push 4e34cf5e62ab77e5821d94de0fb1d6b92ec1e319 (Thu Jan 24 2019)

Status: VERIFIED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=4e34cf5e62ab77e5821d94de0fb1d6b92ec1e319

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

89% sccache hit rate windows2012-64 asan opt taskcluster-c5.4xlarge 0.95 -> 0.11
88% sccache hit rate windows2012-64-noopt debug taskcluster-c4.4xlarge 0.97 -> 0.11
88% sccache hit rate windows2012-64 asan debug taskcluster-c4.4xlarge 0.95 -> 0.11
88% sccache hit rate windows2012-64-noopt debug taskcluster-c5.4xlarge 0.92 -> 0.11
88% sccache hit rate windows2012-64 debug fuzzing taskcluster-c4.4xlarge 0.98 -> 0.12
88% sccache hit rate windows2012-64 asan debug taskcluster-c5.4xlarge 0.94 -> 0.11
88% sccache hit rate windows2012-32-noopt debug taskcluster-c5.4xlarge 0.93 -> 0.11
88% sccache hit rate windows2012-64 opt taskcluster-c5.4xlarge 0.93 -> 0.11
88% sccache hit rate windows2012-64 asan fuzzing-asan taskcluster-c4.4xlarge 0.95 -> 0.11
88% sccache hit rate windows2012-64 opt taskcluster-c4.4xlarge 0.96 -> 0.12
88% sccache hit rate windows2012-64 asan fuzzing-asan taskcluster-c5.4xlarge 0.92 -> 0.11
88% sccache hit rate windows2012-64 debug taskcluster-c5.4xlarge 0.91 -> 0.11
88% sccache hit rate windows2012-64 debug taskcluster-c4.4xlarge 0.94 -> 0.12
88% sccache hit rate windows2012-64 asan opt taskcluster-c4.4xlarge 0.95 -> 0.12
88% sccache hit rate windows2012-32-noopt debug taskcluster-c4.4xlarge 0.98 -> 0.12
87% sccache hit rate windows2012-64 debug fuzzing taskcluster-c5.4xlarge 0.91 -> 0.11
87% sccache hit rate windows2012-32 opt taskcluster-c5.4xlarge 0.91 -> 0.11
87% sccache hit rate windows2012-32 debug taskcluster-c4.4xlarge 0.97 -> 0.12
87% sccache hit rate windows2012-32 opt taskcluster-c4.4xlarge 0.94 -> 0.12
87% sccache hit rate windows2012-32 debug taskcluster-c5.4xlarge 0.91 -> 0.12
55% build times windows2012-64 debug fuzzing taskcluster-c4.4xlarge 2,197.49 -> 3,411.59
52% build times windows2012-32 debug taskcluster-c4.4xlarge 2,208.76 -> 3,357.08
50% build times windows2012-64-noopt debug taskcluster-c4.4xlarge 1,941.75 -> 2,910.07
42% build times windows2012-64 debug taskcluster-c4.4xlarge 2,377.70 -> 3,371.44
40% build times windows2012-64 asan debug taskcluster-c4.4xlarge 2,605.13 -> 3,643.90
34% build times windows2012-64 debug fuzzing taskcluster-c5.4xlarge 1,971.21 -> 2,648.50
34% build times windows2012-64 debug taskcluster-c5.4xlarge 1,942.80 -> 2,598.80
34% build times windows2012-64 asan debug taskcluster-c5.4xlarge 2,318.17 -> 3,097.40
32% build times windows2012-64-noopt debug taskcluster-c5.4xlarge 1,698.80 -> 2,248.66
29% build times windows2012-32 debug taskcluster-c5.4xlarge 1,953.24 -> 2,516.08
28% build times windows2012-64 asan fuzzing-asan taskcluster-c4.4xlarge 2,808.47 -> 3,583.06
27% build times windows2012-64 asan opt taskcluster-c4.4xlarge 2,784.78 -> 3,538.72
26% build times windows2012-32-noopt debug taskcluster-c4.4xlarge 1,982.18 -> 2,506.65
26% build times windows2012-64 opt taskcluster-c4.4xlarge 2,754.42 -> 3,465.85
22% build times windows2012-64 asan opt taskcluster-c5.4xlarge 2,480.55 -> 3,024.13
22% build times windows2012-32 opt taskcluster-c4.4xlarge 2,816.38 -> 3,425.04
22% build times windows2012-64 asan fuzzing-asan taskcluster-c5.4xlarge 2,488.03 -> 3,025.52
20% build times windows2012-32-noopt debug taskcluster-c5.4xlarge 1,650.96 -> 1,977.10
18% build times windows2012-64 opt taskcluster-c5.4xlarge 2,547.22 -> 3,006.99
16% build times windows2012-32 opt taskcluster-c5.4xlarge 2,480.57 -> 2,884.82

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=18863

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Ionuț Goldan [:igoldan]]

If there are any blocked bugs which aren't related to this issue, do remove them from the list.

Comment 2

[(Away)]

If I recall correctly, we record data somewhere about reasons for sccache misses. Can anybody remind me where to find this information?

Comment 3

[(Away)]

OK, thanks to help from ted and mshal, I believe this is because clang-plugin.dll is included in each compilation's hash, and that DLL is not build deterministically (it uses a real timestamp). Patch incoming.

Comment 4

[(Away)]

Attachment: Bug 1522771: Use -brepro to build clang-plugin.dll deterministically.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0d3146dfec52
Use -brepro to build clang-plugin.dll deterministically. r=mshal

Comment 6

[(Away)]

\o/ sccache hit rate debug taskcluster-c5.4xlarge: 0.99

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&searchStr=win%2Cbuild&revision=684d038b8eead6588ed32b16621a1b75138783d3&selectedJob=224056432

By the way, it's really cool that we track sccache hit rates. That was a very useful pointer to the root cause of this. Figuring it out from build time alone would have been a pain.

Comment 7

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/0d3146dfec52

Comment 8

[Ionuț Goldan [:igoldan]]

I confirm this got fixed!