16.3 - 88.63% build times / sccache hit rate (windows2012-32, windows2012-32-noopt, windows2012-64, windows2012-64-noopt) regression on push 4e34cf5e62ab77e5821d94de0fb1d6b92ec1e319 (Thu Jan 24 2019)

VERIFIED FIXED in Firefox 66

Status

defect
VERIFIED FIXED
6 months ago
6 months ago

People

(Reporter: igoldan, Assigned: dmajor)

Tracking

(Blocks 1 bug, {regression})

Trunk
mozilla66
All
Windows
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 unaffected, firefox64 unaffected, firefox65 unaffected, firefox66 fixed)

Details

Attachments

(1 attachment)

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=4e34cf5e62ab77e5821d94de0fb1d6b92ec1e319

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

89% sccache hit rate windows2012-64 asan opt taskcluster-c5.4xlarge 0.95 -> 0.11
88% sccache hit rate windows2012-64-noopt debug taskcluster-c4.4xlarge 0.97 -> 0.11
88% sccache hit rate windows2012-64 asan debug taskcluster-c4.4xlarge 0.95 -> 0.11
88% sccache hit rate windows2012-64-noopt debug taskcluster-c5.4xlarge 0.92 -> 0.11
88% sccache hit rate windows2012-64 debug fuzzing taskcluster-c4.4xlarge 0.98 -> 0.12
88% sccache hit rate windows2012-64 asan debug taskcluster-c5.4xlarge 0.94 -> 0.11
88% sccache hit rate windows2012-32-noopt debug taskcluster-c5.4xlarge 0.93 -> 0.11
88% sccache hit rate windows2012-64 opt taskcluster-c5.4xlarge 0.93 -> 0.11
88% sccache hit rate windows2012-64 asan fuzzing-asan taskcluster-c4.4xlarge 0.95 -> 0.11
88% sccache hit rate windows2012-64 opt taskcluster-c4.4xlarge 0.96 -> 0.12
88% sccache hit rate windows2012-64 asan fuzzing-asan taskcluster-c5.4xlarge 0.92 -> 0.11
88% sccache hit rate windows2012-64 debug taskcluster-c5.4xlarge 0.91 -> 0.11
88% sccache hit rate windows2012-64 debug taskcluster-c4.4xlarge 0.94 -> 0.12
88% sccache hit rate windows2012-64 asan opt taskcluster-c4.4xlarge 0.95 -> 0.12
88% sccache hit rate windows2012-32-noopt debug taskcluster-c4.4xlarge 0.98 -> 0.12
87% sccache hit rate windows2012-64 debug fuzzing taskcluster-c5.4xlarge 0.91 -> 0.11
87% sccache hit rate windows2012-32 opt taskcluster-c5.4xlarge 0.91 -> 0.11
87% sccache hit rate windows2012-32 debug taskcluster-c4.4xlarge 0.97 -> 0.12
87% sccache hit rate windows2012-32 opt taskcluster-c4.4xlarge 0.94 -> 0.12
87% sccache hit rate windows2012-32 debug taskcluster-c5.4xlarge 0.91 -> 0.12
55% build times windows2012-64 debug fuzzing taskcluster-c4.4xlarge 2,197.49 -> 3,411.59
52% build times windows2012-32 debug taskcluster-c4.4xlarge 2,208.76 -> 3,357.08
50% build times windows2012-64-noopt debug taskcluster-c4.4xlarge 1,941.75 -> 2,910.07
42% build times windows2012-64 debug taskcluster-c4.4xlarge 2,377.70 -> 3,371.44
40% build times windows2012-64 asan debug taskcluster-c4.4xlarge 2,605.13 -> 3,643.90
34% build times windows2012-64 debug fuzzing taskcluster-c5.4xlarge 1,971.21 -> 2,648.50
34% build times windows2012-64 debug taskcluster-c5.4xlarge 1,942.80 -> 2,598.80
34% build times windows2012-64 asan debug taskcluster-c5.4xlarge 2,318.17 -> 3,097.40
32% build times windows2012-64-noopt debug taskcluster-c5.4xlarge 1,698.80 -> 2,248.66
29% build times windows2012-32 debug taskcluster-c5.4xlarge 1,953.24 -> 2,516.08
28% build times windows2012-64 asan fuzzing-asan taskcluster-c4.4xlarge 2,808.47 -> 3,583.06
27% build times windows2012-64 asan opt taskcluster-c4.4xlarge 2,784.78 -> 3,538.72
26% build times windows2012-32-noopt debug taskcluster-c4.4xlarge 1,982.18 -> 2,506.65
26% build times windows2012-64 opt taskcluster-c4.4xlarge 2,754.42 -> 3,465.85
22% build times windows2012-64 asan opt taskcluster-c5.4xlarge 2,480.55 -> 3,024.13
22% build times windows2012-32 opt taskcluster-c4.4xlarge 2,816.38 -> 3,425.04
22% build times windows2012-64 asan fuzzing-asan taskcluster-c5.4xlarge 2,488.03 -> 3,025.52
20% build times windows2012-32-noopt debug taskcluster-c5.4xlarge 1,650.96 -> 1,977.10
18% build times windows2012-64 opt taskcluster-c5.4xlarge 2,547.22 -> 3,006.99
16% build times windows2012-32 opt taskcluster-c5.4xlarge 2,480.57 -> 2,884.82

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=18863

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Flags: needinfo?(dmajor)
Component: General → Toolchains
Product: Testing → Firefox Build System

If there are any blocked bugs which aren't related to this issue, do remove them from the list.

If I recall correctly, we record data somewhere about reasons for sccache misses. Can anybody remind me where to find this information?

Flags: needinfo?(dmajor)
No longer blocks: 1512822, 1522289

OK, thanks to help from ted and mshal, I believe this is because clang-plugin.dll is included in each compilation's hash, and that DLL is not build deterministically (it uses a real timestamp). Patch incoming.

Assignee: nobody → dmajor
Attachment #9039119 - Flags: review?(core-build-config-reviews)
Attachment #9039119 - Flags: review?(core-build-config-reviews) → review+
Pushed by dmajor@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0d3146dfec52
Use -brepro to build clang-plugin.dll deterministically. r=mshal

\o/ sccache hit rate debug taskcluster-c5.4xlarge: 0.99

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&searchStr=win%2Cbuild&revision=684d038b8eead6588ed32b16621a1b75138783d3&selectedJob=224056432

By the way, it's really cool that we track sccache hit rates. That was a very useful pointer to the root cause of this. Figuring it out from build time alone would have been a pain.

Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66

I confirm this got fixed!

Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.