Closed Bug 1603136 Opened 6 years ago Closed 6 years ago

sccache and build times performance regresssions on multiple platforms.] 9 to 11 of dec 2019

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: Bebe, Unassigned)

References

(Regression)

Details

(Keywords: perf-alert, regression)

Build metrics performance alerts where triggered between 9 - 11 of dec 2019

The suspected culprit is Bug 1601701 - Bump mingw-w64 and add -lssp support

Notes from slack:

Bebe 3:32 PM we are seeing a lot of build metrics regressions on 9 and 10 of dec
anyone has any ideas why ?
https://treeherder.mozilla.org/perf.html#/alerts?id=24317
the commits of the regression don't seem to be related to the regression

froydnj 4:56 PM @Bebe probably https://bugzilla.mozilla.org/show_bug.cgi?id=1601701 , which tweaked building clang, thereby forcing toolchains to be rebuilt
bugzilla.mozilla.orgbugzilla.mozilla.org

Bebe 5:14 PM @froydnj would that rebuild happen in the same commit or can generate different alerts in different commits
froydnj 5:15 PM @Bebe different alerts in different commits, if I understand your question
Bebe 5:15 PM oki
froydnj 5:16 PM the hit rate went back up after a while, so it's not a problem
Bebe 5:16 PM because we have a big list of alerts
https://treeherder.mozilla.org/perf.html#/alerts?hideDwnToInv=0&framework=2&page=1
improv and regressions
froydnj 5:26 PM presumably the regressions are from the toolchains rolling out to builds, and the improvements are from sccache's cache being re-seeded with the results of the new toolchains

== Change summary for alert #24314 (as of Mon, 09 Dec 2019 14:33:27 GMT) ==

Regressions:

14% sccache hit rate android-5-0-aarch64 opt 0.94 -> 0.81
14% sccache hit rate linux64 asan asan-fuzzing 0.94 -> 0.81
13% sccache hit rate linux64 opt base-toolchains 0.94 -> 0.82

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24314

== Change summary for alert #24317 (as of Mon, 09 Dec 2019 16:25:51 GMT) ==

Improvements:

20% sccache hit rate linux64 opt base-toolchains 0.82 -> 0.98
20% sccache hit rate android-5-0-aarch64 opt 0.82 -> 0.98
19% sccache hit rate linux64 asan asan-fuzzing 0.82 -> 0.97
18% sccache hit rate android-4-2-x86 debug 0.82 -> 0.97
18% sccache hit rate android-4-0-armv7-api16 debug 0.84 -> 0.99
14% sccache hit rate linux64 tsan opt 0.87 -> 0.99
14% sccache hit rate linux64 opt 0.87 -> 0.99
13% sccache hit rate android-5-0-x86_64 debug 0.87 -> 0.99
13% sccache hit rate linux64 opt base-toolchains 0.88 -> 0.99
12% sccache hit rate linux64 debug base-toolchains 0.87 -> 0.98
12% sccache hit rate linux64 debug fuzzing 0.87 -> 0.98
12% sccache hit rate osx-cross debug fuzzing 0.89 -> 0.99
12% sccache hit rate osx-cross debug 0.89 -> 0.99

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24317

== Change summary for alert #24322 (as of Mon, 09 Dec 2019 18:54:34 GMT) ==

Regressions:

181675% sccache cache_write_errors linux64 asan opt 0.33 -> 605.92
10075% sccache cache_write_errors linux64 debug 6.79 -> 691.08
5827% sccache cache_write_errors android-4-2-x86 debug 12.25 -> 726.08
4009% sccache cache_write_errors linux64-aarch64 opt 18.25 -> 749.92
307% sccache cache_write_errors linux64 asan opt 807.75 -> 3,285.58
289% sccache cache_write_errors linux64 asan asan-fuzzing 862.08 -> 3,349.92
91% build times osx-cross asan asan-fuzzing taskcluster-m5.4xlarge 820.93 -> 1,570.55
88% sccache hit rate windows2012-aarch64 debug aarch64 0.95 -> 0.12
85% sccache hit rate linux64 asan asan-fuzzing 0.77 -> 0.11
82% sccache hit rate linux64 asan opt 0.79 -> 0.14
72% build times android-5-0-x86_64 debug taskcluster-m5.4xlarge 986.86 -> 1,697.96
68% build times linux64 tsan opt taskcluster-c5d.4xlarge 868.85 -> 1,462.84
43% build times windows2012-32 debug taskcluster-c4.4xlarge 2,610.21 -> 3,722.39
39% build times windows2012-32 opt taskcluster-c4.4xlarge 2,682.65 -> 3,721.54
39% build times windows2012-64 debug taskcluster-c4.4xlarge 2,654.41 -> 3,679.89
38% build times windows2012-64-noopt debug taskcluster-c4.4xlarge 2,380.90 -> 3,288.54
36% build times windows2012-64 opt taskcluster-c4.4xlarge 2,719.98 -> 3,688.67
33% build times windows2012-32 opt taskcluster-c5.4xlarge 2,304.37 -> 3,075.15
30% build times windows2012-64 asan opt taskcluster-c5.4xlarge 2,397.73 -> 3,122.66
29% build times windows2012-32-noopt debug taskcluster-c4.4xlarge 2,257.00 -> 2,921.97
29% build times windows2012-aarch64 debug aarch64 taskcluster-c4.4xlarge 2,293.96 -> 2,958.40
27% build times windows2012-64 debug taskcluster-c5.4xlarge 2,158.51 -> 2,748.76
25% build times android-4-2-x86 opt taskcluster-m5.4xlarge 1,604.89 -> 2,001.17
22% build times windows2012-aarch64 debug aarch64 taskcluster-c5.4xlarge 1,898.74 -> 2,309.77
20% build times windows2012-32-noopt debug taskcluster-c5.4xlarge 1,978.95 -> 2,379.22
20% sccache hit rate linux64 debug fuzzing 0.99 -> 0.80
20% sccache hit rate android-5-0-aarch64 opt 0.98 -> 0.78
20% sccache hit rate linux64 opt base-toolchains 0.99 -> 0.80
19% sccache hit rate android-5-0-x86_64 opt 0.98 -> 0.79
19% sccache hit rate linux64 debug base-toolchains 0.99 -> 0.81
19% sccache hit rate android-4-0-armv7-api16 debug 0.97 -> 0.79
18% sccache hit rate android-4-2-x86 opt 0.97 -> 0.79
18% sccache hit rate osx-cross asan asan-fuzzing 0.97 -> 0.79
18% sccache hit rate android-5-0-x86_64 debug 0.97 -> 0.79
18% sccache hit rate linux32 debug 0.97 -> 0.80
17% sccache hit rate android-4-2-x86 debug 0.98 -> 0.81
17% sccache hit rate osx-cross debug 0.98 -> 0.81
16% sccache hit rate linux64 tsan opt 0.97 -> 0.81
16% sccache hit rate linux64-aarch64 opt 0.98 -> 0.82
15% sccache hit rate linux64-noopt debug 0.98 -> 0.83
15% sccache hit rate linux64 asan asan-fuzzing 0.97 -> 0.82
15% sccache hit rate linux64 asan debug 0.97 -> 0.83
5% sccache hit rate android-5-0-aarch64 opt 0.78 -> 0.75
4% sccache hit rate android-4-0-armv7-api16 debug 0.79 -> 0.75
4% sccache hit rate android-4-2-x86 opt 0.79 -> 0.75
4% sccache hit rate android-5-0-x86_64 debug 0.79 -> 0.76
4% sccache hit rate android-5-0-x86_64 opt 0.79 -> 0.76

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24322

== Change summary for alert #24375 (as of Tue, 10 Dec 2019 08:08:49 GMT) ==

Improvements:

1028% sccache hit rate windows2012-aarch64 debug aarch64 0.07 -> 0.84
1027% sccache hit rate windows2012-aarch64 opt aarch64 0.07 -> 0.84
742% sccache hit rate windows2012-64-noopt debug 0.11 -> 0.94
720% sccache hit rate windows2012-32-noopt debug 0.11 -> 0.92
704% sccache hit rate windows2012-64 asan opt 0.11 -> 0.92
656% sccache hit rate windows2012-32 opt 0.11 -> 0.85
571% sccache hit rate windows2012-64 opt 0.13 -> 0.84
100% sccache cache_write_errors android-4-0-armv7-api16 debug 798.33 -> 0.00
100% sccache cache_write_errors android-4-0-armv7-api16 opt 855.42 -> 0.00
100% sccache cache_write_errors android-4-2-x86 debug 879.00 -> 0.00
100% sccache cache_write_errors android-4-2-x86 opt 868.83 -> 0.00
100% sccache cache_write_errors android-5-0-aarch64 debug 861.75 -> 0.00
100% sccache cache_write_errors android-5-0-aarch64 opt 868.83 -> 0.00
100% sccache cache_write_errors android-5-0-x86_64 asan 863.33 -> 0.00
100% sccache cache_write_errors android-5-0-x86_64 debug 874.83 -> 0.00
100% sccache cache_write_errors android-5-0-x86_64 opt 867.17 -> 0.00
100% sccache cache_write_errors linux32 debug 927.83 -> 0.00
100% sccache cache_write_errors linux64 asan asan-fuzzing 3,346.17 -> 0.08
100% sccache cache_write_errors linux64 asan debug 912.25 -> 0.00
100% sccache cache_write_errors linux64 asan opt 3,283.08 -> 0.00
100% sccache cache_write_errors linux64 debug 923.25 -> 0.00
100% sccache cache_write_errors linux64 debug base-toolchains 915.83 -> 0.00
100% sccache cache_write_errors linux64 debug fuzzing 855.75 -> 0.00
100% sccache cache_write_errors linux64 opt 914.33 -> 0.00
100% sccache cache_write_errors linux64 opt base-toolchains 919.83 -> 0.00
100% sccache cache_write_errors linux64 opt valgrind 909.08 -> 0.00
100% sccache cache_write_errors linux64 tsan opt 911.33 -> 0.00
100% sccache cache_write_errors linux64 tsan tsan-fuzzing 932.75 -> 0.00
100% sccache cache_write_errors linux64-aarch64 opt 839.08 -> 0.00
100% sccache cache_write_errors osx-cross asan asan-fuzzing 915.00 -> 0.00
100% sccache cache_write_errors osx-cross debug 893.67 -> 0.00
100% sccache cache_write_errors osx-cross debug fuzzing 918.33 -> 0.00
100% sccache cache_write_errors osx-cross-noopt debug 812.33 -> 0.00
100% sccache cache_write_errors windows-mingw32 all 64 clang debug 3,322.92 -> 0.00
100% sccache cache_write_errors windows2012-32 debug 3,906.67 -> 0.00
100% sccache cache_write_errors windows2012-32 opt 3,880.33 -> 0.00
100% sccache cache_write_errors windows2012-32-noopt debug 3,912.92 -> 0.08
100% sccache cache_write_errors windows2012-64 asan debug 3,845.92 -> 0.00
100% sccache cache_write_errors windows2012-64 asan fuzzing-asan 3,830.67 -> 0.00
100% sccache cache_write_errors windows2012-64 asan opt 3,818.83 -> 0.00
100% sccache cache_write_errors windows2012-64 debug 3,900.83 -> 0.00
100% sccache cache_write_errors windows2012-64 debug fuzzing 3,912.58 -> 0.08
100% sccache cache_write_errors windows2012-64 opt 3,875.92 -> 0.00
100% sccache cache_write_errors windows2012-64-noopt debug 3,899.58 -> 0.00
100% sccache cache_write_errors windows2012-aarch64 debug aarch64 3,745.75 -> 0.08
100% sccache cache_write_errors windows2012-aarch64 opt aarch64 3,743.17 -> 0.00
100% sccache cache_write_errors linux64-noopt debug 922.00 -> 0.08
100% sccache cache_write_errors windows-mingw32 all 64 clang opt 1,027.71 -> 0.08
42% build times windows-mingw32 all 32 clang debug taskcluster-c5.4xlarge 1,790.71 -> 1,045.90
33% build times windows2012-64-noopt debug taskcluster-c4.4xlarge 3,457.30 -> 2,333.31
30% build times windows2012-64 asan opt taskcluster-c4.4xlarge 3,924.34 -> 2,751.32
26% build times windows2012-aarch64 debug aarch64 taskcluster-c4.4xlarge 3,146.80 -> 2,333.39
24% sccache hit rate linux64 asan debug 0.77 -> 0.95
24% sccache hit rate linux64 opt 0.77 -> 0.96
17% build times windows2012-aarch64 opt aarch64 taskcluster-c4.4xlarge 4,100.78 -> 3,408.93

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24375

== Change summary for alert #24392 (as of Tue, 10 Dec 2019 12:14:50 GMT) ==

Improvements:

57% build times linux64 tsan opt taskcluster-c5d.4xlarge 1,494.21 -> 646.85
52% build times osx-cross debug fuzzing taskcluster-c5d.4xlarge 2,029.06 -> 977.16
30% build times android-5-0-aarch64 opt taskcluster-c5.4xlarge 1,853.28 -> 1,293.57
28% build times windows2012-32-noopt debug taskcluster-c4.4xlarge 3,090.89 -> 2,224.78
27% build times windows2012-32 debug taskcluster-c5.4xlarge 2,749.88 -> 2,005.14
26% build times windows2012-64 debug fuzzing taskcluster-c5.4xlarge 2,868.36 -> 2,136.45
25% build times windows2012-64 asan debug taskcluster-c5.4xlarge 3,454.76 -> 2,592.37
23% sccache hit rate android-5-0-x86_64 asan 0.81 -> 1.00
19% build times android-5-0-x86_64 asan taskcluster-m5.4xlarge 2,637.99 -> 2,141.05

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24392

== Change summary for alert #24414 (as of Tue, 10 Dec 2019 15:17:19 GMT) ==

Improvements:

55% build times linux64 tsan opt taskcluster-m5.4xlarge 1,547.26 -> 703.46
50% build times android-5-0-x86_64 debug taskcluster-m5.4xlarge 1,745.37 -> 866.04
48% build times linux64 opt base-toolchains taskcluster-m5.4xlarge 2,385.87 -> 1,247.04
48% build times osx-cross debug fuzzing taskcluster-m5.4xlarge 2,252.25 -> 1,180.62
47% build times linux64-noopt debug taskcluster-c5d.4xlarge 1,619.55 -> 856.27
44% build times linux64 asan debug taskcluster-m5.4xlarge 1,692.68 -> 954.64
42% build times android-5-0-x86_64 debug taskcluster-c5.4xlarge 1,598.29 -> 925.37

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24414

tjr can you take a look over the alerts and confirm they have the same culprit.

Flags: needinfo?(tom)
Blocks: 1600879
Keywords: perf-alert, regression
Regressed by: 1601701

Clicking through some of the regressions/improvements here it looks like there was an infra issue with sccache for a brief period but that recovered fully. I don't think there's anything else to investigate here, unless I've missed something.

Flags: needinfo?(tom)

Has everything recovered at this point or are there still lingering regressions?

Flags: needinfo?(fstrugariu)

we are back to normal.

Thanks everyone for the updates.

Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(fstrugariu)
Resolution: --- → WONTFIX
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.