Closed Bug 1617131 Opened 4 years ago Closed 4 years ago

21.76 - 89.83% build times (android-*, linux64, linux64-aarch64, osx-cross, windows2012-32, windows2012-64, windows2012-aarch64) regression on push 7302618003983e94d9bc5b91930e96aa0deb7ff4 (Fri February 21 2020)

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr68 unaffected, firefox73 unaffected, firefox74 unaffected, firefox75 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla75
Tracking Flags:
Tracking Status
firefox-esr68 --- unaffected
firefox73 --- unaffected
firefox74 --- unaffected
firefox75 --- fixed

People

(Reporter: marauder, Assigned: glandium)

References

(Regression)

Details

(Keywords: perf-alert, regression)

Attachments

(2 files)

Bug 1617131 - Sort the list of third party paths.
47 bytes, text/x-phabricator-request
Details | Review
Improvment alert
10.67 KB, text/plain
Details

21.76 - 89.83% build times / sccache hit rate (android-4-0-armv7-api16, android-4-2-x86, android-5-0-aarch64, android-5-0-x86_64, linux64, linux64-aarch64, osx-cross, windows2012-32, windows2012-64, windows2012-aarch64) regression on push 7302618003983e94d9bc5b91930e96aa0deb7ff4 (Fri February 21 2020)

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=7302618003983e94d9bc5b91930e96aa0deb7ff4

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:
90% sccache hit rate windows2012-aarch64 opt aarch64 0.93 -> 0.10
90% sccache hit rate windows2012-aarch64 debug aarch64 0.93 -> 0.09
88% sccache hit rate android-5-0-x86_64 opt gcp 0.96 -> 0.11
88% sccache hit rate android-4-2-x86 opt gcp 0.95 -> 0.11
88% sccache hit rate android-4-0-armv7-api16 debug gcp 0.95 -> 0.12
88% sccache hit rate android-5-0-aarch64 debug gcp 0.95 -> 0.12
88% sccache hit rate android-5-0-x86_64 debug 0.92 -> 0.11
88% sccache hit rate android-4-0-armv7-api16 opt gcp 0.95 -> 0.12
88% sccache hit rate android-4-2-x86 debug 0.92 -> 0.11
88% sccache hit rate android-5-0-aarch64 opt gcp 0.94 -> 0.12
88% sccache hit rate android-5-0-x86_64 opt 0.91 -> 0.11
88% sccache hit rate android-4-0-armv7-api16 debug 0.92 -> 0.11
87% sccache hit rate android-4-0-armv7-api16 opt 0.92 -> 0.12
87% sccache hit rate android-4-2-x86 opt 0.91 -> 0.11
87% sccache hit rate android-5-0-aarch64 opt 0.92 -> 0.12
87% sccache hit rate android-5-0-aarch64 debug 0.91 -> 0.12
87% sccache hit rate linux64 asan asan-fuzzing 0.93 -> 0.12
86% sccache hit rate linux64 tsan opt 0.87 -> 0.12
86% sccache hit rate osx-cross asan asan-fuzzing 0.92 -> 0.13
86% sccache hit rate linux64 tsan tsan-fuzzing 0.88 -> 0.12
86% sccache hit rate windows2012-32 debug 0.94 -> 0.13
86% sccache hit rate windows2012-64 debug fuzzing 0.94 -> 0.13
86% sccache hit rate windows2012-64 opt 0.95 -> 0.13
86% sccache hit rate osx-cross debug fuzzing 0.94 -> 0.13
86% sccache hit rate windows2012-64 debug 0.94 -> 0.13
86% sccache hit rate windows2012-32 opt 0.94 -> 0.13
86% sccache hit rate windows2012-64 asan debug 0.95 -> 0.13
86% sccache hit rate windows2012-64 asan opt 0.94 -> 0.14
86% sccache hit rate windows2012-64 asan fuzzing-asan 0.93 -> 0.13
84% sccache hit rate linux64 debug gcp 0.96 -> 0.15
84% sccache hit rate linux64 asan debug 0.92 -> 0.15
84% sccache hit rate linux64 debug 0.93 -> 0.15
84% sccache hit rate linux64 opt 0.93 -> 0.15
84% sccache hit rate linux64 opt valgrind 0.93 -> 0.15
84% sccache hit rate linux64 asan opt 0.91 -> 0.15
84% sccache hit rate osx-cross debug gcp 0.96 -> 0.16
83% sccache hit rate linux64-aarch64 opt 0.91 -> 0.16
83% sccache hit rate osx-cross debug 0.92 -> 0.16
74% sccache hit rate windows2012-aarch64 opt aarch64 0.93 -> 0.24
72% sccache hit rate android-5-0-x86_64 debug 0.94 -> 0.26
72% sccache hit rate android-5-0-x86_64 opt 0.93 -> 0.26
71% sccache hit rate windows2012-64 debug 0.92 -> 0.27
71% sccache hit rate windows2012-32 debug 0.92 -> 0.27
70% sccache hit rate windows2012-32 opt 0.92 -> 0.27
70% sccache hit rate windows2012-64 asan opt 0.91 -> 0.27
69% sccache hit rate linux64 debug 0.93 -> 0.29
69% sccache hit rate linux64 asan opt 0.93 -> 0.29
68% sccache hit rate osx-cross debug 0.92 -> 0.29
64% build times android-4-0-armv7-api16 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 912.41 -> 1,495.74
62% build times android-4-0-armv7-api16 debug taskcluster-m5.4xlarge 1,218.82 -> 1,972.35
62% build times android-5-0-x86_64 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 937.56 -> 1,517.09
52% build times android-4-0-armv7-api16 debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,132.93 -> 1,717.24
51% build times osx-cross debug fuzzing taskcluster-c5d.4xlarge 1,465.88 -> 2,206.63
49% build times android-5-0-aarch64 opt taskcluster-c5d.4xlarge 988.75 -> 1,469.88
46% build times android-5-0-aarch64 debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,149.05 -> 1,678.77
44% build times osx-cross debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,445.08 -> 2,083.81
43% build times linux64 debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,320.97 -> 1,891.88
39% build times windows2012-32 opt taskcluster-c4.4xlarge 2,667.82 -> 3,699.44
36% build times windows2012-64 asan opt taskcluster-c4.4xlarge 3,030.83 -> 4,108.31
35% build times windows2012-aarch64 debug aarch64 taskcluster-c4.4xlarge 2,610.77 -> 3,534.56
33% build times windows2012-32 debug taskcluster-c5.4xlarge 2,179.46 -> 2,909.20
33% build times windows2012-64 debug taskcluster-c4.4xlarge 2,846.68 -> 3,788.60
33% build times windows2012-64 opt taskcluster-c5.4xlarge 2,107.54 -> 2,800.22
32% build times windows2012-64 asan fuzzing-asan taskcluster-c5.4xlarge 2,529.67 -> 3,344.00
32% build times windows2012-64 opt taskcluster-c4.4xlarge 2,805.29 -> 3,693.18
31% build times windows2012-64 debug fuzzing taskcluster-c4.4xlarge 2,912.40 -> 3,801.86
28% build times windows2012-32 debug taskcluster-c4.4xlarge 2,960.57 -> 3,793.11
27% build times windows2012-32 opt taskcluster-c5.4xlarge 2,241.13 -> 2,850.12
26% build times windows2012-64 asan debug taskcluster-c5.4xlarge 2,916.78 -> 3,680.35
22% build times windows2012-aarch64 debug aarch64 taskcluster-c5.4xlarge 2,104.71 -> 2,562.77

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=25030

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Blocks: 1614561
status-firefox73: --- → unaffected
status-firefox74: --- → unaffected
status-firefox75: --- → affected
status-firefox-esr68: --- → unaffected
Component: Performance → General
Flags: needinfo?(mshal)
Product: Testing → Firefox Build System
Regressed by: 1616630
Target Milestone: --- → mozilla75
Version: Version 3 → unspecified
Has Regression Range: --- → yes

Smells like something is not being generated deterministically with python3, that is included in most places, killing cache hits.

Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Flags: needinfo?(mshal)
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/7d9b567c6bf4
Sort the list of third party paths. r=sylvestre

https://hg.mozilla.org/mozilla-central/rev/7d9b567c6bf4

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox75: affected → fixed
Resolution: --- → FIXED

== Change summary for alert #25059 (as of Fri, 21 Feb 2020 13:45:04 GMT) ==

Regressions:

609% sccache cache_write_errors linux64 debug gcp 1.38 -> 9.75

Improvements:

903% sccache hit rate windows2012-aarch64 debug aarch64 0.10 -> 0.95
824% sccache hit rate windows2012-aarch64 opt aarch64 0.10 -> 0.88
731% sccache hit rate android-4-0-armv7-api16 debug gcp 0.12 -> 0.97
725% sccache hit rate android-4-2-x86 opt gcp 0.12 -> 0.98
716% sccache hit rate android-5-0-aarch64 debug gcp 0.12 -> 0.94
707% sccache hit rate android-5-0-x86_64 opt gcp 0.12 -> 0.96
692% sccache hit rate android-5-0-aarch64 opt gcp 0.12 -> 0.95
686% sccache hit rate linux64 tsan opt 0.12 -> 0.92
658% sccache hit rate android-4-0-armv7-api16 opt gcp 0.13 -> 0.95
652% sccache hit rate android-5-0-aarch64 opt 0.12 -> 0.87
645% sccache hit rate windows2012-64 debug 0.13 -> 0.97
641% sccache hit rate windows2012-64 opt 0.13 -> 0.97
638% sccache hit rate android-4-0-armv7-api16 opt 0.12 -> 0.88
612% sccache hit rate osx-cross asan asan-fuzzing 0.13 -> 0.90
612% sccache hit rate windows2012-64 asan debug 0.13 -> 0.96
599% sccache hit rate osx-cross debug fuzzing 0.13 -> 0.91
599% sccache hit rate linux64 tsan tsan-fuzzing 0.12 -> 0.86
580% sccache hit rate windows2012-64 debug fuzzing 0.13 -> 0.88
563% sccache hit rate windows2012-64 asan opt 0.14 -> 0.90
554% sccache hit rate linux64 debug gcp 0.15 -> 0.98
550% sccache hit rate android-4-2-x86 debug 0.14 -> 0.89
539% sccache hit rate linux64 debug 0.15 -> 0.96
532% sccache hit rate android-5-0-x86_64 debug 0.14 -> 0.88
526% sccache hit rate android-5-0-x86_64 opt 0.14 -> 0.89
520% sccache hit rate android-4-0-armv7-api16 debug 0.14 -> 0.89
509% sccache hit rate linux64 asan opt 0.15 -> 0.91
508% sccache hit rate linux64 opt valgrind 0.16 -> 0.96
502% sccache hit rate linux64 asan debug 0.15 -> 0.89
502% sccache hit rate linux64 asan asan-fuzzing 0.16 -> 0.96
501% sccache hit rate windows2012-32 debug 0.15 -> 0.89
490% sccache hit rate osx-cross debug gcp 0.16 -> 0.95
486% sccache hit rate windows2012-64 asan fuzzing-asan 0.15 -> 0.86
486% sccache hit rate osx-cross debug 0.16 -> 0.95
480% sccache hit rate android-4-2-x86 opt 0.16 -> 0.95
462% sccache hit rate linux64-aarch64 opt 0.16 -> 0.88
411% sccache hit rate linux64 opt 0.18 -> 0.92
364% sccache hit rate android-5-0-aarch64 debug 0.21 -> 0.97
358% sccache hit rate windows2012-32 opt 0.19 -> 0.89
56% build times linux64 asan opt taskcluster-c5d.4xlarge 1,751.77 -> 778.61
55% build times linux64 tsan opt taskcluster-c5.4xlarge 1,221.43 -> 555.70
54% build times linux64 opt taskcluster-c5d.4xlarge 1,685.46 -> 778.01
52% build times osx-cross debug taskcluster-c5d.4xlarge 2,085.27 -> 994.73
50% build times android-4-0-armv7-api16 opt taskcluster-c5d.4xlarge 1,531.31 -> 763.43
50% build times android-5-0-aarch64 opt taskcluster-c5d.4xlarge 1,483.41 -> 742.24
48% build times osx-cross debug fuzzing taskcluster-c5d.4xlarge 2,144.01 -> 1,123.46
47% build times android-5-0-x86_64 opt taskcluster-c5d.4xlarge 1,598.10 -> 848.23
47% build times android-5-0-x86_64 debug taskcluster-m5.4xlarge 1,724.62 -> 919.22
45% build times android-5-0-x86_64 opt taskcluster-m5.4xlarge 1,581.63 -> 866.55
45% build times android-4-0-armv7-api16 opt taskcluster-c5.4xlarge 1,488.32 -> 817.64
43% build times android-5-0-aarch64 debug taskcluster-c5.4xlarge 1,936.29 -> 1,100.60
42% build times android-4-2-x86 debug taskcluster-c5d.4xlarge 1,554.78 -> 903.56
42% build times android-4-0-armv7-api16 debug taskcluster-c5d.4xlarge 1,800.44 -> 1,051.10
41% build times android-5-0-aarch64 debug taskcluster-c5d.4xlarge 1,499.10 -> 877.52
39% build times linux64 debug taskcluster-c5.4xlarge 1,865.11 -> 1,140.70
37% build times android-5-0-aarch64 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,360.16 -> 856.54
35% build times android-4-2-x86 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,400.77 -> 906.02
33% build times android-4-0-armv7-api16 debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,665.40 -> 1,114.62
32% build times windows2012-64 asan opt taskcluster-c4.4xlarge 4,109.65 -> 2,786.31
32% build times android-5-0-aarch64 debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,679.42 -> 1,141.94
31% build times windows2012-64 debug taskcluster-c4.4xlarge 3,787.21 -> 2,595.53
31% build times linux64 debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,885.63 -> 1,294.01
31% build times windows2012-32 opt taskcluster-c4.4xlarge 3,661.17 -> 2,538.79
30% build times android-5-0-x86_64 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 1,426.13 -> 993.42
30% build times windows2012-aarch64 debug aarch64 taskcluster-c4.4xlarge 3,541.05 -> 2,477.59
29% build times windows2012-64 opt taskcluster-c4.4xlarge 3,682.70 -> 2,598.98
29% build times windows2012-64 debug fuzzing taskcluster-c4.4xlarge 3,794.08 -> 2,684.86
29% build times osx-cross debug gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 2,022.95 -> 1,436.58
28% build times windows2012-64 asan fuzzing-asan taskcluster-c4.4xlarge 3,867.79 -> 2,776.19
26% build times windows2012-64 asan fuzzing-asan taskcluster-c5.4xlarge 3,325.28 -> 2,451.41
26% build times windows2012-64 opt taskcluster-c5.4xlarge 2,801.04 -> 2,070.56
26% build times windows2012-32 debug taskcluster-c4.4xlarge 3,791.37 -> 2,808.92
25% build times windows2012-32 debug taskcluster-c5.4xlarge 2,813.79 -> 2,104.15
25% build times windows2012-aarch64 debug aarch64 taskcluster-c5.4xlarge 2,595.41 -> 1,948.25
24% build times windows2012-64 asan debug taskcluster-c5.4xlarge 3,661.81 -> 2,785.05
24% build times windows2012-32 opt taskcluster-c5.4xlarge 2,898.82 -> 2,211.46
23% build times windows2012-64 asan debug taskcluster-c4.4xlarge 4,115.29 -> 3,160.90
21% build times windows2012-aarch64 opt aarch64 taskcluster-c4.4xlarge 4,210.38 -> 3,341.86
14% build times windows2012-aarch64 opt aarch64 taskcluster-c5.4xlarge 3,271.43 -> 2,799.19

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25059

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: