1618720 - 23.91 - 91% build times / sccache hit rate (android-4-0-armv7-api16, android-4-2-x86, android-5-0-aarch64, android-5-0-x86_64, linux64, linux64-aarch64, osx-cross, etc.) regression on push f400f164b3947b4dd54089a36ea31cca2d72805b (Fri February 28 2020)

Status: RESOLVED FIXED

(Developer Infrastructure :: Source Code Analysis, defect)

Description

[Mike Hommey [:glandium]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=f400f164b3947b4dd54089a36ea31cca2d72805b

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

91% sccache hit rate windows2012-aarch64 opt aarch64 0.94 -> 0.08
91% sccache hit rate windows2012-aarch64 debug aarch64 0.92 -> 0.09
89% sccache hit rate android-5-0-x86_64 debug 0.96 -> 0.10
89% sccache hit rate android-5-0-x86_64 opt gcp 0.96 -> 0.10
89% sccache hit rate android-5-0-aarch64 debug gcp 0.97 -> 0.11
89% sccache hit rate android-4-2-x86 opt gcp 0.96 -> 0.10
89% sccache hit rate android-4-0-armv7-api16 opt gcp 0.96 -> 0.11
89% sccache hit rate android-4-0-armv7-api16 debug gcp 0.96 -> 0.10
89% sccache hit rate android-4-0-armv7-api16 debug 0.94 -> 0.10
89% sccache hit rate android-4-2-x86 debug 0.92 -> 0.10
89% sccache hit rate android-5-0-aarch64 opt gcp 0.96 -> 0.11
89% sccache hit rate android-4-2-x86 opt 0.94 -> 0.10
89% sccache hit rate android-5-0-aarch64 opt 0.94 -> 0.10
89% sccache hit rate android-5-0-x86_64 opt 0.93 -> 0.10
89% sccache hit rate android-5-0-aarch64 debug 0.94 -> 0.11
88% sccache hit rate android-4-0-armv7-api16 opt 0.91 -> 0.11
88% sccache hit rate linux64 tsan opt 0.91 -> 0.11
88% sccache hit rate linux64 tsan tsan-fuzzing 0.93 -> 0.11
88% sccache hit rate linux64 asan asan-fuzzing 0.93 -> 0.11
88% sccache hit rate osx-cross asan asan-fuzzing 0.94 -> 0.11
87% sccache hit rate windows2012-64 debug fuzzing 0.94 -> 0.12
87% sccache hit rate windows2012-64 debug 0.93 -> 0.12
87% sccache hit rate osx-cross debug fuzzing 0.92 -> 0.12
87% sccache hit rate windows2012-32 opt 0.93 -> 0.12
87% sccache hit rate windows2012-32 debug 0.91 -> 0.12
87% sccache hit rate windows2012-64 opt 0.92 -> 0.12
87% sccache hit rate windows2012-64 asan debug 0.93 -> 0.12
86% sccache hit rate windows2012-64 asan fuzzing-asan 0.92 -> 0.13
86% sccache hit rate windows2012-64 asan opt 0.91 -> 0.13
85% sccache hit rate linux64 debug gcp 0.97 -> 0.14
85% sccache hit rate linux64 asan opt 0.94 -> 0.14
85% sccache hit rate linux64 asan debug 0.93 -> 0.14
85% sccache hit rate linux64 opt valgrind 0.95 -> 0.14
85% sccache hit rate linux64 opt 0.93 -> 0.14
85% sccache hit rate linux64 debug 0.93 -> 0.14
85% sccache hit rate osx-cross debug gcp 0.96 -> 0.15
85% sccache hit rate linux64-aarch64 opt 0.94 -> 0.15
84% sccache hit rate osx-cross debug 0.94 -> 0.15
52% build times windows2012-64 debug fuzzing taskcluster-c4.4xlarge 2,555.67 -> 3,882.58
31% build times windows2012-64 asan debug taskcluster-c5.4xlarge 2,850.61 -> 3,744.81
24% build times windows2012-aarch64 opt aarch64 taskcluster-c5.4xlarge 2,849.21 -> 3,530.48

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=25149

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1618720 - Generate the ThreadAllow list deterministically.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c55d7fe46e1f
Generate the ThreadAllow list deterministically. r=erahm,dmajor

Comment 3

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/c55d7fe46e1f