Closed Bug 1638302 Opened 5 years ago Closed 5 years ago

[llvm10] 7.72 - 14.15% build times (linux64-shippable, osx-shippable, windows2012-32-shippable, windows2012-64-shippable, windows2012-aarch64) regression on push d254d383887711cbad74ad85391b027c4ed49e56 (Thu May 14 2020)

Categories

(Firefox Build System :: Toolchains, defect)

defect

Tracking

(firefox-esr68 unaffected, firefox76 unaffected, firefox77 unaffected, firefox78 fixed)

RESOLVED FIXED
mozilla78
Tracking Status
firefox-esr68 --- unaffected
firefox76 --- unaffected
firefox77 --- unaffected
firefox78 --- fixed

People

(Reporter: alexandrui, Assigned: away)

References

(Regression)

Details

(Keywords: perf-alert, regression)

Perfherder has detected a build_metrics performance regression from push d254d383887711cbad74ad85391b027c4ed49e56. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

14% build times windows2012-64-shippable opt nightly taskcluster-c5.4xlarge 2,247.52 -> 2,565.58
12% build times windows2012-64-shippable opt nightly taskcluster-m5.4xlarge 2,376.48 -> 2,672.47
12% build times windows2012-aarch64 opt aarch64-no-eme nightly taskcluster-c5d.4xlarge 2,192.11 -> 2,458.88
11% build times windows2012-32-shippable opt nightly taskcluster-c5.4xlarge 2,278.55 -> 2,528.62
11% build times windows2012-32-shippable opt nightly taskcluster-c5d.4xlarge 2,229.44 -> 2,472.99
11% build times osx-shippable opt nightly taskcluster-c5.4xlarge 2,964.18 -> 3,279.40
10% build times windows2012-64-shippable opt nightly taskcluster-c5d.4xlarge 2,213.17 -> 2,440.46
10% build times windows2012-32-shippable opt nightly taskcluster-m5.4xlarge 2,375.03 -> 2,617.00
10% build times osx-shippable opt nightly taskcluster-c5d.4xlarge 2,905.86 -> 3,186.44
10% build times linux64-shippable opt nightly taskcluster-c5d.4xlarge 2,764.77 -> 3,028.63
10% build times osx-shippable opt nightly taskcluster-m5.4xlarge 3,066.88 -> 3,359.51
8% build times linux64-shippable opt nightly taskcluster-m5.4xlarge 2,945.91 -> 3,173.42

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(catlee)

How could that be? Nothing is that patch touched the build jobs at all

Component: Performance → geckodriver

I'm looking again into it.

Flags: needinfo?(catlee) → needinfo?(dmajor)

Yes. Oof, this is pretty bad. There was a bug upstream about compile time regressions during the 10.0 testing cycle, but I had thought most of its spinoffs had been mitigated before the release.

Flags: needinfo?(dmajor)
Component: geckodriver → Toolchains
Product: Testing → Firefox Build System
Version: Version 3 → unspecified
Has Regression Range: --- → yes

See also https://lists.llvm.org/pipermail/llvm-dev/2020-May/141482.html. A lot of Rust folks actually see such a decrease of performance when switching to LLVM 10.

What action should we take on here? Note that this also adds quite a lot of extra CI costs for us.

Flags: needinfo?(dmajor)
Summary: 7.72 - 14.15% build times (linux64-shippable, osx-shippable, windows2012-32-shippable, windows2012-64-shippable, windows2012-aarch64) regression on push d254d383887711cbad74ad85391b027c4ed49e56 (Thu May 14 2020) → [llvm10] 7.72 - 14.15% build times (linux64-shippable, osx-shippable, windows2012-32-shippable, windows2012-64-shippable, windows2012-aarch64) regression on push d254d383887711cbad74ad85391b027c4ed49e56 (Thu May 14 2020)

clang-10 was backed out, so I guess we can mark this as fixed.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Flags: needinfo?(dmajor)
You need to log in before you can comment on or make changes to this bug.