1568152 - 0.32 - 29.75% build times / installer size (android-5-0-x86_64, linux32-shippable, linux64-shippable, osx-shippable, windows2012-64-shippable) regression on push 8ba3c1292475b96e2ccb46c3232c929863451ff6 (Fri July 19 2019)

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Marian Raiciof [:marauder]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=8ba3c1292475b96e2ccb46c3232c929863451ff6

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

30% build times windows2012-64-shippable opt instrumented taskcluster-c4.4xlarge 2,180.68 -> 2,829.33
22% build times android-5-0-x86_64 opt gcp taskcluster-n1-highcpu-64 2,055.18 -> 2,513.20
16% build times linux32-shippable opt nightly taskcluster-c5.4xlarge 4,001.52 -> 4,651.95
14% build times linux64-shippable opt nightly taskcluster-c5.4xlarge 3,819.22 -> 4,360.21
0.32% installer size osx-shippable opt nightly 79,324,388.42 -> 79,581,583.50

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=21961

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 2

[Eric Rahm [:erahm]]

An increase in build times is expected when adding cross-language LTO (we saw this with our original Win64 landings), this is limited to shippable builds and should not impact local dev builds. A 260KiB increase in the OSX installer is also expected with more inlining, if it were android I think we'd be more worried but as-is it's probably okay.

The one thing that's odd is we saw an increase in build times for the shippable instrumented win64 build. Nathan, is that expected to be LTO'd at all?

Comment 3

[Nathan Froyd [:froydnj]]

(In reply to Eric Rahm [:erahm] from comment #2)

The one thing that's odd is we saw an increase in build times for the shippable instrumented win64 build. Nathan, is that expected to be LTO'd at all?

We turn cross-language LTO off for the instrumentation build:

https://searchfox.org/mozilla-central/source/build/moz.configure/toolchain.configure#1560-1562

Thanks to:

https://hg.mozilla.org/mozilla-central/rev/b5c65f2ea1acbc7a95a1c57e72464661540ab821

Maybe what happened is the following:

1. The above patch wanted to turn off LTO for the instrumented build.
2. But the cross-language LTO flags on the Rust side were still getting added directly by the mozconfig.
3. So the patch had only a partial effect (? but it still should have turned off LTO'ing the C++ code, which ought to have made a difference...)
4. The cross-language LTO patch(es) landed.
5. Which correctly only added Rust-side LTO flags during the profile-use phase.
6. ...so we compile all the Rust code into actual object code during the profile-generate phase.

...and that compilation into actual object code is significantly more expensive than turning it into LLVM bitcode and letting the linker sort it out?

That line of argument doesn't make a lot of sense, but I don't have any other ideas why the instrumentation phase would be getting more expensive.

Comment 5

[Marian Raiciof [:marauder]]

Thanks everyone for the updates!