1471521 - 1.69 - 21.95% build times / installer size (osx-cross, windows2012-32, windows2012-64) regression on push 160e085dfb0b546c2968aeefe19947881c76040d (Tue Jun 19 2018)

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=160e085dfb0b546c2968aeefe19947881c76040d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 18%  build times windows2012-64-add-on-devel opt taskcluster-c4.4xlarge     2,688.50 -> 3,175.77
 17%  build times windows2012-32-add-on-devel opt taskcluster-c4.4xlarge     2,607.17 -> 3,061.76
  6%  build times windows2012-32 opt nightly taskcluster-c4.4xlarge          4,440.67 -> 4,700.92
  6%  build times windows2012-64 opt taskcluster-c4.4xlarge                  4,740.45 -> 5,012.86
  6%  build times windows2012-32 opt taskcluster-c4.4xlarge                  4,450.56 -> 4,695.23
  5%  build times windows2012-64 opt nightly taskcluster-c4.4xlarge          4,707.08 -> 4,938.21
  5%  build times windows2012-64 debug taskcluster-c4.4xlarge                2,384.60 -> 2,492.32
  4%  build times windows2012-32 debug taskcluster-c4.4xlarge                2,303.69 -> 2,400.23
  2%  installer size osx-cross opt                                           54,630,761.92 -> 55,577,811.92
  2%  installer size osx-cross opt nightly                                   54,648,830.75 -> 55,569,702.08


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=13980

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

Comment 1

[Ionuț Goldan [:igoldan]]

:jmaher noticed the builds here don't seem to run on autoland/inbound- possibly they don't use c4.4xlarge for those builds on trunk. :gps, :ted is this expected?

Comment 2

[Ionuț Goldan [:igoldan]]

I also require assistance for identifying the installer size regressions as well.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

the installer size regressions are downstream from  bug 1468689

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)
> the installer size regressions are downstream from  bug 1468689

Thanks for tracking that down!

Comment 5

[Gregory Szorc [:gps]]

The regressions are an aggregate result of everything that happened in the 62 development cycle.

It is concerning that build times got slower - in some cases by a few minutes. This is likely the result of more code - especially Rust - in the build. As long as we keep adding things, the build will become slower. It could have also gotten slower due to toolchain changes. Sometimes a toolchain upgrade makes builds slower. That's a fact of life. Fortunately, it looks like Rust 1.28 will be clawing back Rust-related build slowness (see bug 1447116 comment 29).

So we should acknowledge the build slowdown alerts.

Installer size is a bit more worrying. That's something that is user-facing. I know some toolchain changes have recently bloated the installer by a few hundred KB. But those were all signed off on when they happened on trunk.

I'm not saying we should acknowledge the installer size changes as well (that's beyond my authority). But I'm pretty confident saying that there's nothing on the build system side contributing to those that wasn't already known and acknowledged. Most of the non-toolchain changes are likely the long tail of new code being written and new assets being included in the Firefox distribution. This tends to happen with every release (unless we delete large features).

Comment 6

[Ionuț Goldan [:igoldan]]

In bug 1468689 we decided to accept the installer size regressions.