1500344 - 10.94 - 57.3% build times (windows-mingw32, windows2012-32, windows2012-32-noopt, windows2012-64, windows2012-64-noopt) regression on push 364f270ff9bf70fce86e0aa8c5eeb673239a0816 (Thu Oct 18 2018)

Status: VERIFIED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=364f270ff9bf70fce86e0aa8c5eeb673239a0816

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 57%  build times windows-mingw32 all 64 clang debug taskcluster-m5d.4xlarge     611.12 -> 961.27
 49%  build times windows2012-32-noopt debug taskcluster-c5.4xlarge              1,284.04 -> 1,913.98
 47%  build times windows2012-64-noopt debug taskcluster-c4.4xlarge              1,649.10 -> 2,416.46
 46%  build times windows2012-32-noopt debug taskcluster-c4.4xlarge              1,628.87 -> 2,379.30
 45%  build times windows2012-64 debug plain taskcluster-c4.4xlarge              1,775.30 -> 2,578.35
 44%  build times windows2012-64-noopt debug taskcluster-c5.4xlarge              1,309.99 -> 1,889.93
 39%  build times windows2012-64 debug plain taskcluster-c5.4xlarge              1,478.82 -> 2,057.90
 31%  build times windows2012-32 debug taskcluster-c4.4xlarge                    2,400.68 -> 3,135.75
 31%  build times windows2012-64 debug taskcluster-c5.4xlarge                    1,924.37 -> 2,512.07
 31%  build times windows2012-64 debug taskcluster-c4.4xlarge                    2,430.05 -> 3,171.40
 28%  build times windows2012-64 asan debug taskcluster-c4.4xlarge               2,701.82 -> 3,455.83
 27%  build times windows2012-32 debug taskcluster-c5.4xlarge                    1,950.89 -> 2,477.92
 25%  build times windows2012-64 asan debug taskcluster-c5.4xlarge               2,295.73 -> 2,876.85
 25%  build times windows2012-32 pgo taskcluster-c4.4xlarge                      5,049.40 -> 6,321.68
 22%  build times windows2012-32 opt taskcluster-c4.4xlarge                      2,358.37 -> 2,875.66
 22%  build times windows2012-64 opt taskcluster-c4.4xlarge                      2,430.43 -> 2,957.54
 20%  build times windows2012-64 pgo taskcluster-c5.4xlarge                      4,456.25 -> 5,331.11
 19%  build times windows2012-64 asan opt taskcluster-c4.4xlarge                 2,548.57 -> 3,040.04
 18%  build times windows2012-32 opt rusttests taskcluster-c4.4xlarge            2,013.61 -> 2,371.51
 17%  build times windows2012-64 opt plain taskcluster-c4.4xlarge                2,040.36 -> 2,396.55
 15%  build times windows2012-32 opt rusttests taskcluster-c5.4xlarge            1,721.43 -> 1,978.06
 12%  build times windows2012-64 asan opt taskcluster-c5.4xlarge                 2,225.30 -> 2,500.16
 11%  build times windows2012-32 opt taskcluster-c5.4xlarge                      2,114.98 -> 2,346.27

Improvements:

 96%  sccache requests_not_cacheable windows2012-64 asan debug taskcluster-c5.4xlarge        3,767.00 -> 150.00
 96%  sccache requests_not_cacheable windows2012-64 asan debug taskcluster-c4.4xlarge        3,767.00 -> 150.00
 96%  sccache requests_not_cacheable windows2012-64 asan opt taskcluster-c5.4xlarge          3,760.00 -> 150.00
 96%  sccache requests_not_cacheable windows2012-64 asan opt taskcluster-c4.4xlarge          3,760.00 -> 150.00
 96%  sccache requests_not_cacheable windows2012-64 debug taskcluster-c5.4xlarge             3,817.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-64 opt rusttests taskcluster-c5.4xlarge     3,810.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-64-noopt debug taskcluster-c5.4xlarge       3,817.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-64 opt taskcluster-c5.4xlarge               3,810.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-64 opt taskcluster-c4.4xlarge               3,810.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-64 debug taskcluster-c4.4xlarge             3,817.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-64-noopt debug taskcluster-c4.4xlarge       3,817.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32 debug taskcluster-c5.4xlarge             3,691.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32 opt rusttests taskcluster-c5.4xlarge     3,684.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32 opt taskcluster-c5.4xlarge               3,684.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32-noopt debug taskcluster-c5.4xlarge       3,691.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32-noopt debug taskcluster-c4.4xlarge       3,691.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32 debug taskcluster-c4.4xlarge             3,691.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32 opt taskcluster-c4.4xlarge               3,684.00 -> 156.00
 96%  sccache requests_not_cacheable windows2012-32 opt rusttests taskcluster-c4.4xlarge     3,684.00 -> 156.00


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=16902

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

Comment 1

[Ionuț Goldan [:igoldan]]

I believe this is a WONTFIX, as it performs extra work during build time?

Comment 2

[Mike Hommey [:glandium]]

The sccache improvements are a red herring, see bug 1500295.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1500274#c7

The regressions larger than 20% are weird. Most are debug and/or no-opt, maybe the plugin is not compiled with enough optimization enabled on those builds.

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Mike Hommey [:glandium] from comment #2)
> The sccache improvements are a red herring, see bug 1500295.
> 
> See https://bugzilla.mozilla.org/show_bug.cgi?id=1500274#c7
> 
> The regressions larger than 20% are weird. Most are debug and/or no-opt,
> maybe the plugin is not compiled with enough optimization enabled on those
> builds.

Any updates on this bug?

Comment 4

[Mike Hommey [:glandium]]

I identified at least part of the regression here: bug 1501903

Comment 5

[Mike Hommey [:glandium]]

Looking at the graphs, it seems bug 1501903 largely fixed the regression on opt builds, but not completely on debug builds. It also seems we stopped building on c4.xlarge around Oct 20, so there's nothing new to observe on the corresponding graphs.

Comment 6

[Ionuț Goldan [:igoldan]]

(In reply to Mike Hommey [:glandium] from comment #5)
> Looking at the graphs, it seems bug 1501903 largely fixed the regression on
> opt builds, but not completely on debug builds. It also seems we stopped
> building on c4.xlarge around Oct 20, so there's nothing new to observe on
> the corresponding graphs.

This partially resolved half of the regression alerts:

== Change summary for alert #17155 (as of Thu, 25 Oct 2018 18:22:40 GMT) ==

Improvements:

 23%  build times windows2012-32-noopt debug taskcluster-c5.4xlarge       1,830.80 -> 1,416.60
 21%  build times windows2012-64-noopt debug taskcluster-c5.4xlarge       1,862.39 -> 1,464.95
 21%  build times windows2012-64 debug plain taskcluster-c5.4xlarge       1,970.34 -> 1,562.00
 19%  build times windows2012-32 debug taskcluster-c5.4xlarge             2,415.02 -> 1,962.15
 18%  build times windows2012-64 debug taskcluster-c5.4xlarge             2,428.14 -> 1,985.03
 14%  build times windows2012-64 asan debug taskcluster-c5.4xlarge        2,842.66 -> 2,451.16
 12%  build times windows2012-64 pgo taskcluster-c5.4xlarge               5,062.41 -> 4,439.63
 12%  build times windows2012-32 pgo taskcluster-c5.4xlarge               4,961.77 -> 4,368.77
 11%  build times windows2012-64 asan opt taskcluster-c5.4xlarge          2,465.55 -> 2,199.15
  7%  build times windows2012-32 opt rusttests taskcluster-c5.4xlarge     1,867.18 -> 1,732.80
  7%  build times windows2012-32 opt taskcluster-c5.4xlarge               2,252.70 -> 2,092.92

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=17155

Comment 7

[Ionuț Goldan [:igoldan]]

(In reply to Mike Hommey [:glandium] from comment #5)
> Looking at the graphs, it seems bug 1501903 largely fixed the regression on
> opt builds, but not completely on debug builds. It also seems we stopped
> building on c4.xlarge around Oct 20, so there's nothing new to observe on
> the corresponding graphs.

If there's any ongoing work that aims to address these, please link the bugs.

Comment 8

[Mike Hommey [:glandium]]

Still haven't figured out what specifically is going slow.

Comment 9

[Marion Daly [:mdaly]]

Is there an assignee for this?

Comment 10

[Nathan Froyd [:froydnj]]

There is not.  The plugin still seems to be built with optimization, so that idea is out.  glandium, have you found any other plausible lines of investigation?

Comment 11

[Mike Hommey [:glandium]]

Bug 1506756 is the last thing I found. Try suggests it's an improvement, but there might still be a larger overhead on Windows than on Linux from enabling the plugin. Let's see how things play out after that lands. (It worth noting, though, that build times have slowly regressed since the landing of bug 1501903)

Comment 12

[Mike Hommey [:glandium]]

I think bug 1501903 + bug 1506756 reduced the regression enough to call this fixed. Obviously, we can't have zero overhead from static analysis. It might be possible to make it faster, but that would fall outside the scope of this specific regression.