1480708 - 8% build times (windows2012-64) regression on push 084b1e903489a2c5643f99da82b2c788f786bd61 (Thu Aug 2 2018)

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=084b1e903489a2c5643f99da82b2c788f786bd61

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  8%  build times windows2012-64 pgo taskcluster-c4.4xlarge      4,664.21 -> 5,043.46

Improvements:

  8%  sccache hit rate windows2012-64 pgo taskcluster-c4.4xlarge 0.93 -> 1.00


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=14720

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

Comment 1

[(Away)]

> Regressions:
> 
>   8%  build times windows2012-64 pgo taskcluster-c4.4xlarge      4,664.21 ->
> 5,043.46

This is expected given the nature of the changes in bug 1341525. (And we'll expect the same in 32-bit when it lands.)

> Improvements:
> 
>   8%  sccache hit rate windows2012-64 pgo taskcluster-c4.4xlarge 0.93 -> 1.00
> 

Was this the only improvement that got alerted? Based on try pushes I was expecting much more. We have some good news about speedometer and installer size, for example.

When we have the full list of improvements, it will become clear that this bug is an easy wontfix in exchange for lots of perf wins. :)

Comment 2

[(Away)]

For reference here's how the patch looked on try: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=d7d168ac4922&newProject=try&newRevision=9108737fe9d5&framework=1&filter=64&showOnlyConfident=1

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to David Major [:dmajor] from comment #1)
> > Improvements:
> > 
> >   8%  sccache hit rate windows2012-64 pgo taskcluster-c4.4xlarge 0.93 -> 1.00
> > 
> 
> Was this the only improvement that got alerted? Based on try pushes I was
> expecting much more. We have some good news about speedometer and installer
> size, for example.
> 
> When we have the full list of improvements, it will become clear that this
> bug is an easy wontfix in exchange for lots of perf wins. :)

Indeed, these are some of the improvements:

== Change summary for alert #14707 (as of Thu, 02 Aug 2018 09:39:51 GMT) ==

Improvements:

  8%  speedometer windows10-64 pgo e10s stylo     75.33 -> 81.01
  7%  damp windows10-64 pgo e10s stylo            246.51 -> 229.42
  6%  tp6_google windows10-64 pgo e10s stylo      382.90 -> 360.42
  6%  a11yr windows10-64 pgo e10s stylo           166.40 -> 157.13
  6%  tp6_amazon windows10-64 pgo e10s stylo      215.27 -> 203.42

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14707

Comment 4

[(Away)]

Previously the clang-cl "pgo" builds weren't actually doing proper PGO. Now they are. The build time regression is an expected consequence. See previous comments for the perf gains we get in exchange for this.