1429311 - 7.45 - 12.1% build times (windows2012-64) regression on push 5afa32a16a2f7ebbd8bf19e49efc50832110980d (Thu Jan 4 2018)

Status: RESOLVED FIXED

(Infrastructure & Operations :: SRE, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=5afa32a16a2f7ebbd8bf19e49efc50832110980d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 12%  build times windows2012-64 debug static-analysis taskcluster-c4.4xlarge     2,157.82 -> 2,418.85
  7%  build times windows2012-64 opt static-analysis taskcluster-c4.4xlarge       2,740.23 -> 2,944.31


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=11068

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

Comment 1

[Ionuț Goldan [:igoldan]]

Very likely to be an out-of-tree infrastructure change, which caused this.

Comment 2

[Gregory Szorc [:gps]]

#taskcluster didn't seem to think there were any AMI deploys last week.

This /might/ be Amazon patching the underlying hosts for spectre and meltdown :/

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Gregory Szorc [:gps] from comment #2)
> #taskcluster didn't seem to think there were any AMI deploys last week.
> 
> This /might/ be Amazon patching the underlying hosts for spectre and
> meltdown :/

Sounds reasonable.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

actually this would affect ALL versions of Firefox, not just firefox 59

Comment 5

[Carmen Fat [:cfat] - Ecosystem QA]

Marking all versions as affected based on comment 4.

Comment 6

[Ionuț Goldan [:igoldan]]

:gps' comment [1] in bug 1429319 made me check this bug too. Turns out the build times here have also returned to normal.

== Change summary for alert #11208 (as of Wed, 17 Jan 2018 10:56:51 GMT) ==

Improvements:

  7%  build times windows2012-64 debug static-analysis taskcluster-c4.4xlarge     2,271.50 -> 2,119.95
  1%  build times windows2012-64 opt static-analysis taskcluster-c4.4xlarge       2,749.36 -> 2,725.16

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=11208


Marking this bug as resolved.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1429319#c4