Closed Bug 1429311 Opened 2 years ago Closed 2 years ago

7.45 - 12.1% build times (windows2012-64) regression on push 5afa32a16a2f7ebbd8bf19e49efc50832110980d (Thu Jan 4 2018)

Categories

(Infrastructure & Operations :: SRE, defect)

x86_64
Windows
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: igoldan, Unassigned)

References

Details

(Keywords: regression)

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=5afa32a16a2f7ebbd8bf19e49efc50832110980d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 12%  build times windows2012-64 debug static-analysis taskcluster-c4.4xlarge     2,157.82 -> 2,418.85
  7%  build times windows2012-64 opt static-analysis taskcluster-c4.4xlarge       2,740.23 -> 2,944.31


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=11068

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics
Very likely to be an out-of-tree infrastructure change, which caused this.
Blocks: 1429325
#taskcluster didn't seem to think there were any AMI deploys last week.

This /might/ be Amazon patching the underlying hosts for spectre and meltdown :/
(In reply to Gregory Szorc [:gps] from comment #2)
> #taskcluster didn't seem to think there were any AMI deploys last week.
> 
> This /might/ be Amazon patching the underlying hosts for spectre and
> meltdown :/

Sounds reasonable.
actually this would affect ALL versions of Firefox, not just firefox 59
Component: Untriaged → Infrastructure: AWS
Product: Firefox → Infrastructure & Operations
QA Contact: cshields
:gps' comment [1] in bug 1429319 made me check this bug too. Turns out the build times here have also returned to normal.

== Change summary for alert #11208 (as of Wed, 17 Jan 2018 10:56:51 GMT) ==

Improvements:

  7%  build times windows2012-64 debug static-analysis taskcluster-c4.4xlarge     2,271.50 -> 2,119.95
  1%  build times windows2012-64 opt static-analysis taskcluster-c4.4xlarge       2,749.36 -> 2,725.16

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=11208


Marking this bug as resolved.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1429319#c4
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.