1520140 - 47.15% build times (linux64) regression on push 8844751cb1698c69c2fead027fb31e791bc0ab00 (Mon Jan 14 2019)

Status: RESOLVED INVALID

(Firefox Build System :: Task Configuration, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=4d932b82695c7ac58901ef0a03f44fed6ce2d1f0&tochange=8844751cb1698c69c2fead027fb31e791bc0ab00

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

47% build times linux64 opt artifact taskcluster-c4.4xlarge 142.79 -> 210.13

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=18734

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Ionuț Goldan [:igoldan]]

:mshal the patch includes a big amount of bugs. I went through them and identified bug 1519424 as the most related to this regression. I'm not entirely sure I'm correct about this finding, so don't hesitate to comment back.

Comment 2

[Tom Prince [:tomprince]]

I don't see bug 1519424 in the pushlog linked above? Also, it looks like the build times have two clusters, one around 100 and one around (see graph), so I think this might be a false positive.

I do see a couple of spikes to >250 there that might be worth investigating, though they do appear to be intermittent.

Comment 3

[(not currently active) Ted Mielczarek]

That bug can't possibly be at fault--it only affects the linux64-profile task in 3-tiered PGO builds.

Comment 4

[Michael Shal [:mshal]]

Agreed with #c3 - this alert is from artifact builds, which use a separate task configuration. I'm also a bit confused by teh clustering mentioned in #c2 - chmanchester, any ideas there? Is that something worth investigating?

Comment 5

[Chris Manchester (limited bugmail, email directly)]

This particular regression is a false positive, these have been intermittently bimodal like this for a while. It looks like it started around October, but I only see it on the graph for c4.4xlarge instances, so it may not be a regression in the build system. The slowdown itself seems to be time taken during automation steps and may be due to how certain builds choose to prioritize them.

Comment 6

[Ionuț Goldan [:igoldan]]

Baseline returned to normal Sorry for causing this confusion.
I'm closing this as invalid.