1582399 - 8.51% build times (windows2012-32-shippable) regression on push 7438ce6ba80cc644ce4d6dd77c59d6c844b75e05 (Tue August 27 2019)

Status: RESOLVED DUPLICATE of bug 1577747

(Firefox Build System :: General, defect)

Description

[Marian Raiciof [:marauder]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=7438ce6ba80cc644ce4d6dd77c59d6c844b75e05

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

9% build times windows2012-32-shippable opt instrumented taskcluster-c5.4xlarge 1,618.83 -> 1,756.57

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=23132

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Lina Butler (ex-Mozilla)]

Unfortunately, backing that patch out isn't possible; it contains a database migration that has already shipped in Nightly and Beta, and reverting it would break new bookmark sync. 😕 I'm a bit baffled how a dependency update would cause a 9% increase in build time.

What are the next steps here, Marian? Does the tool automatically bisect changesets (and test with the patch backed out), or was that done by hand? Has that persisted through 70 and 71? I don't really know how to interpret that graph, or how Perfherder decided that this particular patch caused the issue...

Thanks in advance for your help!

Comment 2

[Ionuț Goldan [:igoldan]]

Marian, I don't think bug 1575757 caused this build time increase. It's more likely this was caused by a bug from the Firefox Build System component.

Comment 3

[Marian Raiciof [:marauder]]

Thanks for the info everyone, i'll keep investigating this regression.

Comment 4

[Marian Raiciof [:marauder]]

Hi Mike,
Which bug from the "regressed by" list do you think it might be the cause for this regression ?
Thanks!

Comment 5

[Tom Ritter [:tjr]]

It's unlikely Bug 1575975 caused it; but it did add a new library to the linking arguments for non-mingw builds so it's within the realm of possibility...

Comment 6

[Chris Manchester (limited bugmail, email directly)]

Scrolling through autoland this seems pretty definitively to be bug 1575198, but that push's data point is missing from perfherder for some reason.

Mike, is that slowdown expected? Anything else we need to do here?

Comment 7

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

(In reply to Chris Manchester (:chmanchester) from comment #6)

Scrolling through autoland this seems pretty definitively to be bug 1575198, but that push's data point is missing from perfherder for some reason.

This makes much more sense than any of the other suggested tickets.

Comment 8

[Mike Hommey [:glandium]]

(In reply to Chris Manchester (:chmanchester) from comment #6)

Scrolling through autoland this seems pretty definitively to be bug 1575198, but that push's data point is missing from perfherder for some reason.

Because a given push is only going to build once with one type of worker. So the data point is not going to show up in the graph for other types of workers.

Mike, is that slowdown expected? Anything else we need to do here?

For any build type that is not sccached, yes. See bug 1577747 and dupes.

Comment 9

[Chris Manchester (limited bugmail, email directly)]

(In reply to Mike Hommey [:glandium] from comment #8)

(In reply to Chris Manchester (:chmanchester) from comment #6)

Scrolling through autoland this seems pretty definitively to be bug 1575198, but that push's data point is missing from perfherder for some reason.

Because a given push is only going to build once with one type of worker. So the data point is not going to show up in the graph for other types of workers.

In this case it looks like ingestion failed. The build ran on a c5.4xlarge instance, but if I click through from taskcluster to the graph perfherder diplays an error.

Mike, is that slowdown expected? Anything else we need to do here?

For any build type that is not sccached, yes. See bug 1577747 and dupes.

*** This bug has been marked as a duplicate of bug 1577747 ***