1570318 - 15.3 - 15.56% build times (linux64-shippable) regression on push 16d1399f602c8a336e8f6e7a160f0e698aff486d (Fri July 19 2019)

Status: RESOLVED DUPLICATE of bug 1568152

(Testing :: General, defect, P2)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=16d1399f602c8a336e8f6e7a160f0e698aff486d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

16% build times linux64-shippable opt nightly taskcluster-c5d.4xlarge 3,717.69 -> 4,296.07
15% build times linux64-shippable opt nightly taskcluster-c5d.4xlarge 3,705.14 -> 4,272.06

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=22132

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Ed Lee :Mardak]

Comparing to the previous autoland commit, the compile subtest seems to have the most regressions:

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&newProject=autoland&newRevision=16d1399f602c8a336e8f6e7a160f0e698aff486d&originalSignature=1921819&newSignature=1921819&framework=2&originalRevision=0bbd8e64cb9cbef70d12bcf502a8948882b63fe9

compile graph 3650.16 < 4034.57 10.53%

Retriggered some nearby builds to see if they show the regression

https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=16d1399f602c8a336e8f6e7a160f0e698aff486d&searchStr=bpgo%2Clinux64%2C%28b&fromchange=5fff2a9bf0785afbdb774c178135cc9e9ad18211&group_state=expanded

The browser/components/newtab changes shouldn't have much impact on compile measurement as they should be just packaged and not compiled… (?)

Comment 2

[Ed Lee :Mardak]

After some retriggers to run on c5d.4xlarge (see first link comment 1), looks like there isn't much regression from bug 1567243?

Comment 3

[Ed Lee :Mardak]

Here's the commits from the original suspect commit and the next one that ran on c5d.4xlarge:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=16d1399f602c8a336e8f6e7a160f0e698aff486d&tochange=5c6895ea59af3d2e7f2f6e32c5bd9a96a48953bd

Nathan Froyd — Bug 1486042 - default clang-cl pgo to use cross-language LTO; r=dmajor

Could this have caused build times to regress ? (Retrigging some builds right now…)

Comment 4

[Ed Lee :Mardak]

Ah, I see there was bug 1568152 which was marked WONTFIX, so this might be a dupe of that one.

Comment 5

[Ed Lee :Mardak]

Ah yup I see bug 1569202 too.

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Hi Ed, sorry for replying this late. Yes, looks like this is a duplicate of 1568152 and 1569202. Thanks for taking care of this!