1515631 - 27.89 - 30.76% build times (windows2012-64) regression on push 585f7d2135eeaddf35d239c909ef43ffa37fa860 (Wed Dec 19 2018)

Status: VERIFIED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=585f7d2135eeaddf35d239c909ef43ffa37fa860

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 31%  build times windows2012-64 pgo taskcluster-c4.4xlarge     6,107.24 -> 7,985.58
 28%  build times windows2012-64 pgo taskcluster-c5.4xlarge     5,192.30 -> 6,640.42


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=18347

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Nathan Froyd [:froydnj]]

That's a pretty disgusting regression for a clang update.

Comment 2

[(Away)]

Yes, and especially since these toolchains go out to developers via mach bootstrap, I think we should back out until we better understand what's going on. tjr does that sound reasonable?

I don't mind helping with bisecting the compiler if needed, although my availability for the remainder of the year is pretty low.

(Obligatory grumble: there would be less to bisect if we had more frequent updates.)

Comment 3

[Tom Ritter [:tjr]]

(In reply to David Major [:dmajor] from comment #2)
> Yes, and especially since these toolchains go out to developers via mach
> bootstrap, I think we should back out until we better understand what's
> going on. tjr does that sound reasonable?
> 
> I don't mind helping with bisecting the compiler if needed, although my
> availability for the remainder of the year is pretty low.
> 
> (Obligatory grumble: there would be less to bisect if we had more frequent
> updates.)

We will need to back out Bug 1485016 and Bug 1512921; but yes if you think that's the best course of action; I will ask the sheriffs.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

with the backout, we see improvements:
== Change summary for alert #18422 (as of Fri, 21 Dec 2018 10:27:04 GMT) ==

Improvements:

 30%  build times windows2012-64 pgo taskcluster-c5.4xlarge     6,933.47 -> 4,844.84
 24%  build times windows2012-64 pgo taskcluster-c4.4xlarge     8,003.91 -> 6,113.37

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=18422

Comment 5

[Tom Ritter [:tjr]]

We tracked this down to this commit where I left a comment: https://bugs.llvm.org/show_bug.cgi?id=39667#c18

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Calling this specific regression fixed by the backout noted by comment 4. Glad to see that the upstream fix for when it re-landed.