1603430 - 16.69 - 32.91% build times (linux32-shippable, windows2012-32-shippable, windows2012-64-shippable) regression on push 65d69ba8d88ab562557bb61139fca09381ac25d3 (Wed December 11 2019)

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Florin Strugariu [:Bebe]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=65d69ba8d88ab562557bb61139fca09381ac25d3

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

33% build times windows2012-32-shippable opt instrumented taskcluster-c5.4xlarge 2,087.58 -> 2,774.62
29% build times windows2012-64-shippable opt instrumented taskcluster-c5.4xlarge 2,041.02 -> 2,627.49
24% build times windows2012-32-shippable opt instrumented taskcluster-c4.4xlarge 2,588.47 -> 3,199.73
22% build times windows2012-64-shippable opt instrumented taskcluster-c4.4xlarge 2,548.18 -> 3,113.04
17% build times linux32-shippable opt instrumented taskcluster-m5.4xlarge 1,913.43 -> 2,232.81

Improvements:

5% build times windows2012-32-shippable opt nightly taskcluster-c4.4xlarge 3,791.20 -> 3,612.21
4% build times windows2012-64-shippable opt nightly taskcluster-c4.4xlarge 3,891.82 -> 3,729.09

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=24431

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[(Away)]

Interesting, we got faster optimization-phase builds, but (much) slower instrumentation-phase builds. We often say wontfix in the name of runtime perf to these kinds of regressions, but this one is so large that I think it deserves some extra attention.

Out of curiosity, how come instrumented builds don't use sccache? (I can understand why the optimization phase doesn't, but the instrumentation phase seems like it should be doable)

As I'm about to go offline for the holidays, erahm would you mind taking over this investigation?

Comment 2

[Michael Shal [:mshal]]

(In reply to :dmajor from comment #1)

Out of curiosity, how come instrumented builds don't use sccache? (I can understand why the optimization phase doesn't, but the instrumentation phase seems like it should be doable)

We used to, but it was disabled in bug 1583368.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Setting 73 to fix-optional for now, but this seems likely to be a wontfix given that this is an expected outcome from turning on more PGO for builds and we aren't able to enable sccache for affected phases.

Comment 4

[Eric Rahm [:erahm]]

I'm okay accepting this for now. As a follow-up It might be worth having the rustc team take a look at why enabling PGO instrumentation for Rust slows down the build time to such an extent. Michael, is this something we should add to the backlog for 2020H1?

Comment 5

[Michael Woerister]

Yes, let's look into it. I don't seem to have access to the bug that disabled sccache for instrumented builds, but of the top of my head I can't think of a reason why those should not be able use sccache in general.

Comment 6

[Eric Rahm [:erahm]]

(In reply to Michael Woerister from comment #5)

Yes, let's look into it. I don't seem to have access to the bug that disabled sccache for instrumented builds, but of the top of my head I can't think of a reason why those should not be able use sccache in general.

The sccache thing is probably orthogonal, we weren't using it when this regression happened, it was solely due to turning on Rust PGO for Windows shippable builds.

Comment 7

[Michael Woerister]

OK, seems worth investigating for us then.

Comment 8

[Greg Mierzwinski [:sparky]]

Hi :erahm, I'm closing this as WONTFIX given that it's been 5 years with no activity, and the platforms that regressed no longer exist. The analogous regressing metrics also have a lower build time now too: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=1&replicates=0&series=mozilla-central,4946191,1,2&timerange=31536000

Feel free to change this resolution if needed.