1631778 - 0.11 - 51.82% build times / installer size (android-4-0-armv7-api16, android-4-2-x86, android-5-0-aarch64, android-5-0-x86_64, osx-shippable) regression on push b4ebfaf0e6f3ed856786b6e145741e273066b225 (Tue April 21 2020)

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect)

Description

[Marian Raiciof [:marauder]]

Perfherder has detected a 2 performance regression from push b4ebfaf0e6f3ed856786b6e145741e273066b225. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

52% build times android-4-0-armv7-api16 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 901.42 -> 1,368.53
50% build times android-4-2-x86 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 939.93 -> 1,414.02
50% build times android-5-0-aarch64 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 928.57 -> 1,395.18
43% build times android-5-0-x86_64 opt gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728 991.34 -> 1,414.16
0.11% installer size osx-shippable opt instrumented 114,670,240.33 -> 114,798,422.83

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Jamie Nicol [:jnicol]]

Sorry about this!

I believe the build time regressions are due to some indeterminism in the build-time generated file shaders.rs. Sorting the list of shaders before writing them to that file seems to fix it from my testing. Will have a patch up shortly.

For the installer size regression, some increase is I believe unavoidable, because we now need to include a fair bit more text for the shader source code. Marian, do you happen to know what units those numbers are in? Is that an increase of around 130kB? In any case, I should be able to roughly halve it, by only including desktop GL shaders on Mac, and excluding the GLES ones.

Comment 2

[Jamie Nicol [:jnicol]]

Attachment: Bug 1631778 - Ensure webrender's shaders.rs is filled deterministically. r?jrmuizel

The list of pre-optimized shaders was being generated in a
non-deterministic order, causing large build time regressions. This
sort the list of shaders before writing them to the shaders.rs file.

Comment 3

[Jamie Nicol [:jnicol]]

Attachment: Bug 1631778 - Only optimize shaders for expected GL version on each platform r?jrmuizel

The full set of optimized shaders can be quite large, so only optimize
for the GL version we expect to be used on the target platform. That
is GLES on Android and Windows (where we presume ANGLE is being used),
and GL elsewhere.

If a different GL version is used we will simply fall back to the
unoptimized shaders.

Depends on D71905

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/19e4643b0838
Ensure webrender's shaders.rs is filled deterministically. r=jrmuizel

Comment 5

[Marian Raiciof [:marauder]]

Hey Jamie, i don't know what units are used by those numbers.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/55f13e03d6fa
Only optimize shaders for expected GL version on each platform r=jrmuizel

Comment 7

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/19e4643b0838

Comment 8

[Jamie Nicol [:jnicol]]

The second patch here unfortunately undoes the windows perf wins from bug 1604615 :(

I used cfg!(target_os = windows) to decide whether to optimize GLES shaders. But since we cross compile windows builds from Linux and this is in a build script, it thinks the target_os is "linux" so optimizes the GL shaders instead. This means that when we run on windows, we can't find the optimized shaders, so use the unoptimized ones, so don't get the perf win. Android builds will have the same problem.

Comment 9

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/55f13e03d6fa

Comment 10

[Florin Strugariu [:Bebe]]

This generated a big regression on Raptor:
== Change summary for alert #25736 (as of Mon, 27 Apr 2020 06:38:34 GMT) ==

Regressions:

19% raptor-tp6-twitch-firefox-cold fcp windows10-64-shippable-qr opt 149.79 -> 177.50
18% raptor-tp6-twitch-firefox-cold fcp windows10-64-shippable-qr opt 150.88 -> 178.33
9% raptor-tp6-yahoo-mail-firefox-cold fcp windows10-64-shippable-qr opt 324.17 -> 353.75
7% raptor-tp6-yandex-firefox-cold fcp windows10-64-shippable-qr opt 276.33 -> 295.17
4% raptor-tp6-wikipedia-firefox-cold fcp windows10-64-shippable opt 1,275.58 -> 1,320.67
3% raptor-tp6-google-mail-firefox-cold loadtime windows10-64-shippable opt 621.29 -> 642.33
3% raptor-tp6-wikipedia-firefox-cold loadtime windows10-64-shippable opt 1,423.88 -> 1,469.92
3% raptor-tp6-microsoft-firefox-cold fcp windows10-64-shippable-qr opt 505.17 -> 521.17
3% raptor-tp6-sheets-firefox-cold loadtime windows10-64-shippable opt 1,613.00 -> 1,662.92
2% raptor-tp6-outlook-firefox-cold loadtime windows10-64-shippable opt 611.12 -> 626.08
2% raptor-tp6-sheets-firefox-cold windows10-64-shippable opt 888.84 -> 907.64
2% raptor-tp6-yahoo-mail-firefox-cold windows10-64-shippable-qr opt 502.08 -> 512.54

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25736

Comment 11

[Jamie Nicol [:jnicol]]

The regression should already be fixed by bug 1632444.

Comment 12

[Rachel Tublitz [:rachel]]

Just to confirm before we close this out as fixed by 1632444., Marian - are you still able to repro this?

Comment 13

[Marian Raiciof [:marauder]]

Hey Rachel,
The graph shows that the regression has been fixed:
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2273772,1,2&timerange=1209600&zoom=1587424271189,1587583567411,328.731543764074,2173.518433715652

I'm closing this as fixed, thanks!