1834815 - Use full LTO + cross LTO for shippable binaries

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[[:sergesanspaille]]

Using FullLTO combined with cross LTO yields faster binaries on Linux and Windows, as hinted by perfherder results:

• windows: https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=acf7d14780a648bb943fe3a7673a72018823902f&newProject=try&newRevision=fffabd808634347a15c1a34708191b8c48ea3a1e&page=1&framework=13

• linux: https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=784d660e1bee8e3cddfd1605eec7d8765bed525f&newProject=try&newRevision=9db61ace6ab11da941871edf37241b114b761fcb&page=1&framework=13

For our builder to be able to compile fx in full lto, we cannot use the same debug level as we used to, as it generates too much debuginfo. Alternatives are

• -g0: disable debug
• -gline-tables-only
• -g1

Comment 1

[Mike Hommey [:glandium]]

What do build times look like?

Comment 2

[[:sergesanspaille]]

Extra data point : not so good on macosx64 (had to disable debug and increase some timeouts to get this one)

https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=c20256199267d457786959f00a52cde8537df85b&newProject=try&newRevision=51e20f33c33a925f96ecc7e4050a7a24ef01fe68&page=1&framework=13#tableLink-row-3059102041

Comment 3

[[:sergesanspaille]]

Wrt. build times, we're not doing the runs on the same machines (using xlarge suffixes setup for FullLTO builds) yet we see a big impact on compilation time (just for the final FullLTO + PGO runs). All these digits are taken from the above runs.

linux: from 50m to 65m
win64: from 46m to 80m
osx64 (aarch64): 46m to 72m

Given an infinite number of user and a finite number of builds, the overall gain is in favor of spending more time compiling and less time running, but this also has a great impact on automation and developer time.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Which builds will need this full level of optimization? And how many of such builds are done a day?

My guess is that this would be 'shippable' builds. If there's a strong correlation between normal PGO builds and these builds (and I imagine there is, though not 100% correlated), we could do most performance tests against regular PGO builds, and run these builds less often (perhaps only on Nightly builds), or once an hour, or some such. The per-checkin-ish builds are mostly looking for significant performance wins/losses from a checkin; almost all of those would be correlated across opt/PGO/LTO in practice. In the odd case where a regression wasn't picked up in PGO but showed up in LTO a) it's unlikely to be large, b) we can do regression runs to bisect and find the checkin that caused it.

This would require another type of build in addition to PGO, and slightly complicate our monitoring and sheriffing.

Comment 5

[[:sergesanspaille]]

Attachment: Bug 1834815 - Enable full+cross LTO by default for shippable Linux and Windows build r=glandium

This has an important impact on compilation time, so we limit that to
shippable builds.

The benefit on speedometer time is not significant on macosx build, so
we skip that platform.

Comment 6

[[:sergesanspaille]]

Thanks Jesup, I've pushed a first patch that activates full lto for all pgo builds. I'm also concerned by the impact on build time. Glandium, would that be okay to limit that to builds with the shippable suffix?

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/502f57bb0ebb
Enable full+cross LTO by default for shippable Linux and Windows build r=glandium

Comment 8

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/502f57bb0ebb

Comment 9

[Alex Finder]

(In reply to Pulsebot from comment #7)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/502f57bb0ebb
Enable full+cross LTO by default for shippable Linux and Windows build
r=glandium

== Change summary for alert #38934 (as of Fri, 07 Jul 2023 01:51:54 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
3%	wikipedia LastVisualChange	windows10-64-shippable-qr	bytecode-cached fission warm webrender	761.26 -> 739.12	Before/After
3%	webaudio	windows10-64-shippable-qr	fission webrender	96.71 -> 94.25	Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=38934