1676346 - [meta] Some efforts to reduce the aggregated number of included files

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[Simon Giesecke [:sg] [he/him]]

This bug is a meta bug to aggregate some efforts to reduce the aggregated number of included files across the code base, in order to reduce the build times on both incremental and clobber builds.

Preliminary measurements of these efforts show:

• Clang frontend time for a clobber build reduces by 12.6% from 7487s to 6541s
• Wall time for a clobber build reduces by 6% from 881s to 828s (the effect is less visible here e.g. because of the significant impact of Rust compilation)
• Aggregated number of include files reduces by 21% from 2.8 million to 2.2 million
• Number of unified translation units affected by a change to dom/base/Document.h reduces by 52% from 1161 to 557, build wall time reduces by 49% from 535s to 274s
• Number of unified translation units affected by a change to js/StructuredClone.h reduces by 34% from 1290 to 855, build wall time reduces by 29% from 540s to 382s
• Number of unified translation units affected by a change to tools/profiler/public/GeckoProfiler.h reduces by 36% from 1541 to 987, build wall time reduces by 27% from 571s to 417s (this still seems to be quite a bit more than necessary)
• Build wall time after adding a message to PBrowser.ipdl reduces by 10% from 170s to 154s

All numbers are for Linux debug builds on my workstation.

Comment 1

[Simon Giesecke [:sg] [he/him]]

Attachment: Aggregated number of included files per patch

An overview of how the individual patches affect the aggregated number of included headers is in attach

Comment 2

[Simon Giesecke [:sg] [he/him]]

Attachment: Aggregated number of included files per patch

An overview of how the individual patches affect the aggregated number of included headers is in attach

Comment 3

[Simon Giesecke [:sg] [he/him]]

An overview of how the individual patches affect the aggregated number of included headers is found in the attachment.

Note that patches may build upon the effects of earlier patches, so removing a patch that individually doesn't have a significant impact may have adverse effects.

Comment 4

[Sylvestre Ledru [:Sylvestre]]

excellent work, bravo
can you share how you generated this?

Comment 5

[Simon Giesecke [:sg] [he/him]]

Attachment: analyze-build-times-incremental.sh

Script to determine incremental build times, based on some scenarios of modifications. The last scenario requires files to be provided for the current revision manually, the other just touch existing files to trigger rebuilds.

Comment 6

[Simon Giesecke [:sg] [he/him]]

Attachment: analyze-build-times.sh

Script for measuring clobber build time.

ClangBuildAnalyzer is from https://github.com/aras-p/ClangBuildAnalyzer

The mozconfig must contain CXXFLAGS="$CXXFLAGS -ftime-trace"

Comment 7

[Simon Giesecke [:sg] [he/him]]

Attachment: list_includes.py

Script to determine the aggregated number of included files across the build.

It's disregarding the unified build and uses the individual files instead. It counts included files only once per translation unit (if they are properly guarded by header guards).

The script does a bit more than that, which I played around with, and I only partially removed the code for that. Sorry for the confusion.

Comment 8

[Simon Giesecke [:sg] [he/him]]

(In reply to Sylvestre Ledru [:Sylvestre] from comment #4)

excellent work, bravo

Thanks!

can you share how you generated this?

Sure :) Unfortunately, I don't have that in a state that can be readily applied by everyone, as they contain some hard-coded local stuff, but I'll still share what I used. There are two shell scripts and one python script I attached. To generate the table, I ran the python script as:

for a in $(hg ls | tail -n +2 |  cut -c 1-6) ; do hg up $a >/dev/null ; ./mach build export 2>/dev/null >/dev/null ; echo $(hg ls | grep $a) $(~/src/fftools/list_includes.py 2>/dev/null) ; done | tee include-counts.txt

Comment 9

[Simon Giesecke [:sg] [he/him]]

With further patches, which are not yet on Phabricator,

• Wall time for a clobber build is now reduced to 799s
• Frontend time for a clobber build is now reduced to 6399s
• Aggregated number of included files is now reduced to 2.03 million

Comment 10

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1676346 - Add missing include directives etc. r=andi

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7eac42a99651
Add missing include directives etc. r=andi,necko-reviewers,valentin

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/7eac42a99651

Comment 13

[Simon Giesecke [:sg] [he/him]]

All blocking bugs are now resolved, so this specific effort is complete.