Closed Bug 1676346 Opened 1 year ago Closed 9 months ago

[meta] Some efforts to reduce the aggregated number of included files

Categories

(Firefox Build System :: General, task)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: sg, Assigned: sg)

References

(Blocks 1 open bug)

Details

(Keywords: meta)

Attachments

(5 files, 1 obsolete file)

This bug is a meta bug to aggregate some efforts to reduce the aggregated number of included files across the code base, in order to reduce the build times on both incremental and clobber builds.

Preliminary measurements of these efforts show:

  • Clang frontend time for a clobber build reduces by 12.6% from 7487s to 6541s
  • Wall time for a clobber build reduces by 6% from 881s to 828s (the effect is less visible here e.g. because of the significant impact of Rust compilation)
  • Aggregated number of include files reduces by 21% from 2.8 million to 2.2 million
  • Number of unified translation units affected by a change to dom/base/Document.h reduces by 52% from 1161 to 557, build wall time reduces by 49% from 535s to 274s
  • Number of unified translation units affected by a change to js/StructuredClone.h reduces by 34% from 1290 to 855, build wall time reduces by 29% from 540s to 382s
  • Number of unified translation units affected by a change to tools/profiler/public/GeckoProfiler.h reduces by 36% from 1541 to 987, build wall time reduces by 27% from 571s to 417s (this still seems to be quite a bit more than necessary)
  • Build wall time after adding a message to PBrowser.ipdl reduces by 10% from 170s to 154s

All numbers are for Linux debug builds on my workstation.

Depends on: 1660470, 1673424, 1673931
Depends on: 1676356
Depends on: 1676357
Depends on: 1676361
Depends on: 1676362
Depends on: 1676363
Depends on: 1676364
Depends on: 1676365
Depends on: 1676369
Depends on: 1676370
Depends on: 1676372
Depends on: 1676373
Assignee: nobody → sgiesecke
Status: NEW → ASSIGNED
An overview of how the individual patches affect the aggregated number of included headers is in attach
Attachment #9186960 - Attachment is obsolete: true

An overview of how the individual patches affect the aggregated number of included headers is found in the attachment.

Note that patches may build upon the effects of earlier patches, so removing a patch that individually doesn't have a significant impact may have adverse effects.

See Also: → 1651951

excellent work, bravo
can you share how you generated this?

Script to determine incremental build times, based on some scenarios of modifications. The last scenario requires files to be provided for the current revision manually, the other just touch existing files to trigger rebuilds.

Attached file analyze-build-times.sh

Script for measuring clobber build time.

ClangBuildAnalyzer is from https://github.com/aras-p/ClangBuildAnalyzer

The mozconfig must contain CXXFLAGS="$CXXFLAGS -ftime-trace"

Attached file list_includes.py

Script to determine the aggregated number of included files across the build.

It's disregarding the unified build and uses the individual files instead. It counts included files only once per translation unit (if they are properly guarded by header guards).

The script does a bit more than that, which I played around with, and I only partially removed the code for that. Sorry for the confusion.

(In reply to Sylvestre Ledru [:Sylvestre] from comment #4)

excellent work, bravo

Thanks!

can you share how you generated this?

Sure :) Unfortunately, I don't have that in a state that can be readily applied by everyone, as they contain some hard-coded local stuff, but I'll still share what I used. There are two shell scripts and one python script I attached. To generate the table, I ran the python script as:

for a in $(hg ls | tail -n +2 |  cut -c 1-6) ; do hg up $a >/dev/null ; ./mach build export 2>/dev/null >/dev/null ; echo $(hg ls | grep $a) $(~/src/fftools/list_includes.py 2>/dev/null) ; done | tee include-counts.txt
Depends on: 1677466
Depends on: 1677529
Depends on: 1677541
Depends on: 1677542
Blocks: 1677549
Depends on: 1678062

With further patches, which are not yet on Phabricator,

  • Wall time for a clobber build is now reduced to 799s
  • Frontend time for a clobber build is now reduced to 6399s
  • Aggregated number of included files is now reduced to 2.03 million
Depends on: 1678374
Depends on: 1678560
Depends on: 1678561
Depends on: 1678562
Keywords: leave-open
Pushed by sgiesecke@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7eac42a99651
Add missing include directives etc. r=andi,necko-reviewers,valentin
See Also: → 1682477

All blocking bugs are now resolved, so this specific effort is complete.

Status: ASSIGNED → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
See Also: → 1707213
You need to log in before you can comment on or make changes to this bug.