2043983 - Allow gn_processor.py to prune unused source files

Status: RESOLVED FIXED

(Core :: Graphics, task)

Description

[Jamie Nicol [:jnicol]]

In bug 2035541 we plan to switch ANGLE's vendoring method to use the gn_processor.py script to generate the required moz.build files, rather than using the existing update-angle.py script to do so.

One feature the existing update-angle.py script has is that it queries the list of required files programmatically from GN and only vendors the required files. For a complex project such as ANGLE this is important as it drastically reduces the number of files vendored into the source tree.

We could alternatively make use of mach vendor's keep, include and exclude mechanism to prune the tree. But I've found these to be too coarse for our needs, resulting in lengthy and difficult to maintain exclude and include lists, while still not pruning the tree as much as the programmatic solution.

So we should extend gn_processor.py to be able to do this pruning for us. It should be opt-in so as not to affect the existing users of gn_processor.py, that presumably don't want to do this. Additionally we'll probably need a way to manually keep extra files, such as the LICENSE. I envisage both of these being implemented via new prune_unused_files: bool, and keep_files: list[str] options in the JSON config.

Additionally we'll want to keep the GN files that are used, allowing gn_processor.py to be run multiple times without having to run the entire vendoring workflow again. (Not essential, but probably helpful for developers if they are iterating on the script). Thankfully the list of required GN files can be obtained in the GN project.json in json["build_settings"]["gen_input_files"].

Comment 1

[Jamie Nicol [:jnicol]]

Attachment: Bug 2043983 - Allow gn_processor.py to prune unused files. r?glandium,mjf

The existing ANGLE vendoring script (update-angle.py) queries GN
programatically to obtain the list of files required to build ANGLE,
and only vendors those files. This is required to avoid vendoring
large numbers of files that are unused into the tree.

When switching ANGLE's vendoring to use gn_processor.py in bug 2035541
we will want to keep this functionality. The alternative of relying on
mach vendor's "keep", "include", and "exclude" functionality is
undesirable as these are too blunt, requiring long and difficult to
maintain file lists while still not pruning the tree precisely enough.

This patch therefore adds the ability to gn_processor.py to prune
unused files. This is gated behind a "prune_unused_files" JSON config
option, which defaults to false so as to not affect existing users of
the script.

During execution gn_processor builds a list of all required files,
then as a final step when the prune config option is enabled,
gn_processor will delete all files from the srcdir not in this list.

The list of required files includes:

• Files obtained from the generated GN project.json files,
  e.g. "sources", "script", and "inputs" fields from used targets.
• All the build files generated by gn_processor.
• The required input GN files, allowing gn_processor to be executed again.
• Any additional files specified by the "keep_files" config option (intended to
  be used for the LICENSE file, for example).

Comment 2

[Jamie Nicol [:jnicol]]

Attachment: Bug 2043983 - Extract most of gn_processor.py's main() into new process_gn_configs() function. r?glandium,mjf

And additionally return unprocessed GN configs from it in addition to
the processed mozbuild results.

This will allow ANGLE's vendoring script to perform some additional
work making use of the unprocessed GN configs, while still only
requiring a single pass to generate all the GN configs and moz.build
files.

Comment 3

[Jamie Nicol [:jnicol]]

Attachment: Bug 2043983 - Preserve "gen_input_files" and action "inputs" through filter_gn_config(). r?glandium,mjf

This will allow the ANGLE vendoring script to determine a list of all
files required to build ANGLE, which it will use to prune the set of
vendored files.

Note that "gen_input_files" is the list of GN input files from which
the configs were generated. While not technically required to build
ANGLE once vendored, preserving them will allow gn_processor to be ran
repeatedly, which can be useful if iterating on the config or script
itself.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9fc8e758e8d5
https://hg.mozilla.org/integration/autoland/rev/664af368e7ab
Extract most of gn_processor.py's main() into new process_gn_configs() function. r=mjf
https://github.com/mozilla-firefox/firefox/commit/a81689cb139b
https://hg.mozilla.org/integration/autoland/rev/c4e0a0a5221b
Preserve "gen_input_files" and action "inputs" through filter_gn_config(). r=mjf

Comment 5

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/664af368e7ab
https://hg.mozilla.org/mozilla-central/rev/c4e0a0a5221b