2035532 - Support generating angle's build scripts using gn_processor.py

Status: RESOLVED FIXED

(Core :: Graphics, task)

Description

[Jamie Nicol [:jnicol]]

Currently we build ANGLE's shader translator for all platforms, but only build libEGL/libGLESv2 on windows (with the D3D11 backend).

We'd like the ability to build libEGL/libGLESv2 with different backends on other platforms, e.g. Vulkan on Android (bug 1984486), or Metal on macOS (bug 2027951).

ANGLE is currently both vendored into tree and has its moz.build files generated using the update-angle.py script. This must be run on Windows. It works by generating the GN config for the Windows/Direct3D11 build, generating the moz.build files from that, making a few adjustments to ensure the translator will build on all platforms.

This only works because the translator is simple, it doesn't scale to building libEGL/libGLESv2 with different backends on different platforms. To solve this we'll need to do a gn gen for each different target, merge the results somewhow, and generate moz.build files from that.

Luckily there's a script in tree which does that already: gn_processor.py

This bug tracks switching Angle's build script generation to use gn_processor.py, for now just in the current configuration of the translator on all platforms and libEGL/libGLESv2 on Direct3D11 on windows-only. gn_processor.py will need some adjustments to support this. Follow up bugs will add support for additional backends on other platforms.

Comment 1

[Jamie Nicol [:jnicol]]

I'm still figuring out the story for actually vendoring the ANGLE sources into tree (bug 2035541) but the gn_processor side of things is coming together. The script will need a few fixes, most of which I don't think should be too controversial but I'm going to put them up for review shortly to get some feedback.

The gn_processor/mozbuild changes that I expect will be a bit more controversial will be adding support for GN's response_file_contents which I'm still working on, and may split out into a separate bug.

Comment 2

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Allow specifying different targets per OS. r?mjf

ANGLE will require building different GN targets depending on the
target OS, e.g. "//:libEGL" for Windows and "//:translator" elsewhere.

This updates the "gn_target" field of the config so that it can either
be a string like it currently is (which applies to all target OSes),
or it can be a dict. The dict should contain an "*" entry specifying
the default target, as well as optionally an entry for each OS on
which the target should be overridden.

For example, the configuration for ANGLE described above would look
like:

"gn_target": {
"*": "//:translator",
"win": "//:libEGL"
}

Comment 3

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Allow specifying additional GN args in config. r?mjf

ANGLE will require specifying additonal args to GN, some of which will
depend on the target OS. To allow this, this patch adds the "gn_args"
field to the config.

This should be a dict containing a "*" item, itself dict of default
args, and optionally additional per-target_os dicts of args. The
following config would, for example, disable ANGLE's D3D11 backend on
all OSes, except for Windows on which it would be enabled.

"gn_args": {
"*": {
"angle_enable_d3d11": false,
},
"win": {
"angle_enable_d3d11": true,
}
}

Comment 4

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Ensure strings containing quotes are serialzed correctly. r?mjf

ANGLE requires DEFINES to be emitted that contain quotes.
mb_serialize() currently serializes strings simply by wrapping the
value in double quotes, resulting in invalid statements such as this:

DEFINES["ANGLE_MESA_EGL_LIBRARY_NAME"] = ""mesa/src/egl/libEGL""

Instead, the inner quotes need to be escaped. This patch achieves this
by using json.dumps(), which will produce this:

DEFINES["ANGLE_MESA_EGL_LIBRARY_NAME"] = ""mesa/src/egl/libEGL""

Comment 5

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Handle .hpp, .def, and .rc inputs correctly. r?mjf

.hpp files are headers and should be excluded from (UNIFIED_)SOURCES
like we currently do for .h and .inc files.

.def files should be emitted as DEFFILE rather than SYMBOLS_FILE,
and should be relative to the root source dir.

.rc files should be emitted as RCFILE rather than included in
(UNIFIED_)SOURCES by default.

Comment 6

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Emit USE_LIBS from GN deps when FINAL_LIBRARY is unused. r?mjf

Existing gn_processor.py users rely on FINAL_LIBRARY = "xul", so
they do not need explicit USE_LIBS statements. For ANGLE, however, we
cannot use FINAL_LIBRARY for its generated targets: some targets end
up in libxul, some only in shared libraries, and some in both. We must
therefore rely on explicit USE_LIBS statements rather than using
FINAL_LIBRARY.

Retain GN group targets through filtering so they can act as
dependency carriers, but avoid emitting moz.build contexts for them
during processing. During processing iterate each target's GN deps,
recursing through groups, and emit the reachable library targets in
USE_LIBS.

In order to avoid churn of existing gn_processor-generated build
scripts, only emit USE_LIBS when FINAL_LIBRARY is not set.

Comment 7

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Add mozbuild_variable to force linking against gecko's chosen zlib. r?mjf

This is modelled after the existing variables used by gn_processor to
handle linking to system libraries, e.g. INCLUDE_SYSTEM_GBM_HANDLING.

When applied to a target, adds "zlib" to USE_LIBS. This target
already handles linking either to in-tree or system zlib depending
upon the build configuration.

This is intended to be used for ANGLE's zlib_gn target, which we
will stub out rather than compiling chromium's zlib.

Comment 8

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Derive shared library filenames from GN's `output_name`. r?mjf

gn_processor.py suffixes GN target names with "_gn" to avoid
collisions with in-tree targets. That derived name becomes the
LIBRARY_NAME, which would by default determine the shared library
output name.

For shared_library targets, additionally emit SHARED_LIBRARY_NAME
as derived from GN's output_name field so that libraries keep their
expected runtime names, allowing consumers to load them as expected.

On non-Windows platforms, strip any leading "lib" from the output
name, as mozbuild will add the prefix later. On Windows, however, if
the output_name contains such a prefix we must keep it, as it is an
intended part of the DLL name.

As a concrete example, ANGLE contains a shared library target with the
output_name "libEGL". This patch ensures that this results in a
library name such as "libEGL.so" or "libEGL.dll", rather than
"libEGL_gn.so", "liblibEGL.so" or "EGL.dll".

Comment 9

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Use GeckoSharedLibrary template for shared library targets. r?mjf

For ANGLE's libGLESv2 and libEGL targets this matches the existing
configuration, ensuring we link against mozglue so don't get linker
errors about missing moz_malloc symbols.

Since no existing users of gn_processor contain shared library
targets, making GeckoSharedLibrary the default seemed the easiest
solution. If we require non-Gecko SharedLibrary targets in the
future we could add a configuration option.

Comment 10

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Include MOZ_DEBUG in conditional DIRS generation. r?mjf

When generating the top-level DIRS list, account for MOZ_DEBUG the
same way we already do for other conditional moz.build variables.

Without this, directories that contain targets only used in debug
builds are omitted from the root moz.build's DIRS list, which then
causes an error when said targets appear in a USE_LIBS.

Comment 11

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - Add mozbuild support for response files used by GN-generated actions. r?mjf,glandium

Some GN actions pass large lists of input files through a response
file rather than directly on the command line. We need to support this
in order to build ANGLE with moz.build scripts generated by
gn_processor.py.

This adds a response_file_contents attribute to GENERATED_FILES
and the GeneratedFile() template. This is a list of mozbuild style
strings, e.g. the "/" prefix denoting a path relative to the root
source dir. The emitter turns those entries into mozbuild Path
objects, and the make backend writes them to a file in the objdir,
then substitutes said file's path into the action flags via the
{{response_file_name}} placeholder.

The reason this requires first class support in mozbuild rather than
being handled purely in gn_processor.py, is because the response file
contents in this case is a list of paths in the source tree. At build
time the script will be executed from a directory within the
objdir. At vendor time (when gn_processor runs), however, we have no
way of knowing either the absolute path of the source tree or the
relative path from the objdir to the source dir.

Note that a response file can technically contain arbitrary data, but
we only need to support the case where it is a list of paths.
Supporting only this case is simpler than a generic solution that
handles opaque data as well as when required fixing the paths issue
described above.

Comment 12

[Jamie Nicol [:jnicol]]

Attachment: Bug 2035532 - gn_processor.py: Support response_file_contents field of GN actions. r?mjf,glandium

Adds a response_file_contents field to the GeneratedFile
statements emitted for GN actions, if present.

This assumes the response file contents are always a list of paths,
which is not universally true but is the case for the single
response-file-using action that we must support in order to build
ANGLE. The paths obtained from GN must be translated into
mozbuild-style paths. (E.g. prefixed with "/" to denote relativity to
the root source dir.)

Additionally the raw project.json path rewriting step must be made
more specific as the previous "{target_dir}/" replacements were
modifying the paths in the response file contents, making them point
at files that do not actually exist.

Lastly, when finding common attrs when writing the moz.build files,
GeneratedFile dicts must be treated atomically.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/bd208087f1c6
https://hg.mozilla.org/integration/autoland/rev/c24d7b6a2c04
gn_processor.py: Allow specifying different targets per OS. r=mjf
https://github.com/mozilla-firefox/firefox/commit/51b89ae07068
https://hg.mozilla.org/integration/autoland/rev/c4c745947f13
gn_processor.py: Allow specifying additional GN args in config. r=mjf
https://github.com/mozilla-firefox/firefox/commit/ad6e9a5b466a
https://hg.mozilla.org/integration/autoland/rev/7a3099bb1105
gn_processor.py: Ensure strings containing quotes are serialzed correctly. r=mjf
https://github.com/mozilla-firefox/firefox/commit/534704c0d241
https://hg.mozilla.org/integration/autoland/rev/b6b5c12ebef4
gn_processor.py: Handle .hpp, .def, and .rc inputs correctly. r=mjf
https://github.com/mozilla-firefox/firefox/commit/b10828577e7b
https://hg.mozilla.org/integration/autoland/rev/ce1169c25792
gn_processor.py: Emit USE_LIBS from GN deps when FINAL_LIBRARY is unused. r=mjf
https://github.com/mozilla-firefox/firefox/commit/6d681955ea85
https://hg.mozilla.org/integration/autoland/rev/11ebe590a6aa
gn_processor.py: Add mozbuild_variable to force linking against gecko's chosen zlib. r=mjf
https://github.com/mozilla-firefox/firefox/commit/9f861ea435c0
https://hg.mozilla.org/integration/autoland/rev/6e200001adcb
gn_processor.py: Derive shared library filenames from GN's `output_name`. r=mjf
https://github.com/mozilla-firefox/firefox/commit/5490f68917e4
https://hg.mozilla.org/integration/autoland/rev/08159bcbfe5c
gn_processor.py: Use GeckoSharedLibrary template for shared library targets. r=mjf
https://github.com/mozilla-firefox/firefox/commit/4fd1a50146f2
https://hg.mozilla.org/integration/autoland/rev/6c1f940d8e2d
gn_processor.py: Include MOZ_DEBUG in conditional DIRS generation. r=mjf

Comment 14

[rperta]

https://hg.mozilla.org/mozilla-central/rev/c24d7b6a2c04
https://hg.mozilla.org/mozilla-central/rev/c4c745947f13
https://hg.mozilla.org/mozilla-central/rev/7a3099bb1105
https://hg.mozilla.org/mozilla-central/rev/b6b5c12ebef4
https://hg.mozilla.org/mozilla-central/rev/ce1169c25792
https://hg.mozilla.org/mozilla-central/rev/11ebe590a6aa
https://hg.mozilla.org/mozilla-central/rev/6e200001adcb
https://hg.mozilla.org/mozilla-central/rev/08159bcbfe5c
https://hg.mozilla.org/mozilla-central/rev/6c1f940d8e2d

Comment 15

[Jamie Nicol [:jnicol]]

Sorry, I should have marked as leave-open. There are still patches to land here.

Comment 16

[Jamie Nicol [:jnicol]]

Markus made me feel sufficiently guilty about using "leave-open" so I'm splitting the remaining work out into bug 2043320

Comment 17

[Phabricator Automation]

Comment on attachment 9586914 [details]
Bug 2035532 - Add mozbuild support for response files used by GN-generated actions. r?mjf,glandium

Revision D300873 was moved to bug 2043320. Setting attachment 9586914 [details] to obsolete.

Comment 18

[Jamie Nicol [:jnicol]]

Retitling the bug and removing the dependency, as this became about adding support to gn_processor.py for generating angle's build scripts, rather than actually switching to it. Actually switching to using gn_processor.py will happen in bug 2035541