Closed Bug 1384568 Opened 7 years ago Closed 7 years ago

Produce a list of generated source files in the common backend

Categories

(Firefox Build System :: General, enhancement)

enhancement
Not set
normal

Tracking

(firefox56 fixed)

RESOLVED FIXED
mozilla56
Tracking Status
firefox56 --- fixed

People

(Reporter: ted, Assigned: ted)

References

Details

Attachments

(1 file)

To fix bug 1259832 we'll need to upload generated source files from each build. As a first step, we'll need to actually collect the list of generated source files somewhere. This is a little fiddly because we have multiple places that wind up generating source files:
* GENERATED_FILES that wind up in SOURCES or EXPORTS
* XPIDL_SOURCES
* IPDL_SOURCES
* WEBIDL_FILES
For informational purposes, I gathered a list of generated files that wind up in our debug symbols by doing `mach buildsymbols` followed by `rg --no-filename '^FILE \d+ (/build/debug-mozilla-central/.+)$' --replace '$1' /build/debug-mozilla-central/dist/crashreporter-symbols/ | rg -v 'rust/<' | sort -u > /tmp/symbols-files`

The second rg is there to grep out the bogus filenames Rust gives to macro code, like:
/build/debug-mozilla-central/toolkit/library/rust/<print macros>

This produced a list of 3142 files. There are 17 files in the list that don't actually exist:
/build/debug-mozilla-central/gfx/angle/glslang_lex.cpp
/build/debug-mozilla-central/gfx/angle/Tokenizer.l
/build/debug-mozilla-central/gfx/harfbuzz/src/hb-ot-shape-complex-indic-machine.hh
/build/debug-mozilla-central/gfx/harfbuzz/src/hb-ot-shape-complex-indic-machine.rl
/build/debug-mozilla-central/gfx/harfbuzz/src/hb-ot-shape-complex-myanmar-machine.hh
/build/debug-mozilla-central/gfx/harfbuzz/src/hb-ot-shape-complex-myanmar-machine.rl
/build/debug-mozilla-central/gfx/harfbuzz/src/hb-ot-shape-complex-use-machine.hh
/build/debug-mozilla-central/gfx/harfbuzz/src/hb-ot-shape-complex-use-machine.rl
/build/debug-mozilla-central/gfx/harfbuzz/src/NONE
/build/debug-mozilla-central/gfx/skia/layout.flex
/build/debug-mozilla-central/gfx/skia/lex.layout.c
/build/debug-mozilla-central/gfx/skia/lex.sksl.c
/build/debug-mozilla-central/gfx/skia/sksl.flex
/build/debug-mozilla-central/security/nss/cmd/modutil/modutil_modutil/installparse.l
/build/debug-mozilla-central/security/nss/cmd/modutil/modutil_modutil/installparse.y
/build/debug-mozilla-central/security/nss/cmd/modutil/modutil_modutil/lex.Pk11Install_yy.cpp
/build/debug-mozilla-central/security/nss/cmd/modutil/modutil_modutil/ytab.c


I assume most of those are an artifact of importing generated code from upstream projects.

The remaining 3125 files that exist sum to about 65MB total on disk.

With the patch I've written, comparing that list against the sources listed in generated-sources.json finds the following things missing:
* Everything in $objdir/ipc/ipdl/_ipdlheaders. Unfortunately there doesn't seem to be a way to get that list without actually parsing the IPDL files. I guess we could expand on the list of generated sources during build time to capture that.
* A few random files that don't wind up in SOURCES or EXPORTS:
  dist/include/nsStyleStructList.h
  js/src/selfhosted.out.h
  toolkit/library/x86_64-unknown-linux-gnu/debug/build/gleam-0517e247242520ab/out/gles_bindings.rs
  toolkit/library/x86_64-unknown-linux-gnu/debug/build/gleam-0517e247242520ab/out/gl_bindings.rs
  js/src/gc/StatsPhasesGenerated.cpp
  toolkit/library/x86_64-unknown-linux-gnu/debug/build/gleam-0517e247242520ab/out/gl_and_gles_bindings.rs
  dist/include/xpcAccEvents.h
  js/src/frontend/ReservedWordsGenerated.h

There are a couple of dist/include paths there, which seemed odd because symbolstore.py has code to read the dist/include install manifest and map files back to their original locations:
https://dxr.mozilla.org/mozilla-central/rev/7d2e89fb92331d7e4296391213c1e63db628e046/toolkit/crashreporter/tools/symbolstore.py#323

...but following things back, the emitter just uses `add_optional_exists` for EXPORTS that come from the objdir, so it doesn't actually record any information about where those files came from:
https://dxr.mozilla.org/mozilla-central/rev/7d2e89fb92331d7e4296391213c1e63db628e046/python/mozbuild/mozbuild/backend/recursivemake.py#1399

I'll file a separate bug to fix that case, but it looks like it only actually affects two headers.

The remaining things in that list are mostly headers+sources that are generated but not listed in SOURCES/EXPORTS, which must mean they're just #included from wherever they get generated. Maybe it'd be better to simply capture all files in GENERATED_FILES that have a source-file-like extension?

There are also a few generated Rust sources which we don't have any visibility into currently, since they get generated from cargo build scripts. I don't know of any useful way to capture this currently. When cargo adds support for more declarative build scripts in Cargo.toml it should be possible to read that info.
Comment on attachment 8890378 [details]
bug 1384568 - Write a generated-sources.json.

https://reviewboard.mozilla.org/r/161496/#review167102

Seems reasonable and more than a decent first implementation!

::: python/mozbuild/mozbuild/backend/common.py:185
(Diff revision 1)
>          self._idl_manager = XPIDLManager(self.environment)
>          self._webidls = WebIDLCollection()
>          self._binaries = BinariesCollection()
>          self._configs = set()
>          self._ipdl_sources = set()
> +        self._generated_sources = []

It feels like a set would be better?
Attachment #8890378 - Flags: review?(gps) → review+
Comment on attachment 8890378 [details]
bug 1384568 - Write a generated-sources.json.

https://reviewboard.mozilla.org/r/161496/#review167102

> It feels like a set would be better?

You're probably right. I'll change that.
> I'll file a separate bug to fix that case, but it looks like it only actually affects two headers.

bug 1384889
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=269ceba878a8dc1c02a74da958b70477f0192eb4

The Windows builds were failing test_ipdl_sources in test_recursivemake.py, it turns out I had left some bad path handling in common.py:
+        self._handle_generated_sources(mozpath.relpath(mozpath.join(ipdl_dir, f), self.environment.topobjdir) for f in ipdl_cppsrcs)

That should be:
+        self._handle_generated_sources(mozpath.join(ipdl_dir, f) for f in ipdl_cppsrcs)

to pass absolute paths into `_handle_generated_sources`. I fixed that locally and will do another try push to make sure that fixes the tests on Windows.
https://hg.mozilla.org/mozilla-central/rev/1b1cbbf483f6
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Assignee: nobody → ted
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.