623617 - Derecursify the build system

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect, P2)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

For prototyping the proposal at http://etherpad.mozilla.com:9000/OvUWA6CtDt

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: Checkpoint

This applies to df3c1150dd7a and is enough to build xpidl, xpcom/glue, and bits and pieces of xpcom/ (mainly those that don't depend on IDL generated headers from things like xpconnect or anything from js/src).

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: Checkpoint

This is enough to build libsqlite3.so (though not with the right optimization flags!) and a bunch of other pieces of the tree are also converted (netwerk/, intl/, etc)

Comment 3

[Joey Armstrong [:joey]]

Attachment: container makefiles: prototyping for non-recursive sandbox traversal

I have a working prototype for the non-recursive make/builds to try out when you have time.  Directories are subject to change but for now sources are isolated for easy access.  The setup is based on a config that I used with another build system and
was solid in production.


Sources
==========================================
makefiles/makefilegen.pl
makefiles/makefilegen.pm
  - unit test: makefiles/test

% gmake -f client.mk config_bundle

Cookbook
==========================================
  o traverse the target directory (obj-ff-dbg)
  o gather /*akefile*/
  o for a given list of targets
     - generate a container makefile beneath obj-ff-dbg/generate
     - makefile.gen.${target}
     - invoke as:  make -C obj-ff-dbg -f generate/makefile.gen.${target}
   o Only makefiles containing a given target will be included in the
container makefile
   o config_bundle will exit early after makefiles have been generated

Advantages
==========================================
   o scalable - no overhead when adding new targets, simply generate a new container makefile.
   o in theory deps can be used to control when the tool is invoked -- so only as-needed
   o functionality is isolated allowing subdirectories to be targeted w/o affecting the entire sandbox
   o exclusion lists can be used to work around problem makefiles and issues
   o logic is triggered by the presence of makefile targets so we can be very selective about when and where transitions happen
   o makefiles are generated from templates so behavior can easily be altered on a per-target basis when needed.

Unit tests
======================
   % gmake -C makefiles/test clean manual

Comment 4

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Comment on attachment 537797 [details] [diff] [review]
container makefiles: prototyping for non-recursive sandbox traversal

I looked at this yesterday ... there's a bunch of machinery here but no examples of what Makefiles would look like, as far as I can tell.

On a mostly unrelated note, while I'm fine with writing simple little scripts in perl, I don't think we're going to want to write a core piece of the build system in perl.

Comment 5

[Joey Armstrong [:joey]]

(In reply to comment #4)
> Comment on attachment 537797 [details] [diff] [review] [review]
> container makefiles: prototyping for non-recursive sandbox traversal
> 
> I looked at this yesterday ... there's a bunch of machinery here but no
> examples of what Makefiles would look like, as far as I can tell.

A writeup and examples are starting to materialize here:
http://wiki.mozilla.org/User:JoeyArmstrong/makefiles/makefilegen
http://wiki.mozilla.org/User:JoeyArmstrong/makefiles/ContainerMakefileSample
 
> On a mostly unrelated note, while I'm fine with writing simple little
> scripts in perl, I don't think we're going to want to write a core piece of
> the build system in perl.

ps> Writing the tool in perl is just a first pass at functional logic and more importantly having a set of unit tests available as the tool evolves.  Final target language will be python.

Comment 6

[Benjamin Smedberg]

So we're definitely talking about derecursifying using make, instead of ditching make entirely?

I really wish we'd move the entire export phase out of make entirely (relatively straightforward, big perf win), and then we can just derecursify the compilation/linking which make is reasonably good at.

Comment 7

[Joey Armstrong [:joey]]

> I really wish we'd move the entire export phase out of make entirely
> (relatively straightforward, big perf win), and then we can just derecursify
> the compilation/linking which make is reasonably good at.

It should be easy enough to move things around given most of the logic is behind targets and rules.  Containers will help collect everything into one place and help isolate dependencies in the sandbox.

Though I would like to check on the other export target bug that is open to introduce dependencies and have make handle the targets.  Launching make twice on a sandbox for a dependency build should mostly be a NOP but we currently are invoking nsinstall 1788 times/shells spawned with each attempt.

Comment 8

[Joey Armstrong [:joey]]

Attachment: refresh makefilegen/container makefile patch

Patch updated with the latest edits.

moved generation and processing logic out of rules.mk and into config/makefiles/makefilegen.mk and container.mk.
makefile deps added so makefilegen will only be launched when needed.
modified configure target to include container generation as a dependency.

added $(container_make) as a dependency of all & default targetets allowing
container makefiles to be processed before make would process the normal
export/libs/tools targets.

helper script config/makefiles/addtgts added for prototyping.  Script will append moz_ makefile targets in bulk to all makefiles existing beneath a given list of directories.

Added callbacks in the makefile template parser for added flexibility.
syntax <|var=token^makefile_syntax_for_token|> added to support free-form
makefile syntax and macro assignments in additon to returning a simple list.
This allows writing dependency lists in a way that individual list entries
can be commented out while debugging.

Comment 9

[Joey Armstrong [:joey]]

Attachment: run addtgt in debug mode, disable some other stray output

Comment 10

[Joey Armstrong [:joey]]

Attachment: Latest batch of container makefile edits

client.mk - configure target modified to call makefilegen then invoke
generated container makefiles.  Setup is temporary, container_make
will eventually need to become a dependency of realbuild but need to be sure container targets are processed early.

Exclusion list added to allow skipping problematic and platform dependent
makefiles: accessibility/public/{ia2,msaa}, dbm/src/Makefile.win

Hack added to workaround building xpidl, pyxpidl will deprecate and work around ugly dependencies.
Processing logic moved into makefilegen.mk and container.mk for modularity.

container makefiles will now pass CONTAINER_MAKE=1.  Flag initially will
be used to inhibit recursive traversal by makefiles that trip over unmet
dependency problems.

New makefile templates added for export and tools.  As a workaround, added
explicit dependencies to resolve processing order problems.  Exported deps
will be generated before they are needed for compiling.

New makefile template makefile.idl2h.tmpl added to support gathering and
processing arbitrary file types, in addition to makefile::target.

tools/joey/adtgts.pl - can be used to insert moz_* makefile targets where
needed in the sandbox.

Comment 11

[Joey Armstrong [:joey]]

Attachment: Latest batch of container makefile edits

1) targets can now be suffixed with a priority modifier to support ordered processing.  Target ex: moz_export_nodeps, moz_export_level20, moz_export_post(=level_55).  default priority is 50.
  - Added mainly to force xpcom/base header generation very early.
  - Logic will need to be updated to also work with TIER = makefile definitions.

2) Directories containing /tests?/ in the path will be sorted to the end of the list of makefiles to be processed.  Generate all exported headers before attempting to run tests that are dependent on them.

3) Added makefile dependencies within the export template makefile that will allow make to cleanly resolve directory inter-dependencies for header/export generation.

4) Derive generated makefile paths for the exclusion list.  Avoids copy/paste typos when '.in' is not removed from Makefile.in.

5) Added a template for handling ipdl files, processing logic to be implemented.

Comment 12

[Joey Armstrong [:joey]]

Attachment: macros and targets supporting thread safe mkdir w/deps

1) Fixed a chicken-n-egg problem.  Some automake makefiles are generated during recursive traversal.  This ticket inhibits recursion so makefiles may not exist when file gathering is launched after configure.  Workaround: always derive paths to generated makefiles.

2) Added a container makefile to process make-make/automake targets in bulk so all makefiles can be generated prior to a container build.

3) Added platform.mk - includable makefile that defines a few platform specific macros based on values declared in autoconf.mk

4) Platform specific makefiles are now wrappered by ifdef${ARCH} conditionals so unrelated files can be completely excluded from processing.

5) ifdef ENABLE_TESTS conditionals are also wrappered around /tests?/ directories so they can be properly filtered independent of ENABLE_TESTS being burned into each makefile.

6) Added a stub container for gathering/processing xpidl files

7) makefile.moz_exports.tmpl - adding more dependencies so make can build exported headers cleanly.

Comment 13

[Gregory Szorc [:gps]]

I /think/ my work in bug 687388 is starting to significantly overlap with things joey is doing here. If you haven't seen bug 687388 yet and are interested in build system improvements, I highly recommend checking it out.

I'll reach out to joey so we can coordinate efforts.

If you want to see something cool, check out https://gist.github.com/1363034. That builds for me on Linux using GNU make. Doing a `make -s -j8 export` for the headers and IDLs in the current system is ~25s wall time. The linked make file does it in about half that. No-op build time (which is a true make no-op build) completes in ~0.5s. Current implementation is ~5.5s.

Comment 14

[Joey Armstrong [:joey]]

closing this bug off, scope was for an individual task which now falls outside of gps's system.