167254 - (RMCH) Recursive Make Considered Harmful

Status: RESOLVED INCOMPLETE

(Firefox Build System :: General, enhancement, P3)

Description

[hacker formerly known as [e-mail]]

This is a problem that we've known about for years and people routinely drop a
link to this paper, http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html
, which suggests a fix.  The fix, naturally, requires a minor overhaul of the
build system and maybe an adjustment in developer expectations.  What else is new?

For a build, we invoke make approximately 740 time for each build pass.  That
number is just the "core" Mozilla directories so it doesn't include NSPR, LDAP
or NSS.  I think we only have a 100 or so distinct modules so the goal would be
to limit us to a single make invocation per module (per pass).

Bug 163207 contains a preliminary patch to build all of transformiix from the
build & public directories.  I resorted to using cygpath to translate $< into a
proper dos-path rather than sticking with the srcfiles-must-be-in-$srcdr
limitation set in bug 141834.  The extra $(shell) invocations will slow us down
but the gain of not invoking multiple make processes should more than offset that.

One of the major decisions that will need to be made is where to stick the
object & dependency files for builds that use source files from multiple
directories.  The easiest thing would be to just place the depend files into the
build directory and the object files in their respective directory hierarchy. 
But that could be fairly confusing.

The previous decision will also be affected by what we decide to do with the
Makefiles that are no longer necessary.  Do we remove them, thereby saving time
& space, or we leave them so that developers can still built sub-sections of
their module without needing to rebuild the entire module?  Is the convenience
worth the effort of maintain multiple methods to compile a source file?

Comment 1

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Given both strong, correct dependencies and a build system that can compute the
required set of rebuild operations quickly, I don't think many people will have
objections to just doing a module-top rebuild every time.

I know I wouldn't.

Comment 2

[hacker formerly known as [e-mail]]

The win32 dependency generation change in attachment 98265 [details] [diff] [review] causes a 8% slowdown
in the overall build time.  The change to remove extraneous makes from xslt in
attachment 98266 [details] [diff] [review] causes a 16% speed up under both win32 & linux when just
building xslt.  Without the win32 dep changes though, the dependencies are
genearated incorrectly.  On win32, the dependency change + the xslt change
causes a 20% slowdown when building xslt.

Comment 3

[hacker formerly known as [e-mail]]

I improved the win32 dep changes so that they only invoke $(shell cygpath) once
(attachment 98549 [details] [diff] [review]) and that brought the win32 dep + xslt build time to only 10%
slowdown. 

As always, these measurments should be taken with a grain of salt.  They are
made on a P3-650 laptop w/ 128mb ram running win2k & a P3-600 w/384mb ram
running RH7.2.  I would expect a signficantly different results from tests done
with more modern hardware.

Comment 4

[hacker formerly known as [e-mail]]

I attempted to convert dbm to use a single Makefile and I ran into some problems
that I briefly mentioned on IRC a couple of days ago.

1) Our primary targets (SHARED_LIBRARY & PROGRAM) assume that their sources come
from CPPSRCS, CSRCS & OBJS.  This can be worked around by introducing
LIB_CPPSRCS,  LIB_OBJS, etc to rules.mk.

2) Some of our modules build multiple test programs.  As long as these programs
are built from a single source file, we can use SIMPLE_PROGRAMS.  Otherwise, we
have to use PROGRAM to build them and we can only support one PROGRAM per
Makefile.  

3) For most of our libraries that we directly link against, we use internal
defines to indicate that we are building the libraries.  When building a program
that uses the library, we do not use those defines.  So, if we're going to build
 both the library & a program that uses that library from a single Makefile, we
need to use a separate definition list for each target.  Right now, we have no
way of adding defines to individual objects (short of adding a rule per object).

4) We have a similar problem with LIBS.  Any PROGRAM or SIMPLE_PROGRAMS would be
linked against all libraries listed in LIBS.  This may be a non-issue. 

5) EXTRA_LIBS have the same problem as above and they are also used when linking
the SHARED_LIBRARY.


So while looking at potential solutions for the above problems, I noted that
solutions look suspiciously familiar.  They started looking like the automake
syntax.  So here's my latest lament: I still don't think that automake is a
viable solution for mozilla* but I don't really want to reinvent the wheel.

* Reasons why not to use automake:
1) Additional build requirements and a signficantly more complex build
infrastructure for developers who don't regularly muck with Makefiles.
2) Perceived lack of stability.  Afaict, there have been at least 3 major
releases in the past 4 years which have also required upgrading other autotools
packages (autoconf, libtool, etc).  Because not all packages that use autotools
can be upgraded in lock-step (NSPR wants to stay with autoconf-2.1x), this
causes a major problem.
3) Perceived lack of backwards compatiblity.  The more recent releases have
required autoconf 2.5x which we do not support (see bug 104642).  It also
appears that with the changes made to some releases, you'll get a different
incompatible result from some macros than you did with previous releases.

(Note the use of perceived as I haven't actually experienced these problems. 
I've just read others complaints on mailing lists.)

Comment 5

[hacker formerly known as [e-mail]]

I created the RMCH_20021105_BRANCH branch so that I can start getting some of
these changes out of my local tree.

Comment 6

[hacker formerly known as [e-mail]]

I still need to pull some of those changes in from the branch but this is going
to take awhile.

Comment 7

[Benjamin Smedberg]

Attachment: Use manifest.mk, checkpoint

This works well for reducing the total number of spawned processes per build loop. The problem is that it requires make 3.80 for the $(eval) function. This would be fine with me except that MSYS doesn't ship a msys-make3.80 (only a mingw-make, which isn't the same thing at all).

Comment 8

[Benjamin Smedberg]

It would theoretically be possible to fix the "build multiple libraries or programs from the same makefile" using $(eval) and constructs like PROGRAM_foo_CPPSRCRS, but that sounds like overkill at least for this pass. If we can get rid of most of the intermediate static libraries that is more than enough for me.

Comment 9

[Benjamin Smedberg]

Attachment: Use manifest.mk for netwerk, rev. 1

I'm going to do some performance benchmarking with this patch, but I believe that if it performs as I think it will and we can solve the make 3.80 issue, this would be a great improvement.

Comment 10

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

See Bug 623617 where we plan to implement something like comment 8.

Comment 11

[Nicholas Nethercote [inactive]]

The build team is working on a total overhaul of the build system that will encompass this. This bug doesn't need to be open any more.