668796 - make::rules.mk - replace double colon rules with dependency chains

Status: NEW

(Firefox Build System :: General, defect)

Description

[Joey Armstrong [:joey]]

Double colon rule use in rules.mak will contribute to a huge number of shells spawned.  Per directory a shell will be invoked at least 3/4 times to process export, libs, dep and tools targets.

If :: rules are replaced with macros/dependency chains that would be processed as deps within the same make session most of the overhead should go away.

## Added shells
build:
   $(MAKE) export
   $(MAKE) deps
   $(MAKE) libs
   $(MAKE) tools

## Dependency chains
todo=$(NULL)
ifneeded export
  todo += export
endif
ifneeded deps
  todo += deps
endif

build: $(todo)


Also I'm still working through the traversal logic but TIER deps/expansion will likely add an entire new dimension to the number of shells spawned.  If tier/module dependency check/build is needed prior to building a given module there are likely a lot more shells on the way.

Do not have a good answer to address this problem at the moment but one approach that could be taken { in addition to the use of tiers to impose a build order within peer elements } is to create a directory hierarchy that would implicitly impose basic tier ordering between levels 1..[n].

Make is then able to traverse tiers serially and build major dependencies w/o explicitly having to code that dependency knowledge into make.  Dependencies that exist between peer elements within a tier could be handled by the addition of normal make deps or added subdirectories.

For a completely contrived example assuming tier n+1 contains no dependencies on tier n:

tier1 - core and standalone elements
tier2 - firefox / thunderbird
tier2 - modules & add-ons
tier3 - testing

To perform testing in stages along the way fractional tiers { coded into makefiles as a dependency/target } could be introduced:
  tier1.1 - unit & standalone tests

Comment 1

[Benjamin Smedberg]

At least in the first part of your example, the todo: target is still flawed in the face of parallel make: you'd have to add a rule of the form

libs:: export

to ensure that the export phase completed before the libs phase. But even that is not sufficient, because make could decide to start doing compilation tasks needed for the libs phase before export was finished (which won't work), so you'd need to add an export:: dependency on *every* compilation target.

Comment 2

[Joey Armstrong [:joey]]

(In reply to comment #1)
> At least in the first part of your example, the todo: target is still flawed
> in the face of parallel make: you'd have to add a rule of the form
> 
> libs:: export
> 
> to ensure that the export phase completed before the libs phase. But even
> that is not sufficient, because make could decide to start doing compilation
> tasks needed for the libs phase before export was finished (which won't
> work), so you'd need to add an export:: dependency on *every* compilation
> target.

The todo section was only simple pseudo code intended to identify the problem and a possible direction.  Yes there will be a boatload of details to contend with in any implementation.  One basic assumption should be that proper make dependencies are in play or all bets are off.

>> dependency on *every* compilation target.

That answer would be a bit excessive and unmaintainable, esp if circular dependencies are in play.  There are ways to implement synchronization logic like this w/o having to introduce a huge number of new dependencies.