1241022 - Rework affected tiers for the recursive make backend

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

The model we use for affected tiers, started in bug 921003 works less and less properly. Some of them are now blatantly wrong. Some others are subtly wrong.

Plus, they only really apply to the recursive make backend, while kind of defined globally.

Comment 1

[Mike Hommey [:glandium]]

As an example, we are currently doing `make -C $dir export` in 503 directories (grepping "Entering" in a `mach build export -v` log for a local linux build). 315 or those end with "Nothing to be done for 'export'".

Assuming I didn't mess things up, I got that down to 287 `make -C $dir export` and 97 "Nothing to be done for 'export'" (we still have those because we still assume that we can't skip a directory with a Makefile.in in it).

Comment 2

[Mike Hommey [:glandium]]

Attachment: Handle all tiers as opt-in in the recursive make backend

Historically, all tiers were handled as opt-out (may_skip), until we
added the first opt-in tier (no_skip). It doesn't make much sense to
differentiate them anymore, so handle them all as opt-in.

Comment 3

[Mike Hommey [:glandium]]

Attachment: Do not handle HAS_MISC_RULE with affected_tiers

This initiates a move off affected_tiers in VARIABLES definition to
explicit in-backend handling, which will hopfully make things clearer.

HAS_MISC_RULE is currently used to opt-in to the misc tier in a few
directories with a misc:: rule.  It is, in fact, mostly used for custom
xpi creation, which will be separately addressed in bug 1240676, so it
will eventually go away entirely, but in the meantime, we send it as a
throwaway passthru.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Move ANDROID_GENERATED_RESFILES, ANDROID_APK_NAME, ANDROID_APK_PACKAGE

The are passthru variables that don't actually affect any tier per the
backend itself. They do affect the `export` tier by way of the Makefile.in
along the moz.build defining them, and the existence of those
Makefile.in already guarantees those directories not to be skipped for
`export`.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Move ANDROID_*_DIRS, ANDROID_EXTRA_PACKAGES, ANDROID_ECLIPSE_PROJECT_TARGETS and JAVA_JAR_TARGETS

Those are non-passthru variables with the same property as
ANDROID_GENERATED_RESFILES, ANDROID_APK_NAME and ANDROID_APK_PACKAGE, in
that they don't affect tiers through the backend code itself, but from
the Makefile.in along the moz.build they are defined in.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Move GENERATED_FILES

GENERATED_FILES impacts the export tier through the config/rules.mk
definitions, now moved to the backend itself, so that everything is
close to each other.

Comment 7

[Mike Hommey [:glandium]]

Attachment: Move JAR_MANIFESTS

JAR_MANIFESTS affects the libs tiers through config/rules.mk rules.
While we could move the rules in the backend, they are too complex to
just do that now.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Move FINAL_TARGET_PP_FILES and TEST_HARNESS_FILES

They are respectively using PP_TARGETS and INSTALL_TARGETS. Both affect
the misc tier since bug 1240671, per the *_TARGET value they set in the
backend.

This has the nice side effect of now skipping directories where test
harness files are handled by install manifests.

Comment 9

[Mike Hommey [:glandium]]

Attachment: Move IPDL_SOURCES, XPIDL_SOURCES and *WEBIDL_*

Those are the worst offenders of affected_tiers. The rules to handle
them are all in directories that is not necessarily related to where
the variables are defined, each of which has a Makefile.in for those
rules, which forces export to go through them.

Comment 10

[Mike Hommey [:glandium]]

Attachment: Fix XPI_NAME and FILES_PER_UNIFIED_FILE

PI_NAME affects no tier on its own, as it merely changes paths where
things end up that are defined by other variables.

FILES_PER_UNIFIED_FILE had an erroneous value.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Remove code handling affected_tiers.

Comment 12

[Mike Hommey [:glandium]]

Attachment: Remove affected_tiers

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8709838 [details] [diff] [review]
Handle all tiers as opt-in in the recursive make backend

Review of attachment 8709838 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/backend/recursivemake.py
@@ +784,5 @@
>                          {'path': makefile}, 'Substituting makefile: {path}')
>                      self._makefile_in_count += 1
>  
> +                    if bf.relobjdir:
> +                        for tier in ('export', 'libs',):

This feels like it needs a comment. But I suspect you'll address this in a subsequent patch.

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8709848 [details] [diff] [review]
Remove affected_tiers

Review of attachment 8709848 [details] [diff] [review]:
-----------------------------------------------------------------

Nice series.

Comment 15

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/452612abdc8c
https://hg.mozilla.org/integration/mozilla-inbound/rev/3a96b4ee3c1f
https://hg.mozilla.org/integration/mozilla-inbound/rev/5bb5c3f415f3
https://hg.mozilla.org/integration/mozilla-inbound/rev/b62b1d1fc6d3
https://hg.mozilla.org/integration/mozilla-inbound/rev/e24d733ce715
https://hg.mozilla.org/integration/mozilla-inbound/rev/6bb5987487b5
https://hg.mozilla.org/integration/mozilla-inbound/rev/6153051b0291
https://hg.mozilla.org/integration/mozilla-inbound/rev/2997c593fd63
https://hg.mozilla.org/integration/mozilla-inbound/rev/f10de1a56091
https://hg.mozilla.org/integration/mozilla-inbound/rev/d2692a330fb0
https://hg.mozilla.org/integration/mozilla-inbound/rev/7b9c54abd2fe

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/452612abdc8c
https://hg.mozilla.org/mozilla-central/rev/3a96b4ee3c1f
https://hg.mozilla.org/mozilla-central/rev/5bb5c3f415f3
https://hg.mozilla.org/mozilla-central/rev/b62b1d1fc6d3
https://hg.mozilla.org/mozilla-central/rev/e24d733ce715
https://hg.mozilla.org/mozilla-central/rev/6bb5987487b5
https://hg.mozilla.org/mozilla-central/rev/6153051b0291
https://hg.mozilla.org/mozilla-central/rev/2997c593fd63
https://hg.mozilla.org/mozilla-central/rev/f10de1a56091
https://hg.mozilla.org/mozilla-central/rev/d2692a330fb0
https://hg.mozilla.org/mozilla-central/rev/7b9c54abd2fe