910453 - Annotate processed moz.build files with tier info

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

I will explain this better in the description for the soon-to-be-attached patch.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Annotate read moz.build files with tier info

This patch annotates the processed moz.build sandboxes with tier
metadata. The goal of this patch is to eventually enable the make-based
backend to easily generate a make file that incurs submakes for every
directory. i.e. instead of performing recursive traversal, we'll just
list each directory as a flat list and have make evaluate each one using
whatever concurrency it is capable of. This will likely be done
differently for export, libs, tools, etc. But that's followup bug
territory.

Since we may add additional metadata later, I decided to make the
metadata a dict rather than hardcode a "tier" argument/annotation
everywhere.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Part 2: Write tier directory info into make backend

While I was here, I figured we should write out the full list of tier
directories in the root backend.mk file. Next step is to modify
/Makefile.in and the tier traversal rules to leverage this data to build
non-recursively.

Comment 3

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #1)
> This patch annotates the processed moz.build sandboxes with tier
> metadata. The goal of this patch is to eventually enable the make-based
> backend to easily generate a make file that incurs submakes for every
> directory. i.e. instead of performing recursive traversal, we'll just
> list each directory as a flat list and have make evaluate each one using
> whatever concurrency it is capable of.

I'm close to a patch doing that already, but i didn't need that tier info.

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 796895 [details] [diff] [review]
Annotate read moz.build files with tier info

Review of attachment 796895 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/frontend/reader.py
@@ +642,5 @@
>          # It's very tempting to use a set here. Unfortunately, the recursive
>          # make backend needs order preserved. Once we autogenerate all backend
>          # files, we should be able to convert this to a set.
> +        recurse_info = []
> +        dirs = set()

Just use an OrderedDict for recurse_info, and don't keep a separate dirs list, which will be recurse_info.keys().

Comment 5

[Mike Hommey [:glandium]]

Attachment: Annotate read moz.build files with tier info

Since i was going to add stuff on top of this, i needed it updated, so this is your patch with my review comment applied. Just getting it through you in case you have something to add.

Comment 6

[Mike Hommey [:glandium]]

Comment on attachment 796906 [details] [diff] [review]
Part 2: Write tier directory info into make backend

Review of attachment 796906 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not convinced this tier summary is any useful outside the recursive make backend. That is, it doesn't seem to me something the emitter should be emitting.

Comment 7

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fdaa11265464

Comment 8

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/fdaa11265464