The Tup backend should honor MOZ_PARALLEL_BUILD

RESOLVED FIXED in Firefox 63

Status

enhancement
RESOLVED FIXED
Last year
Last year

People

(Reporter: jwatt, Assigned: mshal)

Tracking

Trunk
mozilla63

Firefox Tracking Flags

(firefox63 fixed)

Details

Attachments

(1 attachment)

When building using make I can set the number of parallel jobs. For Tup it seems like this is going to be especially important in order to avoid the build using every single hyperthread and making the machine virtually unusable for any other work while a build is going on. It's possible to pass -j to `mach build`, but annoying to have to remember to do that each time. Can the Tup backend be made to honor MOZ_PARALLEL_BUILD like the make backend, please? :)
This seems like a sort of accidental feature, tbh. mach passes the contents of the `-j` argument down to client.mk using `MOZ_PARALLEL_BUILD` and then client.mk uses that to invoke make in the objdir:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/python/mozbuild/mozbuild/base.py#621
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/client.mk#44

This is presumably a hack around client.mk not being parallel safe, although I'm not sure if that's strictly true anymore.

In any event, the tup backend invokes tup here, where it honors any `-j` option you pass to mach, or defaults to the number of cores on your machine:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/python/mozbuild/mozbuild/backend/tup.py#300

Making that honor `MOZ_PARALLEL_BUILD` ought to be simple.
Ah. I only just discovered MOZ_PARALLEL_BUILD in the source, and assumed it was something that I wasn't aware of that people were using instead of MOZ_MAKE_FLAGS nowadays. If that isn't the case, can we create a new define with a different name? The name MOZ_PARALLEL_BUILD doesn't make it clear that this thing is supposed to take a number, which is the job count. It sounds more like a boolean that should be 1 or 0. Maybe MOZ_JOB_COUNT would be better?
Keywords: in-triage
I can take a look at this. I think we already have the mozconfig make info available in the Tup backend, so it shouldn't be too hard to parse out.
Assignee: nobody → mshal
Keywords: in-triage
If you specify 'mk_add_options MOZ_PARALLEL_BUILD=X' in your mozconfig,
this variable ends up being passed into client.mk, which tells the make
backend to use that as the number of parallel jobs. However, there is no
equivalent in alternate backends aside from specifying '-jX' at the
commandline on each build invocation.

Rather than making a new mechanism to do this, we can check the
mozconfig for the MOZ_PARALLEL_BUILD flag and use that number for the
jobs parameter if no override was specified on the commandline.

MozReview-Commit-ID: 4YHG30N6tmi
Comment on attachment 9000065 [details]
Bug 1481441 - Honor MOZ_PARALLEL_BUILD in non-make backends; r?Build

Chris Manchester (:chmanchester) has approved the revision.
Attachment #9000065 - Flags: review+
Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/66c2aec021c7
Honor MOZ_PARALLEL_BUILD in non-make backends; r=chmanchester
https://hg.mozilla.org/mozilla-central/rev/66c2aec021c7
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
You need to log in before you can comment on or make changes to this bug.