mach should respect MOZ_MAKE_FLAGS

RESOLVED FIXED in mozilla35

Status

Firefox Build System
General
RESOLVED FIXED
5 years ago
3 months ago

People

(Reporter: mak, Assigned: glandium)

Tracking

Trunk
mozilla35
x86_64
Windows 7

Firefox Tracking Flags

(firefox33 affected, firefox34 affected)

Details

(Whiteboard: [mach])

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

5 years ago
I have 8 cores (4 physical, 4 logical), on my Windows box -j12 is quite faster than -j8, so I have MOZ_MAKE_FLAGS="-j12" set, though mach ignores it and builds with -j8

Comment 1

5 years ago
This is with |mach build <target>| since that uses a slightly different dispatch path from |mach build|.

Comment 2

5 years ago
Either `mach` ignores the jobs flag or it limits the number to 8 jobs. I confirm that running `./mach build -j100` seems to only ever use 8 threads.

Comment 3

5 years ago
How are you setting MOZ_MAKE_FLAGS? You need to be using |mk_add_options MOZ_MAKE_FLAGS| in your mozconfig.

Are you sure that mozconfig is the one loaded by mach?

Updated

5 years ago
Component: mach → Build Config
Whiteboard: [mach]

Comment 4

5 years ago
I am not using mozconfig, but `./mach help build` says:

    --jobs jobs, -j jobs  Number of concurrent jobs to run. Default is the number of CPUs.
I see this too, and it's frustrating, because it slows my machine to a crawl when compiling.

mk_add_options MOZ_MAKE_FLAGS="-s"

is in my $(SRCDIR)/.mozconfig.

How do I verify that that's the .mozconfig in use?
(In reply to Dan Mosedale (:dmose) - not reading bugmail; needinfo? for response from comment #5)
> I see this too, and it's frustrating, because it slows my machine to a crawl
> when compiling.
> 
> mk_add_options MOZ_MAKE_FLAGS="-s"
> 
> is in my $(SRCDIR)/.mozconfig.
> 
> How do I verify that that's the .mozconfig in use?

cat obj-dir/.mozconfig should show what .mozconfig was used.
gps: i just verified that this problem does indeed occur even using mk_add_options as described above, and that the mozconfig that has that is definitely in use in the build that displays the behavior.
(Assignee)

Comment 8

4 years ago
(In reply to Dan Mosedale (:dmose) - not reading bugmail; needinfo? for response from comment #5)
> I see this too, and it's frustrating, because it slows my machine to a crawl
> when compiling.
> 
> mk_add_options MOZ_MAKE_FLAGS="-s"
> 
> is in my $(SRCDIR)/.mozconfig.
> 
> How do I verify that that's the .mozconfig in use?

mach defaults to -s, so what is your problem exactly?
glandium: the fact that i'm not specifying -j at all, and it's compiling with -j8, which eats all my cores.
(Assignee)

Comment 10

4 years ago
(In reply to Dan Mosedale (:dmose) - not reading bugmail; needinfo? for response from comment #9)
> glandium: the fact that i'm not specifying -j at all, and it's compiling
> with -j8, which eats all my cores.

The default for mach is to use all cores. If you want only one core to be used, you need to explicitely add -j1. Either in MOZ_MAKE_FLAGS or on mach command line.
(Assignee)

Comment 11

4 years ago
BTW, reading when this bug was filed, I'm sure the code handling MOZ_MAKE_FLAGS has been changed after this bug was filed, so it should be checked whether this still applies or not. It's also not clear what this bug is about (mach build or mach build foo).
Flags: needinfo?(mak77)
glandium: sorry, I should have been more communicative.  I tried a whole bunch of things, and that was where I ended up.

I just did a clobber, switched back to using

mk_add_options MOZ_MAKE_FLAGS="-j1" 

and verified that that change propagated through to my objdir.

Despite this, -j8 is not being overridden (I can see it ps), and all 8 cores are being used.
And in my cases, this is just executing "mach build".
(Assignee)

Comment 14

4 years ago
(In reply to Dan Mosedale (:dmose) - not reading bugmail; needinfo? for response from comment #12)
> glandium: sorry, I should have been more communicative.  I tried a whole
> bunch of things, and that was where I ended up.
> 
> I just did a clobber, switched back to using
> 
> mk_add_options MOZ_MAKE_FLAGS="-j1" 
> 
> and verified that that change propagated through to my objdir.
> 
> Despite this, -j8 is not being overridden (I can see it ps), and all 8 cores
> are being used.

So, I can confirm this, and this is fallout from bug 762358, which landed a couple months ago, so is unrelated to the original bug. So, this bug currently exists, but has nothing to do with why it was originally filed.
(Assignee)

Updated

4 years ago
Blocks: 762358
(Assignee)

Comment 15

4 years ago
Created attachment 8493544 [details] [diff] [review]
Fix MOZ_MAKE_FLAGS handling for client.mk

After bug 762358 mk_add_options MOZ_MAKE_FLAGS was simply ignored in client.mk
processing. At the same time, mach environment was expecting a list of options
while the mozconfig reader returned a single string, so straighten this up at
the same time.
Attachment #8493544 - Flags: review?(mshal)
(Assignee)

Updated

4 years ago
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
(Assignee)

Updated

4 years ago
status-firefox33: --- → affected
status-firefox34: --- → affected
(Reporter)

Comment 16

4 years ago
For sure ./mach build <target> doesn't respect my -j12... that said, if it is respected for ./mach build I'm happy enough.
Flags: needinfo?(mak77)
Thanks much for the quick patch, Mike!
Comment on attachment 8493544 [details] [diff] [review]
Fix MOZ_MAKE_FLAGS handling for client.mk

This looks fine, though it doesn't seem to apply to './mach build foo' - was your intention to fix that as well? Just a './mach build' works now though.
Attachment #8493544 - Flags: review?(mshal) → review+
(Assignee)

Comment 19

4 years ago
mach build foo is a different beast. It needs a separate fix.
(Assignee)

Updated

4 years ago
Keywords: leave-open
(Assignee)

Comment 20

4 years ago
Created attachment 8494180 [details] [diff] [review]
Respect MOZ_MAKE_FLAGS when running make commands from mach

This addresses the issue for make commands that don't go through client.mk.
Attachment #8494180 - Flags: review?(mshal)
Comment on attachment 8494180 [details] [diff] [review]
Respect MOZ_MAKE_FLAGS when running make commands from mach

Seems fine, though note it doesn't work if you just use MOZ_MAKE_FLAGS=-j (ie: unlimited jobs). I don't know if we really care to support that, though.
Attachment #8494180 - Flags: review?(mshal) → review+
(Assignee)

Comment 23

4 years ago
(In reply to Michael Shal [:mshal] from comment #22)
> Comment on attachment 8494180 [details] [diff] [review]
> Respect MOZ_MAKE_FLAGS when running make commands from mach
> 
> Seems fine, though note it doesn't work if you just use MOZ_MAKE_FLAGS=-j
> (ie: unlimited jobs). I don't know if we really care to support that, though.

I think it's actually a good feature that -j is ignored. However, the following option shouldn't be eaten. I'll fix this.
(Assignee)

Comment 24

4 years ago
Created attachment 8495552 [details] [diff] [review]
Respect MOZ_MAKE_FLAGS when running make commands from mach
Attachment #8495552 - Flags: review?(mshal)
(Assignee)

Updated

4 years ago
Attachment #8494180 - Attachment is obsolete: true

Updated

4 years ago
Attachment #8495552 - Flags: review?(mshal) → review+
(Assignee)

Updated

4 years ago
Keywords: leave-open
https://hg.mozilla.org/mozilla-central/rev/72e22f3d7cc6
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
I suppose this change is what broke my local build.

mk_add_options MOZ_MAKE_FLAGS="-s -j 12"

Leads to:
4:22.03 mozmake.EXE[2]: *** No rule to make target '12'.  Stop.

On a fresh clobber.
Flags: needinfo?(mh+mozilla)
Confirmed that I can build again with rev 72e22f3d7cc6 backed out.
(Assignee)

Comment 29

4 years ago
It's a consequence of client.mk not supporting spaces between -j and 12 when setting MOZ_PARALLEL_BUILD, which predates this bug, but that this bug unhides, because now mach almost always sets MOZ_PARALLEL_BUILD. It also means before this bug, the build would have failed if you ran e.g. mach build -j8. Please file a separate bug.
Flags: needinfo?(mh+mozilla)
Flags: qe-verify-

Updated

3 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.