132848 - MOZ_MAKE_FLAGS=-j2 doesn't work with gnumake on windows

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, defect, P3)

Description

[TVL]

MOZ_MAKE_FLAGS=-j2 doesn't work with gnumake on windows.  seems there's an 
order problem, build dies early on as follows:

[...]
cl -Fothreads/combined/./pruthr.obj -c     -W3 -nologo -GF -Gy -MD -O2   -
UDEBUG -U_DEBUG  -DMOZILLA_CLIENT=1 -DNDEBUG=1 -DXP_PC=1 -DWIN32=1 -DWIN95=1 -
D_PR_GLOBAL_THREADS_ONLY=1 -D_X86_=1  -DFORCE_PR_LOG -D_NSPR_BUILD_ -
Ic:/stockmoz/mozilla/dist/include/nspr -I../../pr/include -
I../../pr/include/private  threads/combined/./pruthr.c
pruthr.c
make[7]: Leaving directory 
`/cygdrive/c/stockmoz/mozilla/nsprpub/pr/src/threads/combined'
cl -Foprdump.obj -c     -W3 -nologo -GF -Gy -MD -O2   -UDEBUG -U_DEBUG  -
DMOZILLA_CLIENT=1 -DNDEBUG=1 -DXP_PC=1 -DWIN32=1 -DWIN95=1 -
D_PR_GLOBAL_THREADS_ONLY=1 -D_X86_=1  -DFORCE_PR_LOG -D_NSPR_BUILD_ -
Ic:/stockmoz/mozilla/dist/include/nspr -I../../../pr/include -
I../../../pr/include/private  prdump.c
prdump.c
threads/combined/./pruthr.c(1319) : warning C4700: local variable 'status' used 
without having been initialized
make[5]: *** No rule to make target `nspr4.lib'.  Stop.
make[5]: *** Waiting for unfinished jobs....
rc.exe -DMOZILLA_CLIENT=1 -DNDEBUG=1 -DXP_PC=1 -DWIN32=1 -DWIN95=1 -
D_PR_GLOBAL_THREADS_ONLY=1 -D_X86_=1 -DFORCE_PR_LOG -D_NSPR_BUILD_ -
Ic:/stockmoz/mozilla/dist/include/nspr -I../../pr/include -
I../../pr/include/private -Fonspr.res nspr.rc

make[5]: *** Waiting for unfinished jobs....
make[5]: *** Waiting for unfinished jobs....
./nspr.res finished
make[6]: Leaving directory `/cygdrive/c/stockmoz/mozilla/nsprpub/pr/src/threads'

make[5]: Leaving directory `/cygdrive/c/stockmoz/mozilla/nsprpub/pr/src'
make[4]: *** [export] Error 2
make[4]: Leaving directory `/cygdrive/c/stockmoz/mozilla/nsprpub/pr'
make[3]: *** [export] Error 2
make[3]: Leaving directory `/cygdrive/c/stockmoz/mozilla/nsprpub'
make[2]: *** [nspr] Error 2
make[2]: Leaving directory `/cygdrive/c/stockmoz/mozilla'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/cygdrive/c/stockmoz/mozilla'
make: *** [build] Error 2

Comment 1

[hacker formerly known as [e-mail]]

This looks like bug 123423.  A workaround was checked into the trunk a week or
so ago.  Can you try it on your branch build to see if it clears up the problem?

Comment 2

[TVL]

123424 fixes it for nspr, but it fails in ldap now (nsldab32v40.lib).

do you want me to move this comment over to bug 123424 and then you can close
this out as a dup?

Comment 3

[hacker formerly known as [e-mail]]

File a separate bug on the ldap product.  They use a different build system that
piggy backs ontop of NSPR's build system and that's what's causing the problem
here.  Strangely enough, we don't see that particular problem when using -j4 on
linux.  You may be able to get away with a hack similar to the one done for NSPR.

Comment 4

[TVL]

bug 133521 filed for ldap.  close this out as a dup of bug 123424?

Comment 5

[hacker formerly known as [e-mail]]

Does the rest of the build work besides nspr & ldap?  If so, close it out.

Comment 6

[TVL]

adding:
  MAKE := $(patsubst -j%,,$(MAKE)) -j1
to directory/c-sdk/ldap/Makefile.in get's ldap building, but it fails next in 
js.  so i get the feeling there's something that just doesn't work with the 
baseline rules for win32 gmake and -j2.  (in js it's failing in the same way, 
it wants js3250.lib and it hasn't been built yet.)

Comment 7

[hacker formerly known as [e-mail]]

So I looked at this a bit closer and it appears that the problem is triggered by
IMPORT_LIBRARY.  IMPORT_LIBRARY is the .lib stub library that have you have to
actually link against on win32 & OS/2 when you want to link against a dll.   On
win32, this library is automatically generated when the dll is created.  On
OS/2, it appears as though IMPORT_LIBRARY is built in a separate step so we have
an OS/2 only make target for IMPORT_LIBRARY. mkaply, can you confirm that the
extra ruleset is still needed?  

The problem in -jx builds is that both SHARED_LIBRARY & IMPORT_LIBRARY are
listed as dependencies of libs::.  On non -j1 builds, this isn't a problem as
SHARED_LIBRARY is listed first so IMPORT_LIBRARY has been built by time the
dependency is checked.  What I'm seeing in -jx builds is that the dependencies
of libs:: are being evaluated in parallel so the build attempts to build
IMPORT_LIBRARY before SHARED_LIBRARY has finished.  If I remove IMPORT_LIBRARY
from the dependency list, js builds fine with -j2.  An alternative fix is to add
a win32 dummy ruleset for IMPORT_LIBRARY that consists of just a dependency upon
SHARED_LIBRARY.

Comment 8

[Mike Kaply [:mkaply]]

We definitely need a separate target for the IMPORT_LIB because it 
happens in a separate step.

Comment 9

[hacker formerly known as [e-mail]]

So after adding the stub IMPORT_LIBRARY rule for win32 and fixing some
dependency problems when calling midl, I ran into another problem.  Anyone know
exactly what these .pdb files are used for?  When building jpeg, I ran into the
problem that both files being compiled were attempting to write to jpeg3250.pdb.
 I added a hack to use .NOTPARALLEL: if PDBFILE is defined and that seems to get
around the problem for now.

Comment 10

[timeless]

pdb is described in 
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q121366

Comment 11

[hacker formerly known as [e-mail]]

Attachment: win32 make -j v1

So I'm not sure how useful this make -j support really is.  We appear to use
PDBFILE by default in a lot of places.	It's definitely turned on in
debug/profile builds.

Comment 12

[Brian Ryner (not reading)]

Comment on attachment 78309 [details] [diff] [review]
win32 make -j v1

so, this essentially makes -j have no effect, but that's better than failing to
build.	r=bryner.

Comment 13

[hacker formerly known as [e-mail]]

Hmm. So I looked at this again and we use -Fd$(PDBFILE) by default everywhere. 
However, xpcom built fine with -j and jpeg didn't.  It looks like even though we
specify -Fd, it isn't used (atm) unless you build with debugging info.  Jpeg
adds -Zi, supposedly for mmx support, which adds debug info to the pdb. If I
remove the -Zi then jpeg compiles with -j just fine.  Not sure if we should
change the patch or not.

Comment 14

[hacker formerly known as [e-mail]]

Attachment: win32 make -j v2

Only off -j if MOZ_DEBUG or MOZ_PROFILE are set.  Turn it off always for jpeg &
xpconnect since they use certain compiler flags that actually touch the .pdb
files.

Comment 15

[Brian Ryner (not reading)]

Comment on attachment 79928 [details] [diff] [review]
win32 make -j v2

r=bryner

Comment 16

[hacker formerly known as [e-mail]]

The patch has been checked in on the trunk.

Comment 17

[Paul Wyskoczka]

Why do we need to implement this on the branch at this time?  Do we plan to ship
beta using gmake on Windows?

Comment 18

[hacker formerly known as [e-mail]]

This is just one of a growing list of build fixes (see bug 138348) that aren't
critical for releasing the beta to end-users but are wanted for our "long-lived"
1.0 branch that we're recommending to embeddors.  And we are only recommending
the gmake builds to embeddors.  So the fix doesn't have to go in *right now* but
it does need to go into the 1.0 branch at some point.  Should I be using
adt1.0.1 for these non-end-user affecting but branch-bound fixes instead?

Comment 19

[Jaime Rodriguez, Jr.]

This is not needed for MachV beta, based on Seawood's comments adt1.0.0-. Pls
get hit in after MachV branches for beta.

Comment 20

[hacker formerly known as [e-mail]]

Giving up on drivers.