31365 - parallel build failes in nsprpub

Status: VERIFIED FIXED

(NSPR :: NSPR, defect, P2)

Description

[Axel Hecht]

trying a parallel build on solaris 2.7, pull from Mar 10 20:12 MET, egcs-2.91.66
make -f client.mk build
-j12 on 12cpu machine with sources in /tmp

first few errors:

/tmp/mozilla/nsprpub/pr/src/io/./prfdcach.c:19: primpl.h: No such file or
directory
/tmp/mozilla/nsprpub/pr/src/io/./prmwait.c:19: primpl.h: No such file or
directory
/tmp/mozilla/nsprpub/pr/src/io/./prmwait.c:20: pprmwait.h: No such file or
directory

doing it again I gets it passed thru nsprpub, but then hangs on bug 31364

Comment 1

[Gervase Markham [:gerv]]

[e-mail] - have you had better luck recently?

Gerv

Comment 2

[Axel Hecht]

Hi Gerv,
sorry for the delay, been on vacation. Just tried, and no, nothing changed.

Axel

Comment 3

[cls]

As I pointed out in bug 31364, GNU make versions greater than 3.77 have known
problems building in parallel with mozilla.  See if the problem still persists
if you downgraded to 3.76.1.

Comment 4

[cls]

Is anyone seeing this problem using gnu make <= 3.77?

Comment 5

[Gervase Markham [:gerv]]

Marking INVALID; reporter was using unsupported version of make.

Gerv

Comment 6

[cls]

Reopening as we're going to make an effort to support parallel builds with the
newer makes.

I think I discovered what the problem here is. In nsprpub/pr/src/Makefile{,.in}
,  we create libnspr.so from the object files in various subdirs.  The problem
is that we list the object files in OBJS rather than some separate dependency
variable.  This means that we can build those object files from the current
directory instead of recursing down to the subdir.  It also means that we need
to have the exact same set of includes & defines that we do in the subdirs. 
This is not happening.  

gcc -o priometh.o -c   -pipe -ansi -pthread -g -fPIC  -UNDEBUG  -DUSE_AUTOCONF=1
-DDEBUG=1 -DDEBUG_cls=1 -DXP_UNIX=1 -D_POSIX_SOURCE=1 -D_BSD_SOURCE=1
-D_SVID_SOURCE=1 -DLINUX=1 -DHAVE_LCHOWN=1 -DHAVE_STRERROR=1 -D_REENTRANT=1 
-DFORCE_PR_LOG -D_PR_PTHREADS -UHAVE_CVAR_BUILT_ON_SEM -D_NSPR_BUILD_
-I/usr/cls/src/obj/dist/include -I../../../../../mozilla/nsprpub/pr/include
-I../../../../../mozilla/nsprpub/pr/include/private 
../../../../../mozilla/nsprpub/pr/src/io/priometh.c
gcc -o io/./prfdcach.o -c   -pipe -ansi -pthread -g -fPIC  -UNDEBUG 
-DUSE_AUTOCONF=1 -DDEBUG=1 -DDEBUG_cls=1 -DXP_UNIX=1 -D_POSIX_SOURCE=1
-D_BSD_SOURCE=1 -D_SVID_SOURCE=1 -DLINUX=1 -DHAVE_LCHOWN=1 -DHAVE_STRERROR=1
-D_REENTRANT=1  -DFORCE_PR_LOG -D_PR_PTHREADS -UHAVE_CVAR_BUILT_ON_SEM
-D_NSPR_BUILD_ -I/usr/cls/src/obj/dist/include  
../../../../mozilla/nsprpub/pr/src/io/./prfdcach.c
../../../../mozilla/nsprpub/pr/src/io/./prfdcach.c:19: primpl.h: No such file or
directory

Both priometh.o & prfdcach.o should be built in the pr/src/io dir. 
Note the extra includes that priometh.o has.

There are about 3 ways to go about fixing this. 

1) Add the extra includes to INCLUDES in pr/src.  It's quick-n-dirty but works
until another dependency or variable is added to a subdir's Makefile but not
pr/src's .

2) Build sublibraries or archives in the subdirs and link the sublibraries like
we do in Mozilla.  Very ugly, IMO, and just mentioned for completeness.

3) Create a objs.mk in the subdirs which contains the rules (relative to the
nspr tree) for building those object files from any subdirectory.  This way you
can say libnspr.so depends upon $(DEP_OBJS) and it will rebuild them if they do
not exist.

Known adv:
2 & 3 put the logic for building specific object file in a single place.

Known cons:
1 requires that any variables that affect build rules be synchronized across
multiple makefiles.
2 & 3 may require invoking additional make processes to handle the dependencies.

Comment 7

[cls]

Attachment: Quick fix for nspr parallel build problem

Comment 8

[cls]

The quick fix has been checked into the tip & the NSPRPUB_CLIENT_BRANCH:

Checking in pr/src/Makefile;
/cvsroot/mozilla/nsprpub/pr/src/Makefile,v  <--  Makefile
new revision: 3.36; previous revision: 3.35
done
Checking in pr/src/Makefile.in;
/cvsroot/mozilla/nsprpub/pr/src/Makefile.in,v  <--  Makefile.in
new revision: 1.15; previous revision: 1.14
done


Checking in nsprpub/pr/src/Makefile;
/cvsroot/mozilla/nsprpub/pr/src/Makefile,v  <--  Makefile
new revision: 3.30.2.4; previous revision: 3.30.2.3
done
Checking in nsprpub/pr/src/Makefile.in;
/cvsroot/mozilla/nsprpub/pr/src/Makefile.in,v  <--  Makefile.in
new revision: 1.9.2.4; previous revision: 1.9.2.3
done

Comment 9

[Christopher Blizzard (:blizzard)]

Mine still fails.  I'll attach a log in a sec.

Comment 10

[Christopher Blizzard (:blizzard)]

Attachment: output

Comment 11

[Axel Hecht]

Blizzard,
that is http://bugzilla.mozilla.org/show_bug.cgi?id=31364, already filed.

Marking this one as verified.

Axel

Comment 12

[cls]

Actually, if you look at the build log, he's hitting a problem with multiple gcc
processes writing to the same object file and corrupting it. I'm working on an
extended solution so that object files aren't built in pr/src .

Comment 13

[cls]

Attachment: Shortened version of extended solution

Comment 14

[Axel Hecht]

I tested this on a 12cpu sparc, on clobbered builds and from scratch.
Works for my configuration, the patch looks ok, so r=[e-mail]

Axel

Comment 15

[Axel Hecht]

Something is strange, I have that patch in my tree, and now it won't build
libnspr4.so. The .done is there, but it won't build the lib. It installs the
links though.
Don't know why this went smooth the first time, and I don't have the time to 
debug this right now.
So I issue a <ouch> here.

Axel

Comment 16

[Axel Hecht]

There might be an easy remedy:

in nsprpub/config/rules.mk
change
$(DONE): $(OBJS)
to
$(DONE): $(TARGETS)

This built the lib, and didn't reintroduce the races, as far as I tested.

Hope this helps

Axel

Comment 17

[cls]

I just checked in a change to the nspr makefiles that removes the unneeded
recursive makes for the libs & install phrases.  With those gone, I don't see to
have a problem doing a 'make -j 25' from nsprpub/ .  This is when using gmake
3.79.1 on a dual processor box and without any of the patches from this bug.

Comment 18

[Decklin Foster]

I'm still seeing this w/ a current tree:
this is with 'make -j3' in nsprpub.

cd src; /usr/local/bin/gmake export
gmake[2]: Entering directory `/home/decklin/seamonkey/mozilla/nsprpub/pr/src'
cd io; /usr/local/bin/gmake export
gmake[2]: *** No rule to make target `io/Linux2.4.0_x86_PTH_OPT.OBJ/prfdcach.o',
needed by
 `Linux2.4.0_x86_PTH_OPT.OBJ/libnspr4.a'.  Stop.
gmake[2]: *** Waiting for unfinished jobs....

Comment 19

[cls]

The classic nspr build is well on its way to being obsolete.  Use
--enable-nspr-autoconf .

Comment 20

[Decklin Foster]

It looks that everything is fixed after turning that on, i'll be sure once the
current bustage is cleared. thanks.

Comment 21

[Decklin Foster]

OK, whatever was causing me problems the other day seems to be fixed; i tried a
few more clobbers last night and this morning w/ various -j settings and
everything went smoothly. Marking fixed. thanks!

Comment 22

[Jon Granrose]

verified.