parallel build failes in nsprpub

VERIFIED FIXED

Status

defect
P2
normal
VERIFIED FIXED
20 years ago
19 years ago

People

(Reporter: axel, Assigned: cls)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

trying a parallel build on solaris 2.7, pull from Mar 10 20:12 MET, egcs-2.91.66
make -f client.mk build
-j12 on 12cpu machine with sources in /tmp

first few errors:

/tmp/mozilla/nsprpub/pr/src/io/./prfdcach.c:19: primpl.h: No such file or
directory
/tmp/mozilla/nsprpub/pr/src/io/./prmwait.c:19: primpl.h: No such file or
directory
/tmp/mozilla/nsprpub/pr/src/io/./prmwait.c:20: pprmwait.h: No such file or
directory

doing it again I gets it passed thru nsprpub, but then hangs on bug 31364
axel@pike.org - have you had better luck recently?

Gerv
Hi Gerv,
sorry for the delay, been on vacation. Just tried, and no, nothing changed.

Axel
As I pointed out in bug 31364, GNU make versions greater than 3.77 have known
problems building in parallel with mozilla.  See if the problem still persists
if you downgraded to 3.76.1.
Is anyone seeing this problem using gnu make <= 3.77?

Target Milestone: --- → M18
Marking INVALID; reporter was using unsupported version of make.

Gerv
Status: UNCONFIRMED → RESOLVED
Closed: 20 years ago
Resolution: --- → INVALID
Reopening as we're going to make an effort to support parallel builds with the
newer makes.

I think I discovered what the problem here is. In nsprpub/pr/src/Makefile{,.in}
,  we create libnspr.so from the object files in various subdirs.  The problem
is that we list the object files in OBJS rather than some separate dependency
variable.  This means that we can build those object files from the current
directory instead of recursing down to the subdir.  It also means that we need
to have the exact same set of includes & defines that we do in the subdirs. 
This is not happening.  

gcc -o priometh.o -c   -pipe -ansi -pthread -g -fPIC  -UNDEBUG  -DUSE_AUTOCONF=1
-DDEBUG=1 -DDEBUG_cls=1 -DXP_UNIX=1 -D_POSIX_SOURCE=1 -D_BSD_SOURCE=1
-D_SVID_SOURCE=1 -DLINUX=1 -DHAVE_LCHOWN=1 -DHAVE_STRERROR=1 -D_REENTRANT=1 
-DFORCE_PR_LOG -D_PR_PTHREADS -UHAVE_CVAR_BUILT_ON_SEM -D_NSPR_BUILD_
-I/usr/cls/src/obj/dist/include -I../../../../../mozilla/nsprpub/pr/include
-I../../../../../mozilla/nsprpub/pr/include/private 
../../../../../mozilla/nsprpub/pr/src/io/priometh.c
gcc -o io/./prfdcach.o -c   -pipe -ansi -pthread -g -fPIC  -UNDEBUG 
-DUSE_AUTOCONF=1 -DDEBUG=1 -DDEBUG_cls=1 -DXP_UNIX=1 -D_POSIX_SOURCE=1
-D_BSD_SOURCE=1 -D_SVID_SOURCE=1 -DLINUX=1 -DHAVE_LCHOWN=1 -DHAVE_STRERROR=1
-D_REENTRANT=1  -DFORCE_PR_LOG -D_PR_PTHREADS -UHAVE_CVAR_BUILT_ON_SEM
-D_NSPR_BUILD_ -I/usr/cls/src/obj/dist/include  
../../../../mozilla/nsprpub/pr/src/io/./prfdcach.c
../../../../mozilla/nsprpub/pr/src/io/./prfdcach.c:19: primpl.h: No such file or
directory

Both priometh.o & prfdcach.o should be built in the pr/src/io dir. 
Note the extra includes that priometh.o has.

There are about 3 ways to go about fixing this. 

1) Add the extra includes to INCLUDES in pr/src.  It's quick-n-dirty but works
until another dependency or variable is added to a subdir's Makefile but not
pr/src's .

2) Build sublibraries or archives in the subdirs and link the sublibraries like
we do in Mozilla.  Very ugly, IMO, and just mentioned for completeness.

3) Create a objs.mk in the subdirs which contains the rules (relative to the
nspr tree) for building those object files from any subdirectory.  This way you
can say libnspr.so depends upon $(DEP_OBJS) and it will rebuild them if they do
not exist.

Known adv:
2 & 3 put the logic for building specific object file in a single place.

Known cons:
1 requires that any variables that affect build rules be synchronized across
multiple makefiles.
2 & 3 may require invoking additional make processes to handle the dependencies.

Status: RESOLVED → UNCONFIRMED
Component: Build Config → NSPR
OS: Solaris → All
Priority: P3 → P2
Product: Browser → NSPR
QA Contact: cyeh → granrose
Hardware: Sun → All
Resolution: INVALID → ---
Target Milestone: M18 → ---
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
The quick fix has been checked into the tip & the NSPRPUB_CLIENT_BRANCH:

Checking in pr/src/Makefile;
/cvsroot/mozilla/nsprpub/pr/src/Makefile,v  <--  Makefile
new revision: 3.36; previous revision: 3.35
done
Checking in pr/src/Makefile.in;
/cvsroot/mozilla/nsprpub/pr/src/Makefile.in,v  <--  Makefile.in
new revision: 1.15; previous revision: 1.14
done


Checking in nsprpub/pr/src/Makefile;
/cvsroot/mozilla/nsprpub/pr/src/Makefile,v  <--  Makefile
new revision: 3.30.2.4; previous revision: 3.30.2.3
done
Checking in nsprpub/pr/src/Makefile.in;
/cvsroot/mozilla/nsprpub/pr/src/Makefile.in,v  <--  Makefile.in
new revision: 1.9.2.4; previous revision: 1.9.2.3
done
Status: ASSIGNED → RESOLVED
Closed: 20 years ago19 years ago
Resolution: --- → FIXED
Mine still fails.  I'll attach a log in a sec.
Blizzard,
that is http://bugzilla.mozilla.org/show_bug.cgi?id=31364, already filed.

Marking this one as verified.

Axel
Status: RESOLVED → VERIFIED
Actually, if you look at the build log, he's hitting a problem with multiple gcc
processes writing to the same object file and corrupting it. I'm working on an
extended solution so that object files aren't built in pr/src .
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
I tested this on a 12cpu sparc, on clobbered builds and from scratch.
Works for my configuration, the patch looks ok, so r=axel@pike.org

Axel
Something is strange, I have that patch in my tree, and now it won't build
libnspr4.so. The .done is there, but it won't build the lib. It installs the
links though.
Don't know why this went smooth the first time, and I don't have the time to 
debug this right now.
So I issue a <ouch> here.

Axel
There might be an easy remedy:

in nsprpub/config/rules.mk
change
$(DONE): $(OBJS)
to
$(DONE): $(TARGETS)

This built the lib, and didn't reintroduce the races, as far as I tested.

Hope this helps

Axel
I just checked in a change to the nspr makefiles that removes the unneeded
recursive makes for the libs & install phrases.  With those gone, I don't see to
have a problem doing a 'make -j 25' from nsprpub/ .  This is when using gmake
3.79.1 on a dual processor box and without any of the patches from this bug.


I'm still seeing this w/ a current tree:
this is with 'make -j3' in nsprpub.

cd src; /usr/local/bin/gmake export
gmake[2]: Entering directory `/home/decklin/seamonkey/mozilla/nsprpub/pr/src'
cd io; /usr/local/bin/gmake export
gmake[2]: *** No rule to make target `io/Linux2.4.0_x86_PTH_OPT.OBJ/prfdcach.o',
needed by
 `Linux2.4.0_x86_PTH_OPT.OBJ/libnspr4.a'.  Stop.
gmake[2]: *** Waiting for unfinished jobs....
The classic nspr build is well on its way to being obsolete.  Use
--enable-nspr-autoconf .

Status: REOPENED → ASSIGNED
It looks that everything is fixed after turning that on, i'll be sure once the
current bustage is cleared. thanks.
OK, whatever was causing me problems the other day seems to be fixed; i tried a
few more clobbers last night and this morning w/ various -j settings and
everything went smoothly. Marking fixed. thanks!
Status: ASSIGNED → RESOLVED
Closed: 19 years ago19 years ago
Resolution: --- → FIXED
verified.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.