836220 - To allow parallel build of nss

Status: RESOLVED FIXED

(NSS :: Build, defect)

Description

[Mike Hommey [:glandium]]

+++ This bug was initially created as a clone of Bug #832272 +++

I'm going to work around the NSS deficiencies in bug 832272, so I'm creating a separate bug to address these NSS deficiencies.

The first one is that of https://bugzilla.mozilla.org/attachment.cgi?id=707044 , but it is tied to the second one.

The second one is that MAKE_OBJDIR is racy on unix. It expands to:
    if test ! -d $(@D); then rm -rf $(@D); $(NSINSTALL) -D $(@D); fi
When you have, for example, a directory with several C source files, each C source is compiled to an object file, but MAKE_OBJDIR is called before each. When doing so in parallel, the rm -rf can actually run after one of the C source files was compiled, and the file then goes missing.

Comment 1

[Mike Hommey [:glandium]]

As noted by Martin when he filed bug 832272, there's also the issue of private_export racing with export when doing make -j export.

Comment 2

[Mike Hommey [:glandium]]

There is also an issue with TARGETS containing IMPORT_LIBRARY on windows, while there's no rule for IMPORT_LIBRARY. This creates a race condition when trying to run these targets in parallel.

Comment 3

[barry_davis]

Is anyone looking at this? I'm building nss-3.30.2 with 56 cores and this builds so slowly as I'm forced to only build on one of them.

  if test ! -d $(@D); then rm -rf $(@D); mkdir $(@D); fi

So if a dir doesn't exist then try deleting it then try making it, but fail if someone else made it first... I can't see how that ever made sense. It would be better replaced with:

  mkdir -p $(@D)

As other pointed out, this doesn't fix everything.


--- nss-3.30.2/nss/coreconf/mkdepend/Makefile   2017-05-25 17:35:36.158582657 +0100
+++ nss-3.30.2/nss/coreconf/mkdepend/Makefile   2017-05-25 17:36:28.238993949 +0100
@@ -55,6 +55,6 @@
 
 # Redefine MAKE_OBJDIR for just this directory
 define MAKE_OBJDIR
-if test ! -d $(@D); then rm -rf $(@D); mkdir $(@D); fi
+mkdir -p $(@D)
 endef
 
--- nss-3.30.2/nss/coreconf/nsinstall/Makefile  2017-05-25 17:35:43.028636910 +0100
+++ nss-3.30.2/nss/coreconf/nsinstall/Makefile  2017-05-25 17:37:20.619407576 +0100
@@ -37,6 +37,6 @@
 
 # Redefine MAKE_OBJDIR for just this directory
 define MAKE_OBJDIR
-if test ! -d $(@D); then rm -rf $(@D); mkdir $(@D); fi
+mkdir -p $(@D)
 endef

Comment 4

[(not currently active) Ted Mielczarek]

We implemented an alternate build system that uses gyp for the frontend and ninja for the backend, you should try that instead:
https://wiki.mozilla.org/NSS/Build_System

It will do parallel builds and also does proper dependency tracking.

Comment 5

[Franziskus Kiefer [:franziskus]]

Ted is right. We have a parallel build system now since 8 month (3.28 I think). I think we can resolve this.