867861 - certdata.c does not seem to be generated fast enough when building NSS as part of Firefox on Windows

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Wan-Teh Chang]

I just installed the latest version of MozillaBuild on Windows 7.
I have Visual Studio 2010.

I cloned mozilla-inbound, which has NSS_3_15_BETA2 right now, and ran
the command
    ./mach build

The build failed in NSS:

 6:28.53 /usr/bin/sh: d:/src/mozilla-inbound.2/inbound-src/obj-i686-pc-mingw32/s
ecurity/nss/lib/ckfw/builtins/certdata.c: No such file or directory
 6:28.54 d:\src\mozilla-inbound.2\inbound-src\security\nss\lib\ckfw\builtins\Mak
efile:54:0: command 'perl certdata.perl < certdata.txt > d:/src/mozilla-inbound.
2/inbound-src/obj-i686-pc-mingw32/security/nss/lib/ckfw/builtins/certdata.c' fai
led, return code 1
 6:28.54 <d:/src/mozilla-inbound.2/inbound-src/obj-i686-pc-mingw32/security/nss/
lib/ckfw/builtins/certdata.obj>: Found error
 6:28.54 <d:/src/mozilla-inbound.2/inbound-src/obj-i686-pc-mingw32/security/nss/
lib/ckfw/builtins/nssckbi.dll>: Found error

I found that certdata.c wasn't in the obj directory:

$ ls obj-i686-pc-mingw32/security/nss/lib/ckfw/builtins/
anchor.obj  binst.obj    bsession.obj  btoken.obj   constants.obj  nssckbi.res
bfind.obj   bobject.obj  bslot.obj     ckbiver.obj  nssckbi.def    vc100.pdb

I ran ./mach build again, then it worked.

It seems that this is the problem that Mike Hommey wrote a patch
for (see bug 832272 comment 11). Mike, could you confirm this?
Do you know why Mozilla's build bots can go without your patch?
Is parallel build turned off on the build bots?

Comment 1

[Mike Hommey [:glandium]]

There's a rule in security/build/Makefile.in that works around that lack of fix for what you point to. So I don't think this is it. But without more log, it's hard to say what it is.

Comment 2

[Wan-Teh Chang]

(I am at home now. My home Windows PC has a quad-core AMD CPU. The
Windows PC I use at work has a total of 16 cores or hardware threads.
I only ran into this bug on my Windows PC at work. This comment
reports my debugging on my home Windows PC.)

Mike: thanks for the info. That makefile rule doesn't seem to
be working as intended:

496 # Work around NSS build system race condition creating certdata.c in
497 # security/nss/lib/ckfw/builtins. See bug #836220.
498 libs-nss/lib: $(call mkdir_deps,../nss/lib/ckfw/builtins)

I manually monitored the creation and contents of security/nss/lib/ckfw
in my build tree. I captured two snapshots:

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib/ckfw
crypto.obj  vc90.pdb

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib/ckfw
crypto.obj  hash.obj      mutex.obj   session.obj  slot.obj   vc90.pdb
find.obj    instance.obj  object.obj  sessobj.obj  token.obj  wrap.obj

You can see that the |builtins| subdirectory had not been created
when we were already building in the security/nss/lib/ckfw directory.
In addition, when NSS had been built, there was no .mkdir.done timestamp
file created, which is further proof that the makefile function
    $(call mkdir_deps,../nss/lib/ckfw/builtins)
was not executed.

If I changed that makefile rule as follows:

-libs-nss/lib: $(call mkdir_deps,../nss/lib/ckfw/builtins)
+libs-nss/lib/ckfw: $(call mkdir_deps,../nss/lib/ckfw/builtins)

then it created the security/nss/lib/ckfw/builtins directory,
with a .mkdir.done timestamp file in it, shortly after we
started to build NSS.  Here is my manual monitoring output:

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib
ls: security/nss/lib: No such file or directory

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib
ckfw  jar  util  zlib

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib
certhigh  ckfw  jar  util  zlib

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib
certhigh  ckfw  cryptohi  jar  pk11wrap  util  zlib

...

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib/ckfw
builtins

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls security/nss/lib/ckfw/builtins/

Wan-Teh Chang@PHENOM2-PC /d/mozilla-inbound/src/obj-i686-pc-mingw32
$ ls -a security/nss/lib/ckfw/builtins/
.  ..  .mkdir.done

Comment 3

[Mike Hommey [:glandium]]

ah, thanks Wan-Teh, that would be a regression from bug 648407

Comment 4

[Mike Hommey [:glandium]]

Attachment: Fixup NSS race condition work around after bug 648407

Wan-Teh, can you double-check this works for you?

Comment 5

[Wan-Teh Chang]

Comment on attachment 744635 [details] [diff] [review]
Fixup NSS race condition work around after bug 648407

Mike: this patch fixes the problem for me. Note that your patch in
bug 832272 comment 11 also fixes the problem for me.

Comment 6

[Mike Hommey [:glandium]]

(In reply to Wan-Teh Chang from comment #5)
> Note that your patch in bug 832272 comment 11 also fixes the problem for me.

That's not enough. MAKE_OBJDIR needs to be fixed too. See bug 836220.

Comment 7

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b4ec5d600b90

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/b4ec5d600b90

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 744635 [details] [diff] [review]
Fixup NSS race condition work around after bug 648407

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 648407
User impact if declined: Possible win32 and mac build failure when building nss, although afaik, this hasn't happened on buildbots. But wtc hitting it is proof it can happen.
Testing completed (on m-c, etc.): wtc confirmed it fixes his build failures. m-i and m-c didn't burn on landing :)
Risk to taking this patch (and alternatives if risky): Low
String or IDL/UUID changes made by this patch: None

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/967e8f864531
https://hg.mozilla.org/releases/mozilla-beta/rev/f747f023a3ee