511515 - NSS is forking on init during startup

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

We already know we want to get rid of NSS initialization during startup, but until we can do that, we should fix the problems in there.  On OSX, it RNG_SystemInfoForRNG is forking twice to call netstat, first from:

#
#0  0x92ccbfc0 in fork ()
#1  0x1cd81cc3 in RNG_SystemInfoForRNG ()
#2  0x1cda43d1 in rng_init ()
#3  0x00040a33 in PR_CallOnce ()
#4  0x1cda4475 in RNG_RNGInit ()
#5  0x00562bf0 in nsc_CommonInitialize ()
#6  0x00562e0f in NSC_Initialize ()
#7  0x0046df01 in secmod_ModuleInit ()
#8  0x0046e4de in SECMOD_LoadPKCS11Module ()
#9  0x00478863 in SECMOD_LoadModule ()
#10 0x004788c7 in SECMOD_LoadModule ()
#11 0x004511ba in nss_Init ()
#12 0x004515e2 in NSS_InitReadWrite ()
#13 0x02877b1f in nsNSSComponent::InitializeNSS (this=0x1cfdeb40, showWarningBox=1) at ../../../../../mozilla/security/manager/ssl/src/nsNSSComponent.cpp:1607

and then from:

#0  0x92ccbfc0 in fork ()
#1  0x1cd81cc3 in RNG_SystemInfoForRNG ()
#2  0x00562c0a in nsc_CommonInitialize ()
#3  0x00562e0f in NSC_Initialize ()
#4  0x0046df01 in secmod_ModuleInit ()
#5  0x0046e4de in SECMOD_LoadPKCS11Module ()
#6  0x00478863 in SECMOD_LoadModule ()
#7  0x004788c7 in SECMOD_LoadModule ()
#8  0x004511ba in nss_Init ()
#9  0x004515e2 in NSS_InitReadWrite ()
#10 0x02877b1f in nsNSSComponent::InitializeNSS (this=0x1cfdeb40, showWarningBox=1) at ../../../../../mozilla/security/manager/ssl/src/nsNSSComponent.cpp:1607


So two problems here:

1) RNG_SystemInfoForRNG is getting called twice
2) RNG_SystemInfoForRNG should not fork (ever, really).

Comment 1

[Mike Shaver (:shaver emeritus)]

OS X has /dev/random, from what I read, which provides at least 160 bits of entropy via a Yarrow impl.  Probably more than netstat provides, and maybe a bit faster, too.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Why is this 12 year old behavior in FIPS validated code suddenly critical?

Comment 3

[Mike Shaver (:shaver emeritus)]

Startup performance is an area of focus for us (as widely discussed in Mozilla project meetings, blogs, and such), and unfortunately we initialize NSS during startup.  Critical might be an overstatement here, in the context of NSS; please accept the adjustment with my heartfelt apologies.

I don't imagine that FIPS validation gave us feedback on how much entropy we were gathering through netstat, so that we know what might be sufficient to replace it?

(I've been meaning to ask: what does FIPS test, that corresponds to quality of product?  I don't really know if they look at performance, test under memory-correctness tools, or whatever else.  URLs welcome.)

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

I think I should resolve this bug as incomplete for having no version 
number information.  

The first half of this complaint appears to be a duplicate of bug 489811, 
which is already fixed in the version of NSS now included in FF 3.5.x 
and (IINM) FF 3.0.x.

Comment 5

[Mike Shaver (:shaver emeritus)]

Vlad is using a post-3.5 trunk build, I believe, but I'll let him fill in the NSS version info.  I can't tell from bug 489811 if it landed on the version that's currently used for trunk (mozilla-central) Firefox builds.  If not, I hope that we make sure it's included in whatever version of NSS we're considering taking in the future on 3.5.x, to get CRLDP and other such!

If bugs that don't have version number specified should be resolved INCOMPLETE (rather than just asking for more information), then I do wonder why "unspecified" is one of the permitted version options!

Comment 6

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

How do I figure out the NSS version from a build?  It's Gecko 20090729, and running strings on libnss3 gives:

@(#)NSS 3.12.4.1 Basic ECC Beta  Jun 12 2009 19:07:49

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

@Johnathan,
> I do wonder why "unspecified" is one of the permitted version options!

Me too!  I don't know how it got there.  I'd like to see it removed.

@Vlad,
3.12.4.1 is one of the release candidates of NSS 3.12.4 
That's the number I wanted.  

Bug 489811 was fixed in the next subsequent release candidate: 3.12.4.2.
The final candidate was 3.12.4.5, released last week.

Comment 8

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Great -- then we can ignore problem #1 here.

Comment 9

[Mike Shaver (:shaver emeritus)]

OK, is there a bug on advancing Firefox's tag or whatever to that version?  I'll happily make sure that's addressed promptly, I just didn't see anything in bug 489811 that indicated we needed to do anything else to get the fix onto trunk (and therefore 3.6, etc.).

Thanks for the version guidance, a smaller problem to deal with here now.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

[Bug 504080] Upgrade 1.9.1 to an NSS 3.12.4
[Bug 511413] Move 1.9.2 branch to NSS 3_12_4_RTM

Comment 11

[Mike Shaver (:shaver emeritus)]

Great, thanks!

Is there one for trunk?

Comment 12

[Nelson Bolyard (seldom reads bugmail)]

> Is there one for trunk?

Already done.  See bug 504080 comment 25

Comment 13

[Mike Beltzner [:beltzner, not reading bugmail]]

I approved bug 511413 for the mozilla-1.9.2 branch.

Comment 14

[Mike Beltzner [:beltzner, not reading bugmail]]

So is this FIXED by those other bugs, then?

Comment 15

[Wan-Teh Chang]

No.  NSS is still forking during NSS initialization on Mac OS X.
Those other bugs only affect whether NSS forks once or twice.

Comment 16

[Mike Beltzner [:beltzner, not reading bugmail]]

Wan-Teh: anyone planning on working on this?
Vlad: what's the expected perf bonus here?

Need this information to make a blocking decision.

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

Mike, this is in FIPS validated code which is now frozen. 
Changing this now invalidates the FIPS validation (for which MoFo has 
contracted) before MoCo even ships it.

Comment 18

[Wan-Teh Chang]

This bug should not block mozilla 1.9.2.  We have created
a SOFTOKEN_3_13_BRANCH for changes to files that are locked
down for FIPS.  Let's fix this bug on that branch, and the
fix will appear in a future version of NSS.

Comment 19

[Mike Shaver (:shaver emeritus)]

Money spent on FIPS certification by MoFo should not, and I predict will not, have significant influence on what we do with the Firefox product.  If we need FIPS validation for Firefox 3.6, then *that* can affect our decision about what to ship, and where.

If FIPS certification continues to be a major barrier to improvements that we need to the codebase, perhaps amplified by the current architecture of NSS (c.f. the Camellia change, which I don't think is inherently subject to FIPS scrutiny), then we will have to figure out how to weigh that trade-off.  We shouldn't presume either outcome (FIPS over improvements, or all improvements at the cost of FIPS) at this stage.

Comment 20

[Wan-Teh Chang]

Attachment: Proposed patch (for SOFTOKEN_3_13_BRANCH)

Vlad, please test this patch on the Mac.

Let's get a fix into the SOFTOKEN_3_13_BRANCH first.

Comment 21

[Wan-Teh Chang]

Comment on attachment 399541 [details] [diff] [review]
Proposed patch (for SOFTOKEN_3_13_BRANCH)

I found that Kaspar Brand proposed a good patch in
bug 174993 comment 19, which got two reviews.  But
for some reason I don't remember (probably because
of Nelson's cautionary comment in bug 174993 comment
23), I told Kai Engert in bug 174993 comment 36 to
check in just the Linux part of that patch.  So it
seems that this bug is my fault.

I propose that we use Kaspar Brand's patch (attachment
272917 [review]) instead.  What do you think?

Comment 22

[Doug Turner (:dougt)]

although this may be a separate issue/bug, but why don't we disable NSS's entropy collection and have the application solely be responsible for initial collection.  When we need to startup NSS, we can pass our entropy bits to it.  I would guess (we should measure.  hamming tools in 466745), that the application can collect much better entropy (application start-up time, lengths of files read in, ect.) than the library.  I think this would minimize the startup costs of NSS.

Comment 23

[Nelson Bolyard (seldom reads bugmail)]

> Why don't we disable ... ?

Because its expressly disallowed by the relevant NIST/FIPS docs.  You can 
always add more, but you can't disable the module's own entropy gathering.

Comment 24

[Doug Turner (:dougt)]

I am not sure we/i care about FIPS on mobile platforms.  Do you?  Would you image there could be a mode of NSS that allow for such a feature?

Comment 25

[Nelson Bolyard (seldom reads bugmail)]

The problem is that we have one source tree from which Mozilla builds all 
browsers of a certain version.  That source tree gets FIPS certified for
some platforms at some point in time.  If you subsequently change it, then 
it is no longer certified for any platform.  

I think the answer to this is that Mozilla needs two source trees for NSS,
one that is FIPS certified and one that is not.  The FIPS sources should be
used to build on (at least) the FIPS certified platforms, and the non-FIPS 
sources may be used on any other platforms.  Getting to that point would 
not be a huge adjustment for the NSS team, IMO, but I think it would be a 
huge adjustment for MoCo.  

Another way out of this is for MoCo to separate the NSS builds from the 
rest of the browser builds.  Build NSS separately, and combine the NSS 
shared libs (which MoCo has built) into the various browsers when the 
browsers are built.  But I suspect that's an even bigger adjustment for 
MoCo than having two NSS trees.

Comment 26

[Doug Turner (:dougt)]

can't we just have a build configure and use the same tree?  e.g. if an environment variable is set or if a #define is set, NSS will not collect, but instead will require that the application layer do entropy collection?

Comment 27

[Nelson Bolyard (seldom reads bugmail)]

The next time we do a FIPS evaluation, we can do anything we like. 
But we just finished a FIPS evaluation, and the source code is now frozen
in the evaluated state.  Any change we make now will invalidate that evaluation.

I propose that we keep the present NSS tree intact and inviolate, and make a 
new NSS tree that is, by DEFINITION, not a FIPS evaluated tree, and use that 
tree ONLY for those platforms that are not FIPS targets (like WinCE and other
PDAs).  In that tree, you can play all you want.

Comment 28

[Wan-Teh Chang]

Attachment: Kaspar Brand's patch (checked in to Softoken 3.13 and 3.12.7)

Skip safe_popen() calls on platforms with /dev/urandom.

This is Kaspar Brand's patch from bug 174993 comment 19,
updated for the SOFTOKEN_3_13_BRANCH.  I also added
HP-UX to the list.  From our FIPS 140-2 validation docs:
(https://wiki.mozilla.org/VE_07KeyMgmt#Key_Generation)

  We obtain 1024 bytes from the system random number
  generator. ... On Solaris, HP-UX, Linux, and Mac OS X,
  we read from the special device /dev/urandom.

    On HP-UX B.11.11, one must install the HP-UX Strong
    Random Number Generator (KRNG11i) bundle to add the
    /dev/random and /dev/urandom special files. The
    KRNG11i bundle can be downloaded for free from the
    HP software depot, and installation is
    straightforward.

I examined the nss packages in FreeBSD, NetBSD, and
OpenBSD.
- FreeBSD: NSS 3.12.4.  Does not patch unix_rand.c.
- NetBSD (and DragonFly): NSS 3.11.5.  Does not patch
  unix_rand.c.
- OpenBSD: NSS 3.12.3.1.  Patches unix_rand.c to use
  /dev/arandom instead of /dev/urandom, and to not
  fork netstat.
  http://www.openbsd.org/cgi-bin/cvsweb/ports/security/nss/Makefile?rev=1.15
  http://www.openbsd.org/cgi-bin/cvsweb/ports/security/nss/patches/patch-mozilla_security_nss_lib_freebl_unix_rand_c?rev=1.6

I checked in this patch on the SOFTOKEN_3_13_BRANCH
(Softoken 3.13).

Checking in unix_rand.c;
/cvsroot/mozilla/security/nss/lib/freebl/unix_rand.c,v  <--  unix_rand.c
new revision: 1.38.4.1; previous revision: 1.38
done

Comment 29

[Wan-Teh Chang]

Comment on attachment 400315 [details] [diff] [review]
Kaspar Brand's patch (checked in to Softoken 3.13 and 3.12.7)

I checked in this patch on the NSS trunk (3.12.7) for FIPS revalidation.

Checking in unix_rand.c;
/cvsroot/mozilla/security/nss/lib/freebl/unix_rand.c,v  <--  unix_rand.c
new revision: 1.39; previous revision: 1.38
done