288401 - FIPS is broken in all command-line tools that use NSS static libraries

Status: NEW

(NSS :: Libraries, defect, P3)

Description

[Julien Pierre]

I discovered this while working on the rsaperf PKCS#11 changes.
If I enabled FIPS with modutil, NSS_Init would fail with CKR_DEVICE_ERROR .

I traced the failure to the signature check, specifically the following :

t@1 (l@1) stopped in BLAPI_SHVerify at line 104 in file "shvfy.c"
  104       char *shName = PR_GetLibraryFilePathname(name, addr);
(dbx) p name
name = 0x65d5e8 "libsoftokn3.so"
(dbx) p addr
addr = 0x5385e0 = &`rsaperf`pkcs11.c`pk11_closePeer(PRBool isFIPS)
(dbx) n
t@1 (l@1) stopped in BLAPI_SHVerify at line 105 in file "shvfy.c"
  105       char *checkName = NULL;
(dbx) p shName
shName = 0x69c8e0
"/h/monstre/export/home/tip/mozilla/security/nss/cmd/rsaperf/SunOS5.10_i86pc_64_DBG.OBJ/rsaperf"
(dbx)

As you can see above, when BLAPI_SHVerify calls PR_GetLibraryFilePathName, it
passes a pointer to a function, pk11_closePeer. Because rsaperf has NSS
statically linked, the pointer returns an address in the rsaperf executable, and
shName is the name of the executable, "rsaperf". There is no signature file for
rsaperf, and therefore the FIPS signature check fails !

There are several possible courses of action :

1) Do something differently in the code that's linked statically, perhaps skip
the check

2) sign every executable that has NSS linked statically using shlibsign

3) try to get rid of all the programs in cmd that have USE_STATIC_LIBS=1

In the specific case of rsaperf, the fix would be to link only libfreebl.a to
the program, rather than all of NSS.

Comment 1

[Julien Pierre]

The programs affected are :
bltest
certcgi
crmf-cgi
fipstest
makepqg
ocspclnt
rsaperf
shlibsign
signtool
signver
swfort

Comment 2

[Robert Relyea]

I would vote for #3. 

freebl and rsapref are the only ones that I believe may have to link statically
(though they really just need to link with freebl. One question, though, you've
made changes to rsapref so that it can use both NSS and freebl in it's
performance testing. Do you mean that it should just link with freebl statically
and NSS dynamically?

shlibsign may also be an issue. It's the program that actually generates the
signatures.

swfort should go away with nelson's fortezza removal.

How many of the programs listed are actually compiled?

bob

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

Bob asked:
> Do you mean that it should just link with freebl statically and NSS dynamically?

That seems like a good idea to me ... at least until we make freebl a 
stand-alone shared lib on all platforms (which I want to do for 3.11).

Bob, what do you think of that idea?

Comment 4

[Wan-Teh Chang]

shlibsign must not check the .chk file, otherwise
we have a chicken-and-egg problem.  The easiest
solution is to not run shlibsign in FIPS mode.

Comment 5

[Julien Pierre]

Bob,

Of the list, only the following 3 programs aren't currently compiled :
crmf-cgi
fipstest
swfort

Re: rsaperf, yes, I mean that it should link freebl statically and the rest of
NSS dynamically, as opposed to everything statically, which it does at the
moment. I believe both bltest and rsaperf need to link with freebl statically.

Nelson,

I like the idea of making freebl a shared library. I think you should file an
RFE for that for 3.11.

Wan-Teh,

Indeed, we shouldn't use shlibsign in FIPS mode to do the signing of the
softoken and freebl libraries during the NSS build. However, shlibsign could be
used subsequently in FIPS mode to sign other files, if this bug was fixed.

Comment 6

[Julien Pierre]

FYI, here is the list of static libraries linked to those programs :

EXTRA_LIBS += \
        $(DIST)/lib/$(LIB_PREFIX)smime.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)ssl.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)nss.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)ssl.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)sectool.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)pkcs12.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)pkcs7.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)certhi.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)pk11wrap.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)cryptohi.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)certhi.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)nsspki.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)pk11wrap.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)softokn.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)certdb.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)nsspki.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)nssdev.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)nssb.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)swfci.$(LIB_SUFFIX) \
        $(CRYPTOLIB) \
        $(DIST)/lib/$(LIB_PREFIX)secutil.$(LIB_SUFFIX) \
        $(DIST)/lib/$(LIB_PREFIX)dbm.$(LIB_SUFFIX) \
        $(NULL)

You may note that pk11wrap and certhi both appear twice ... Is there any reason
for that ?

Comment 7

[Robert Relyea]

Historical reasons.

At one point we had several circular dependencies in our libraries. On some
platforms we had to include the libraries twice to resolve them. I suspect they
are no longer needed.

bob

Comment 8

[Julien Pierre]

FYI, I verified that linking freebl statically to rsaperf and the rest of NSS
dynamically does not allow FIPS mode to work. On Solaris, the multiple copies of
freebl in memory cause a problem with initialization. Only one of them gets
initialized, and nothing works right subsequently.

The only way to get things working properly is to make freebl a shared library
of its own. When there is a bug for that RFE, it should be set to block this bug.

Comment 9

[glen beasley]

we learned that FIPS-140-2 required the use of shared libraries/dlls.Only thing
we can do is to modify the command line tools is to link with the shared
libraries/dlls. 

Should we close this bug as invalid?

Comment 10

[Julien Pierre]

Glen,

The bug is valid - many programs can't run in FIPS mode today. The fix to make
FIPS work is to make the programs all use the dynamic libraries.

Certain programs (eg. rsaperf, bltest) need access to freebl functions directly,
not through softoken  . In order for them to continue to work, freebl has to be
a shared library. It would have to be made that way for all platforms, even if
there is only a single implementation for the particular platform .

shlibsign doesn't run in FIPS mode by definition, since it generates the chk
files. And it already uses shared libraries.