429039 - libPKIX abusing NSS environment variables

Status: RESOLVED INCOMPLETE

(NSS :: Libraries, defect, P2)

Description

[Nelson Bolyard (seldom reads bugmail)]

While working on documenting NSS's use of environment variables today in
http://developer.mozilla.org/en/docs/NSS_reference:NSS_environment_variables
I became aware of numerous problems with the ways in which libPKIX is using
environment variables.  In some cases, libPKIX is overloading old environment 
variables with new unrelated meanings.  In other cases, it is using them in
an apparently inconsistent fashion.  

I very much want to see all these things fixed before 3.12.0 RTM.
Please treat this as P1.

#1) PKIX_PL_Initialize uses NSS_STRICT_SHUTDOWN to enable NSPR logging.  See
http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/security/nss/lib/libpkix/pkix_pl_nss/system/pkix_pl_lifecycle.c&rev=1.19#171

This means that one cannot do strict shutdowns without logging, and one cannot
do logging without strict shutdown.  The control of logging needs to be separated from strict shutdown.

#2) confused meaning of PKIX_OBJECT_LEAK_TEST_ABORT_ON_LEAK.
There are two places in certhigh/certvfypkix.c where the code tests for the
presence of the environment variable PKIX_OBJECT_LEAK_TEST_ABORT_ON_LEAK,
and decides whether to "abortOnLeak" or not, depending on its presence.
But in one place, the presence of the variable causes abortOnLeak to be TRUE
and in the other place, it causes the variable to be FALSE.  One of those 
is wrong.  I think the error is at line 2046.

#3) SOCKTTRACE violates the principle of least astonishment.
In non-optimized builds, libPKIX's socket tracing feature is compiled in
and is enabled by default.  There is an environment variable that pretends 
to control it, but it actually serves to DISABLE tracing if it is defined
with any value other than "1".  Nothing in the name suggests that it is 
related to libPKIX.  

The desired changes are:
a) turn socket tracing OFF by default.
b) Change the name to include NSS and/or PKIX, e.g. PKIX_SOCKET_TRACE

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

Oh, and the page 
http://developer.mozilla.org/en/docs/NSS_reference:NSS_environment_variables
must be updated when these changes are made.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

On the web page http://wiki.mozilla.org/NSS:Tracing Kai has documented some
of the problems mentioned in this bug, including 

- one must define NSS_STRICT_SHUTDOWN to get any NSPR logging from libPKIX

He also documents that: "NSS_TRACE_OCSP="1" is supposed to enable OCSP tracing. However, it's not sufficient."  I'll file a separate bug about that.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

This really is P1 for 3.12.1 at Sun.
I will take this and write a patch.