Closed Bug 94187 Opened 23 years ago Closed 23 years ago

cmsutil will not work in FIPS mode

Categories

(NSS :: Libraries, defect, P1)

3.2.1

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bugz, Assigned: julien.pierre)

References

Details

Attachments

(2 files)

The new S/MIME library does often passes NULL as the wincx variable in calls
below the API.  This prevents the passing of a password handler down to PKCS#11,
which will cause problems in FIPS mode.

There may be other problems.  For fun, I tried using cmsutil while in FIPS mode
and watched it crash all over.
Target this for NSS 3.4.
Priority: -- → P1
Target Milestone: --- → 3.4
Assigned the bug to Julien.
Assignee: wtc → jpierre
Ian,

Will this affect the browser as well, if running in FIPS mode ?
It depends on PSM's implementation.  I'm fairly sure that the problems are a
result of the FIPS token not receiving the key database password.  PSM, in
general, does a much better job of handling passwords than our own tools.

However, there may be other bugs.  The following patch is for tests/smime.sh,
once you apply the patch all the databases used in the S/MIME test will be in
FIPS mode during the tests.  You can see what happens - all but two fail.
cc ddrinan.
David, would you please check how PSM deals with passwords in FIPS mode while
using the s/mime libraries.
I thought I should look into this a little more.  Unfortunately, I bring only
bad news.

When Christian wrote the S/MIME libraries, he defined some of the object
initialization calls to take a PK11PasswordFunc callback and an argument (e.g.,
NSS_CMSDecoder_Start).  I presume this was to avoid having to propagate "void
*wincx" arguments throughout the API, but it does not work.  First of all, those
functions then set the *global* PKCS#11 password callback.  This is in no way
context-specific, or thread-safe, and will simply not work with PSM.

I don't know if you've come across this problem yet, David, but there is a
possible solution for part of it.  That is to always pass NULL to functions
looking for a PKCS11PasswordFunc, and let your global password handler take care
of things.  However, you need to pass the window context through the next argument.

For example, set:

PK11_SetPasswordFunc(myglobalfunc);

then call NSS_CMSEncoder_Start with pwfn=NULL and pwfn_arg=wincx.

Another problem is that NSS_CMSMessage_Create does not store pwfn_arg with the
NSSCMSMessage, which it needs to do.  A way to work around this is to call
NSS_CMSMessage_SetEncodingParams after creating the CMSMessage.

I will attach a patch I made to cmsutil that reflects these changes, and gets it
nearly working in FIPS mode. 
Is it an option to extend the APIs to take the needed wincx argument, in order
to make this work better, rather than patch and use the global callback ?
I wouldn't think that would be a 3.X option, given that it breaks backward
compatibility.  OTOH, only PSM is using this library (that we know of).

I agree that the long term solution is to review the API as a whole.  But I
don't see how we could take that on now.

Using the global callback is fairly standard practice in NSS.  However, most
functions allows you to set the parameter passed through to that callback
(wincx).  None of the CMS functions do; you can only set it once, as I showed in
the patch.
Even though we can't change the existing functions to take
an extra wincx parameter, we can add new functions (with
different names) that take the needed wincx parameter.

Will that work, Ian?
Wan-Teh,
You responded before I got a chance - this is in fact what I wanted to do, add
new APIs with the extra parameters. We can deprecate the existing ones in 3.4
without eliminating them. We can delete them in the next major release.
IIRC, the S/MIME API was never really properly reviewed.  There are other minor
problems, like the fact that it doesn't quite use the NSS 4.0 naming scheme
(NSS_CMSSignedData_XXX instead of NSSCMSSignedData_XXX).

I would say that extending portions of the API for 3.X releases is overkill for
this bug, given that we know how to fix it within the current API spec.  I would
prefer to save API redefinition for a later time, when we have a more complete
idea of the changes we would like to make.  But that's just my preference, and
extending the API is certainly one way to fix this bug.
Ian,

You mentioned in today's NSS meeting that this problem might be solved by your
patch. However, I only see a patch for cmsutil, nothing in the S/MIME library.
Should I assume that we need to have PSM do the same sort of thing as your patch
does to cmsutil in order to get FIPS mode to work with S/MIME in PSM ?
That's correct.  At this time, the problem needs to be worked around by the
application.

However, currently the libraries are blocking this bug.  See 115660.
Depends on: 115660
Wan-Teh,

As the release date for NSS 3.4 is nearing, do we want to simply let the
application work around this problem and move this bug to 4.0 ?
Bob, Nelson, Ian,

Do you think we need to fix this in NSS 3.4?  If
applications can work around this bug (see comment #15),
I suggest that we lower the priority of this bug and
move it to 4.0.
I agree.
What exactly is the workaround?  

I don't think we should release NSS 3.4 unless S/MIME can be used in 
FIPS mode by PSM.
The workaround is shown in the second attachment.  This patch shows what changes
had to be made to cmsutil for it to run in FIPS mode.

The problem was described in comment #7.  I don't remember exactly what
functions were problematic, but I found that the changes shown in the patch were
able to consistently provide the correct argument to the password callback function.
Ian,

1. Can we say that although the S/MIME library will not work in
FIPS mode, cmsutil now works in FIPS mode with your workaround?

2. Does the workaround rely on some global state to implicitly
pass the required info to the S/MIME functions?  If so, the
workaround may not work when multiple threads are doing S/MIME
simultaneously.

3. Is it a lot of work to implement the correct solution (new
functions that take the additional argument)?  Have we identified
all the S/MIME functions that won't work in FIPS mode?
Also, would fixing the listing of certs issue fix this problem for the FIPs
case. (I have a fix for that bug now as well).

BTW, we don't loose our FIPS certification if we allow certs, public keys, and
trust to be found in the FIPs token if the token hasn't been logged in do we?

bob
> 1. Can we say that although the S/MIME library will not work in
> FIPS mode, cmsutil now works in FIPS mode with your workaround?

Perhaps the subject is misleading.  Because cmsutil will work in FIPS mode with
the changes in the patch, the S/MIME library *will* work in FIPS mode.  It's
just in a non-obvious way.  That is, the API purports to take the password
callback function as an argument to some functions, but then doesn't use it
correctly.  The patch shows how to force correct usage.  In other words, it's
not that the library won't work, but that the API is deficient.  Fixing the API
is certainly a Future issue, IMO.

> 2. Does the workaround rely on some global state to implicitly
> pass the required info to the S/MIME functions?  If so, the
> workaround may not work when multiple threads are doing S/MIME
> simultaneously.

No.

> 3. Is it a lot of work to implement the correct solution (new
> functions that take the additional argument)?  Have we identified
> all the S/MIME functions that won't work in FIPS mode?

I'm not sure.  I would have to start over by letting cmsutil fail without the
patch, and track down what functions failed.

I don't consider the patch to be that ugly of a workaround, as it is.  There are
simply two things to note:

1.  Don't pass an value to the PK11PasswordFunc argument when calling
NSS_CMSDecoderStart and NSS_CMSEncoderStart, but do pass a value to the pwfn_arg
argument.  The value to pass is the usual wincx variable.

2.  After creating an NSSCMSMessage to encode, call
NSS_CMSMessage_SetEncodingParams, and for the third argument pass the wincx
variable.

> BTW, we don't loose our FIPS certification if we allow certs, public keys, and
> trust to be found in the FIPs token if the token hasn't been logged in do we?

We received a validation with those operations as public functions.  They always
were public, therefore never required login.
Hmm. Actually they way Ian described the CMS calls is exactly how the calls
worked in PKCS7. The inline password function was a depricated function in
PKCS7. It sounds to me that there is nothing to do for NSS 3.4 here. 

bob
Changed the bug summary to "cmsutil will not work in FIPS mode".
Marked the bug fixed.
Status: NEW → RESOLVED
Closed: 23 years ago
Resolution: --- → FIXED
Summary: S/MIME library will not work in FIPS mode → cmsutil will not work in FIPS mode
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: