75770 - RSA_NewKey returns an incomplete key

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Jamie Nicolson]

Under certain circumstances, RSA_NewKey will return an incomplete key. This
causes a SEGV in the PKCS #11 code when it tries to access one of the key's
components, which is NULL.

Here's what happens: in the do loop at rsa.c:261, rsa_keygen_from_primes fails,
with PORT_GetError() returning SEC_ERROR_NEED_RANDOM, which gets stored into
prerr. Then the loop repeats, and this time rsa_keygen_from_primes returns
SECSuccess, so we break out of the loop. However, prerr is still
SEC_ERROR_NEED_RANDOM, so after the loop we "goto cleanup" before setting
key->prime1 and key->prime2.

Cleanup thinks the operation was a success because err is 0 and rv is
SECSuccess, so it returns the incomplete key as if nothing was wrong.

Comment 1

[Jamie Nicolson]

Fixed by setting prerr=0 at the beginning of the loop each time. Reviewed by
mcgreer and nelsonb.

Checked in the following change to rsa.c on the tip:

RCS file: /cvsroot/mozilla/security/nss/lib/freebl/rsa.c,v
retrieving revision 1.24
retrieving revision 1.25
diff -r1.24 -r1.25
38c38
<  * $Id: rsa.c,v 1.24 2001/01/31 15:49:19 mcgreer%netscape.com Exp $
---
>  * $Id: rsa.c,v 1.25 2001/04/12 20:23:42 nicolson%netscape.com Exp $
261a262
>       prerr = 0;

Comment 2

[Jamie Nicolson]

Marking this fixed.