87865 - Mac-Cannot get certs-No public key generated.

Status: VERIFIED FIXED

(Core Graveyard :: Security: UI, defect, P1)

Description

[John Unruh]

1.) Visit http://www.verisign.com/client/enrollment/index.html and request a 60 
day free cert.
What happens: The following page states that "The public key was not generated"
2.) Visit the above in-house URL and try any of the CMS 4.2 tests.
What happens: After you click submit, the watch cursor appears for a few seconds 
and then returns you to the same page. No cert appears in the cert manager.

Comment 1

[Stephane Saux]

-> javi
P1
target -> 2.0

Comment 2

[Bob Lord]

The keygen process is required to get Verisign or intranet certs.  This is a
critical bug to fix.

Comment 3

[Javier Delgadillo]

This is fialing in RSA_NewKey

This function call fails:
http://lxr.mozilla.org/mozilla/source/security/nss/lib/freebl/rsa.c#264

trying to generate a prime for p.

I'm gonna need nelsonb and mcgreer to help on this since they know what this
code is supposed to be doing.

Comment 4

[Javier Delgadillo]

Attachment: Patch to fix keygen in NSS for the Mac.

Comment 5

[Javier Delgadillo]

The problem was in using the sizeof operator on the variable sieve.  On all
platforms except the Mac, sieve was a static array of size 32*1024 (SIEVE_SIZE).

On the Mac, it was of type unsigned char* (this is due to not being able to
allocate that much memory on the stack for the Mac).  So a loop that was
supposed to execute 32*1024 times was only executing 4 times on the Mac (which
is sizeof unsigned char*)

That patch replaces all occurences of "sizeof sieve" with "SIEVE_SIZE" which was
already defined in the file.

Comment 6

[Ian McGreer]

I worked with Javi on this and believe it is correct.  So r=mcgreer.

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

Yes, this patch looks right to me.

Comment 8

[Christopher Blizzard (:blizzard)]

sr=blizzard

Comment 9

[Javier Delgadillo]

This has been checked-in to trunk builds of Mozilla and to the trunk of NSS.

Comment 10

[Javier Delgadillo]

Fixed checked into 0.9.2 branch as well.

Comment 11

[John Unruh]

Verified.