261333 - New PK11_PubEncryptPKCS1 function always fails

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P1)

Description

[Nelson Bolyard (seldom reads bugmail)]

In NSS 3.9.3, we added several new functions.  One of them is failing in tests.
This makes NSS 3.9.3 unusable by one internal "customer" for whom it is 
being released.  

The bug is in function pk11_PubEncryptRaw.  It initializes the variable "out",
which is the size of the output buffer, with the length of data in the input
buffer.  This was always wrong.  The output buffer length always should be 
the length of the modulus (exclusing leading zero bytes in the modulus).  
But it always worked because the SSL2 code always passed in an input buffer 
whose length matched the modulus length.  

So, this is an old bug, now surfacing.  I should have caught this in my 
testing, and didn't.  Mea Culpa.  

I'd like to fix this for NSS 3.9.3 (even though the release candidate was
built yesterday).

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

tentatively marking P1 for 3.9.3

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Attachment: Patch v1

fix incorrect output buffer length.  (fix tested with customer program).

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

Thanks for the quick review, Jullien.  Checked in on 3.9 branch.

Checking in pk11skey.c;
/cvsroot/mozilla/security/nss/lib/pk11wrap/pk11skey.c,v  <--  pk11skey.c
new revision: 1.84.2.5; previous revision: 1.84.2.4

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

Checked in on trunk.

Checking in pk11obj.c;
/cvsroot/mozilla/security/nss/lib/pk11wrap/pk11obj.c,v  <--  pk11obj.c
new revision: 1.3; previous revision: 1.2