In NSS 3.9.3, we added several new functions. One of them is failing in tests.
This makes NSS 3.9.3 unusable by one internal "customer" for whom it is
The bug is in function pk11_PubEncryptRaw. It initializes the variable "out",
which is the size of the output buffer, with the length of data in the input
buffer. This was always wrong. The output buffer length always should be
the length of the modulus (exclusing leading zero bytes in the modulus).
But it always worked because the SSL2 code always passed in an input buffer
whose length matched the modulus length.
So, this is an old bug, now surfacing. I should have caught this in my
testing, and didn't. Mea Culpa.
I'd like to fix this for NSS 3.9.3 (even though the release candidate was
tentatively marking P1 for 3.9.3
Created attachment 159940 [details] [diff] [review]
fix incorrect output buffer length. (fix tested with customer program).
Thanks for the quick review, Jullien. Checked in on 3.9 branch.
Checking in pk11skey.c;
/cvsroot/mozilla/security/nss/lib/pk11wrap/pk11skey.c,v <-- pk11skey.c
new revision: 220.127.116.11; previous revision: 18.104.22.168
Checked in on trunk.
Checking in pk11obj.c;
/cvsroot/mozilla/security/nss/lib/pk11wrap/pk11obj.c,v <-- pk11obj.c
new revision: 1.3; previous revision: 1.2