In NSS 3.9.3, we added several new functions. One of them is failing in tests. This makes NSS 3.9.3 unusable by one internal "customer" for whom it is being released. The bug is in function pk11_PubEncryptRaw. It initializes the variable "out", which is the size of the output buffer, with the length of data in the input buffer. This was always wrong. The output buffer length always should be the length of the modulus (exclusing leading zero bytes in the modulus). But it always worked because the SSL2 code always passed in an input buffer whose length matched the modulus length. So, this is an old bug, now surfacing. I should have caught this in my testing, and didn't. Mea Culpa. I'd like to fix this for NSS 3.9.3 (even though the release candidate was built yesterday).
tentatively marking P1 for 3.9.3
Created attachment 159940 [details] [diff] [review] Patch v1 fix incorrect output buffer length. (fix tested with customer program).
Thanks for the quick review, Jullien. Checked in on 3.9 branch. Checking in pk11skey.c; /cvsroot/mozilla/security/nss/lib/pk11wrap/pk11skey.c,v <-- pk11skey.c new revision: 220.127.116.11; previous revision: 18.104.22.168
Checked in on trunk. Checking in pk11obj.c; /cvsroot/mozilla/security/nss/lib/pk11wrap/pk11obj.c,v <-- pk11obj.c new revision: 1.3; previous revision: 1.2