266158 - Bad encryption /decryption with PK11_PubEncryptRaw and PK11_PubDecryptRaw

Status: RESOLVED INVALID

(NSS :: Libraries, defect)

Description

[Philippe Camacho]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040524
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040524

We take 300 plain text messages. Then we encrypt each of them with
PK11_PubEncryptRaw. We obtain 300 new messages. Then we decrypt these new
messages (with the same key of course and using PK11_PubDecryptRaw). We should
obtain the 300 plain text messages, but this is not the case: some decrypted
messages are good (i.e. equal to the plain) and some are not. It depends...

Reproducible: Always
Steps to Reproduce:
1.Encrypt N different messages (with N big)
2.Decrypt them
3.Compare the result with the original messages
(use PK11_PubEncryptRaw PK11_PubDecryptRaw)

Actual Results:  
Some decrypted message correspond to the plain text messages. Some not.

Expected Results:  
Every decrypted message should be equal to the plain text one.

You can find the source that shows the problem at
http://www.ducktools.com/bugzilla/NSSPKCS1/

Comment 1

[Wan-Teh Chang]

Nelson, could you take a look at this?

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

The RSA public and private key algorithms require that the input value,
which is treated like a large integer, be numerically less than the 
value of the modulus.  The output values will only be the ones you 
expect them to be if the inputs meet the requirements.  PKCS1 v1.5
encoding ensures this, among other things, but the Raw interface
purposefully avoid PKCS1.

The test program produces input values that will be greater than the 
modulus in some significant percentage of the cases.  

BTW, there is a good chance that, in the cases where the output value
is not as expected, if you add the modulus to the output value (as a
big integer), you will get the expected value.