326498 - softoken returns wrong length when output buffer too small

Status: NEW

(NSS :: Libraries, defect, P3)

Description

[Nelson Bolyard (seldom reads bugmail)]

PKCS#11 v2.20 section 11.2 entitled
"Conventions for functions returning output in a variable-length buffer" says

> 2. If pBuf is not NULL_PTR, then *pulBufLen must contain the size in bytes of
> the buffer pointed to by pBuf. If that buffer is large enough to hold the
> cryptographic output produced from the input to the function, then that
> cryptographic output is placed there, and CKR_OK is returned by the function.
> If the buffer is not large enough, then CKR_BUFFER_TOO_SMALL is returned. In
> either case, *pulBufLen is set to hold the exact number of bytes needed to
> hold the cryptographic output produced from the input to the function.

Notice that last sentence.  
Even if a CKR_BUFFER_TOO_SMALL error occurs, the function must output 
"the exact number of bytes needed to hold the ... output produced from 
the input to the function."

This applies only to those functions that are defined to follow section 11.2,
which include: C_GetOperationState, C_Encrypt, C_EncryptUpdate, C_EncryptFinal,
C_Decrypt, C_DecryptUpdate, C_DecryptFinal, C_Digest, C_DigestFinal, 
C_Sign, C_SignFinal, C_SignRecover, C_VerifyRecover, C_DigestEncryptUpdate,
C_DecryptDigestUpdate, C_SignEncryptUpdate, C_DecryptVerifyUpdate, C_WrapKey,
and C_UnwrapKey (for CKM_KEY_WRAP_SET_OAEP only).

I looked at some of the code in NSC_EncryptUpdate and NSC_DecryptUpdate and
found that the results vary.  In some cases, we don't set the length at 
all on errror. In others, it has been set to a possibly incorrect value.

I don't think that NSS's own PK11wrap layer depends on this behavior, but
other software might.  Andreas?

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

Marking P3, but we may revise this if anyone cares.

Comment 2

[Konstantin Andreev]

The case, when CKR_BUFFER_TOO_SMALL occur, is one of the dark places of PKCS#11 (see relevant discussions in "cryptoki" mail list @ RSA lab). Returning exact number of bytes is minor thing to support. 

IMO, applications shouldn't rely on particular behavior of CKR_BUFFER_TOO_SMALL at all.