2075851 - C_Encrypt with CKM_AES_GCM: the length query omits the tag, and CKR_BUFFER_TOO_SMALL neither reports the needed length nor keeps the operation active

Status: UNCONFIRMED

(NSS :: Libraries, defect, P3)

Description

[Eric Knauel]

Attachment: gcm_encrypt_size.c

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36

Steps to reproduce

build and run the attached gcm_encrypt_size.c against softoken, with a fresh database created by certutil -N:

cc -I <dir containing the OASIS pkcs11.h> -o gcm_encrypt_size gcm_encrypt_size.c -ldl
NSS_LIB_PARAMS=configDir=<db dir> ./gcm_encrypt_size <path to libsoftokn3> <user PIN>

The program:

1. generates a 128-bit AES key;
2. initialises single-part AES-GCM encryption: 12-byte IV, ulIvBits = 96, no AAD, 128-bit tag;
3. encrypts 32 bytes following the two-call convention: a length query with a NULL buffer, then C_Encrypt into a buffer of the announced size;
4. if that returns CKR_BUFFER_TOO_SMALL, retries with a 256-byte buffer;
5. as a control, re-initialises and encrypts once into a 256-byte buffer.

Actual results

size query (NULL buffer):    32 bytes
C_Encrypt,  32 byte buffer: rv 0x150 CKR_BUFFER_TOO_SMALL, length 32
retry,     256 byte buffer: rv 0x91 CKR_OPERATION_NOT_INITIALIZED, length 256

control: single C_Encrypt into 256 bytes: rv 0x0 CKR_OK, length 48

The attached pkcs11-spy log (NSS 3.120) shows the same calls. The encryption itself works (see the control run). But a caller following the two-call convention of section 5.2 cannot get ciphertext: it only succeeds by guessing a large enough buffer before the first call.

Expected results

PKCS#11 3.1 requires three things here, and all three fail:

1. Section 5.2: the length query returns "a number of bytes which would suffice to hold the cryptographic output". Here that is at least 48, i.e. 32 bytes of ciphertext plus a 16-byte tag. NSS returns 32.
2. Section 5.2: "If the buffer is not large enough, then CKR_BUFFER_TOO_SMALL is returned and *pulBufLen is set to at least the number of bytes needed". NSS leaves *pulEncryptedDataLen at 32.
3. Section 5.2.2 (C_Encrypt): "A call to C_Encrypt always terminates the active encryption operation unless it returns CKR_BUFFER_TOO_SMALL or is a successful call [...] to determine the length of the buffer needed". NSS terminates the operation anyway, so the retry fails with CKR_OPERATION_NOT_INITIALIZED.

Comment 1

[Eric Knauel]

Attachment: gcm_encrypt_size-pkcs11-spy.log