1968502 - C_Encrypt of softokn PKCS#11 API with GCM does not follow the Section 5.2 semantics of PKCS#11

Status: UNCONFIRMED

(NSS :: Libraries, defect, P3)

Description

[Jakub Jelen]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0

Steps to reproduce:

Trying to do AES-GCM encryption, the using the C_Encrypt() PKCS#11 API on softokn results in CKR_BUFFER_TOO_SMALL even though the buffer provided has the size returned from the previous call (with NULL buffer).

I believe this is the case where the softonkn does not account for the TAG length.

Version: nss-softokn-3.110.0-2.fc41.x86_64
(was not in the list of versions above)

Actual results:

8: C_CreateObject
P:382317; T:0x140112265844416 2025-05-26 14:39:47.328
[in] hSession = 0x1000001
[in] pTemplate[4]:
CKA_CLASS CKO_SECRET_KEY
CKA_KEY_TYPE CKK_AES
CKA_VALUE 00007f6e68000e50 / 16
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
CKA_ENCRYPT True
[out] *phObject = 0x1
Returned: 0 CKR_OK

9: C_EncryptInit
P:382317; T:0x140112265844416 2025-05-26 14:39:47.328
[in] hSession = 0x1000001
[in] pMechanism->type = CKM_AES_GCM
[in] pMechanism->pParameter->pIv[ulIvLen] 00007f6e6dd57e20 / 12
00000000 00 00 00 00 00 00 00 00 00 00 00 00 ............
[in] pMechanism->pParameter->ulIvBits = 0x60
[in] pMechanism->pParameter->pAAD[ulAADLen] 00007f6e6dd57e2f / 0
00000000
[in] pMechanism->pParameter->ulTagBits = 96
[in] hKey = 0x1
Returned: 0 CKR_OK

10: C_Encrypt
P:382317; T:0x140112265844416 2025-05-26 14:39:47.328
[in] hSession = 0x1000001
[in] pData[ulDataLen] 00007f6e6dd57e30 / 32
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[out] pEncryptedData[*pulEncryptedDataLen] NULL [size : 0x20 (32)]
Returned: 0 CKR_OK

11: C_Encrypt
P:382317; T:0x140112265844416 2025-05-26 14:39:47.328
[in] hSession = 0x1000001
[in] pData[ulDataLen] 00007f6e6dd57e30 / 32
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[out] pulEncryptedDataLen = 0x20
Returned: 336 CKR_BUFFER_TOO_SMALL

Expected results:

The encryption works and follows the Section 5.2 semantics.

Comment 1

[Jakub Jelen]

There is also an issue with the C_EncryptFinal, that is not strictly following the Section 5.2 semantics. When the Final step does not have any data to return, it finalizes the operation (even though it is just the size query and the part=NULL) so the real attempt to finalize the operation fails:

16: C_EncryptFinal
P:505924; T:0x140465870952128 2025-05-27 14:25:34.926
[in] hSession = 0x1000001
[out] pLastEncryptedPart[*pulLastEncryptedPartLen] NULL [size : 0x0 (0)]
Returned:  0 CKR_OK

17: C_EncryptFinal
P:505924; T:0x140465870952128 2025-05-27 14:25:34.926
[in] hSession = 0x1000001
Returned:  145 CKR_OPERATION_NOT_INITIALIZED

This works well with every other PKCS#11 module that I tried that follows the Section 5.2 semantics.

Comment 2

[Robert Relyea]

The first issue sounds like a real issue we should fix. I suspect there's a layering issue where C_Encrypt is guessing that the output buffer should equal the input buffer because it does most of the time, and doesn't know to call down to freebl to get the real info (or maybe freebl may not have an iterface to tell it.
The second is a well known quirk in softoken. Softoken never really advertised that it was a strict PKCS #11 module. It's primary goal is to implement PKCS #11 well enough to support NSS, and to encourage good behavior in NSS. NSS seldom uses the 'tell me how much space I need' interface, instead prefering to understand enough of the algorithms to figure it out for itself. That means softoken could easily have bugs and weirdness there.

Upshot it's not something that affects NSS directly or Firefox specifically, but it is a good note toward making softoken more compliant to PKCS #11. I seems some people have started using it as a software PKCS #11 module and so it's probably good to start fixing tis. I think benjamin's priority and severity on this is spot on.