2029901 - Heap buffer overflow write in [@ sftk_ike_prf_plus_raw] via integer overflow in PR_ROUNDUP(CKA_VALUE_LEN, macSize)

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P3)

Description

[Bugmon [:jkratzer for issues]]

NSS softoken's IKEv2 PRF+ key derivation (sftk_ike_prf_plus_raw in security/nss/lib/softoken/sftkike.c) computes the output buffer size as outKeySize = PR_ROUNDUP(keySize, macSize) where both operands are 32-bit unsigned int. keySize is taken directly from the caller-supplied CKA_VALUE_LEN attribute in NSC_DeriveKey (pkcs11c.c:8415) with no upper-bound validation. When keySize is in the range [UINT_MAX - macSize + 2, UINT_MAX], the addition inside PR_ROUNDUP ((x + y - 1) / y * y) wraps modulo 2^32 and yields outKeySize == 0. PORT_Alloc(0) then returns a minimal 1-byte heap allocation.

The subsequent expansion loop is bounded by the original (unwrapped) keySize, so it still executes and calls prf_final(&context, thisKey, macSize) which memcpys a full HMAC block (e.g. 32 bytes for SHA-256) into the 1-byte buffer on the very first iteration, and would continue advancing thisKey += macSize for up to 255 iterations (until the currentByte == 255 guard fires), giving up to 255 * macSize bytes of out-of-bounds heap writes of PRF output in a non-ASAN build.

The defect is reachable through the standard PKCS#11 C_DeriveKey(CKM_IKE2_PRF_PLUS_DERIVE) entry point (also CKM_NSS_IKE_PRF_PLUS_DERIVE) and through the exported NSS wrapper PK11_DeriveWithTemplate. The same PR_ROUNDUP overflow pattern also exists in sftk_ike1_appendix_b_prf (sftkike.c) reachable via CKM_IKE1_EXTENDED_DERIVE. Any application that loads NSS softoken and forwards an attacker-influenced derived-key length to one of these mechanisms (e.g. an IKE/IPsec daemon, or any code exposing PKCS#11 derive to less-trusted input) is exposed to heap memory corruption. The IKE code path is exercised by the FIPS power-on self-tests in security/nss/lib/softoken/fipstest.c (sftk_fips_IKE_PowerUpSelfTests).

Build Info

• Branch: main
• Revision: 4b851f6b592ecf1112ee47dd25e8de28c892ad67
• Timestamp: 2026-04-04T17:51:39+00:00

Affected Code

File: security/nss/lib/softoken/sftkike.c, line 923-985

    macSize = prf_length(&context);
    outKeySize = PR_ROUNDUP(keySize, macSize);            // 32-bit overflow -> 0
    outKeyData = PORT_Alloc(outKeySize);                  // returns 1-byte buffer
    if (outKeyData == NULL) {
        crv = CKR_HOST_MEMORY;
        goto fail;
    }
    ...
    thisKey = outKeyData;
    for (getKeySize = 0; getKeySize < keySize; getKeySize += macSize) {  // keySize still ~UINT_MAX
        if (currentByte == 255) {
            crv = CKR_KEY_SIZE_RANGE;
            goto fail;
        }
        ...
        crv = prf_final(&context, thisKey, macSize);      // OOB write of macSize bytes
        ...
        lastKey = thisKey;
        thisKey += macSize;
    }

File: security/nss/lib/softoken/pkcs11c.c, line 8414-8416

        if (pTemplate[i].type == CKA_VALUE_LEN) {
            keySize = *(CK_ULONG *)pTemplate[i].pValue;   // attacker-controlled, no upper bound
        }

File: security/nss/lib/softoken/pkcs11c.c, line 8559-8568

        case CKM_NSS_IKE_PRF_PLUS_DERIVE:
        case CKM_IKE2_PRF_PLUS_DERIVE:
            if (pMechanism->ulParameterLen !=
                sizeof(CK_IKE2_PRF_PLUS_DERIVE_PARAMS)) {
                crv = CKR_MECHANISM_PARAM_INVALID;
                break;
            }
            crv = sftk_ike_prf_plus(hSession, att,
                                    (CK_IKE2_PRF_PLUS_DERIVE_PARAMS *)pMechanism->pParameter,
                                    key, keySize);        // keySize passed as unsigned int
            break;

File: nsprpub/pr/include/prtypes.h, line 157

#define PR_ROUNDUP(x,y) ((((x)+((y)-1))/(y))*(y))

keySize flows from the caller's CKA_VALUE_LEN template attribute in NSC_DeriveKey (no range check) into sftk_ike_prf_plus -> sftk_ike_prf_plus_raw as a 32-bit unsigned int. PR_ROUNDUP performs (keySize + macSize - 1) in 32-bit unsigned arithmetic; with keySize = 0xFFFFFFFF and macSize = 32 this wraps to 30, yielding outKeySize = 0. PORT_Alloc(0) succeeds (1-byte region), so the NULL check is bypassed, and the loop driven by the original huge keySize writes full HMAC blocks past the allocation via prf_final/HMAC_Finish/SHA256_End memcpy.

Exploit Chain

1. Caller (or attacker who controls input reaching the PKCS#11 layer) issues C_DeriveKey / PK11_DeriveWithTemplate with mechanism CKM_IKE2_PRF_PLUS_DERIVE and a derive template containing CKA_VALUE_LEN = 0xFFFFFFFF (or any value >= UINT_MAX - macSize + 2).
2. NSC_DeriveKey copies CKA_VALUE_LEN into local keySize without bounds checking and dispatches to sftk_ike_prf_plus -> sftk_ike_prf_plus_raw.
3. PR_ROUNDUP(keySize, macSize) overflows 32-bit unsigned arithmetic to 0; PORT_Alloc(0) returns a valid 1-byte heap pointer, bypassing the NULL/CKR_HOST_MEMORY check.
4. The PRF+ expansion loop runs (bounded by the unwrapped keySize and the 255-iteration counter) and each iteration memcpy's macSize bytes (20/32/48/64 depending on PRF) into and beyond the 1-byte buffer.
5. Result is a controlled-length (up to 255*macSize bytes) heap-buffer-overflow WRITE of PRF output, corrupting adjacent heap metadata/objects and enabling potential code execution in the process hosting softoken.

Steps to Reproduce

1. Build NSS with AddressSanitizer (e.g. ./build.sh --asan from security/nss).
2. Append the provided GTest Pkcs11IkeOverflowTest.PrfPlusKeySizeOverflow to security/nss/gtests/pk11_gtest/pk11_ike_unittest.cc and rebuild pk11_gtest.
3. Run: dist/Debug/bin/pk11_gtest --gtest_filter=Pkcs11IkeOverflowTest.PrfPlusKeySizeOverflow
4. Observe ASAN heap-buffer-overflow WRITE of size 32 at sftk_ike_prf_plus_raw (sftkike.c:979), allocation of 1 byte at sftkike.c:925.

Security Impact

• Severity: High
• Attacker capability: Heap buffer overflow WRITE of up to 255 * macSize bytes (e.g. ~8 KB for HMAC-SHA256, ~16 KB for HMAC-SHA512) of PRF-derived data past a 1-byte allocation inside the softoken PKCS#11 module. This is a memory-safety violation that can corrupt adjacent heap objects and may lead to arbitrary code execution in the process hosting NSS softoken.
• Preconditions: An attacker must be able to influence the CKA_VALUE_LEN attribute passed to C_DeriveKey for CKM_IKE2_PRF_PLUS_DERIVE / CKM_NSS_IKE_PRF_PLUS_DERIVE (or CKM_IKE1_EXTENDED_DERIVE) on the NSS softoken. This is directly reachable by any local PKCS#11 client of softoken; remote exploitability depends on the embedding application (e.g. an IKE/VPN daemon or other consumer) forwarding an untrusted key-length value to this derivation mechanism. Not directly reachable from web content in Firefox.

ASAN Report

==15360==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5020000a6e91 at pc 0x7c452dd73303 bp 0x7ffdb679e240 sp 0x7ffdb679d9e8
WRITE of size 32 at 0x5020000a6e91 thread T0
    #0 0x7c452dd73302 in memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115
    #1 0x7c4529eaca76 in SHA256_End ../../lib/freebl/sha512.c:591
    #2 0x7c4529d3bf55 in RawHash_SHA256_End ../../lib/freebl/rawhash.c:51
    #3 0x7c4529cbb560 in HMAC_Finish ../../lib/freebl/alghmac.c:177
    #4 0x7c452c9ce784 in HMAC_Finish ../../lib/freebl/loader.c:1533
    #5 0x7c452c9b9a0f in prf_final ../../lib/softoken/sftkike.c:412
    #6 0x7c452c9bbd9c in sftk_ike_prf_plus_raw ../../lib/softoken/sftkike.c:979
    #7 0x7c452c9bc0d9 in sftk_ike_prf_plus ../../lib/softoken/sftkike.c:1015
    #8 0x7c452c987fca in NSC_DeriveKey ../../lib/softoken/pkcs11c.c:8566
    #9 0x7c452db7f8a5 in PK11_DeriveWithTemplate ../../lib/pk11wrap/pk11skey.c:1783
    #10 0x59c2f37038bc in nss_test::Pkcs11IkeOverflowTest_PrfPlusKeySizeOverflow_Test::TestBody() ../../gtests/pk11_gtest/pk11_ike_unittest.cc:233

0x5020000a6e91 is located 0 bytes after 1-byte region [0x5020000a6e90,0x5020000a6e91)
allocated by thread T0 here:
    #0 0x7c452dd759c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7c452d7486e7 in PR_Malloc ../../../../pr/src/malloc/prmem.c:425
    #2 0x7c452da0a9fa in PORT_Alloc_Util ../../lib/util/secport.c:87
    #3 0x7c452c9bbac8 in sftk_ike_prf_plus_raw ../../lib/softoken/sftkike.c:925
    #4 0x7c452c9bc0d9 in sftk_ike_prf_plus ../../lib/softoken/sftkike.c:1015
    #5 0x7c452c987fca in NSC_DeriveKey ../../lib/softoken/pkcs11c.c:8566
    #6 0x7c452db7f8a5 in PK11_DeriveWithTemplate ../../lib/pk11wrap/pk11skey.c:1783
    #7 0x59c2f37038bc in nss_test::Pkcs11IkeOverflowTest_PrfPlusKeySizeOverflow_Test::TestBody() ../../gtests/pk11_gtest/pk11_ike_unittest.cc:233

SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115 in memcpy
==15360==ABORTING

Comment 1

[Bugmon [:jkratzer for issues]]

Attachment: pk11_ike_unittest.cc

Comment 2

[Bugmon [:jkratzer for issues]]

Attachment: fix.patch

Comment 3

[Bugmon [:jkratzer for issues]]

Attachment: crash_stack.txt

Comment 5

[Dennis Jackson]

IKE is not used in Fx or Thunderbird. The write being PRF output makes it uncontrolled

Comment 6

[Benjamin Beurdouche [:beurdouche]]

Attachment: (secure)

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/nss/rev/8e7d01c13d0e
Guard against keySize overflow in IKE PRF/PRF+ output sizing. r=rrelyea,nss-reviewers