1848183 - SHA fails with Large Data Tests - FIPS related

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P3)

Description

[Samuel Henrique :samueloph]

Attachment: nss_test_large_data.c

Steps to reproduce:

Hello,

Our FIPS lab (checking for FIPS140-3), when checking for the Amazon Linux
package for NSS (at 3.90), came back with an issue related to the large data
tests.

During ACVP testing of NSS, we noticed that the Large Data Tests fail.
These tests are part of the ACVP tests, introduced in 2020 by NIST. The
test provides data input greater than 4GB to see if the module can
handle this input data correctly. NSS does not.

Attached is a proof of concept file showing that data input to
C_DigestUpdate is subject to 32-bit integer wrap-around, causing a hash
collision for this data input. If your NSS is installed in a
non-standard location, you can compile and execute it as follows:

$ LIBRARY_PATH=/opt/nss/lib C_INCLUDE_PATH=/opt/nss/include/nss:/opt/nss/include/nspr gcc -Wall -pedantic -Werror -Wno-format-extra-args -g -lnspr4 -lnss3 -o nss_test_large_data nss_test_large_data.c
$ LD_LIBRARY_PATH=/opt/nss/include/nss:/opt/nss/include/nspr ./nss_test_large_data
/opt/nss/lib/libsoftokn3.so
Digest: 785b0751fc2c53dc14a4ce3d800e69ef9ce1009eb327ccf458afe09c242c26c9
Digest: 785b0751fc2c53dc14a4ce3d800e69ef9ce1009eb327ccf458afe09c242c26c9

Possible solutions pointed out by our lab:

• Set service indicator to non-approved if size is greater than 4GB
• Reject inputs greater than 4GB
• Fix the underlying issue by moving from unsigned int to size_t (e.g.
  https://github.com/nss-dev/nss/blob/master/lib/freebl/sha_fast.c#L131).

I have not seen any commits on the git repo which could address this,
so I assume it also impacts the current HEAD.

While we look into fixing this for Amazon Linux (hopefully in a way that the
patches will be useful to you as upstream), I wanted to report the issue so
you're aware of it as well.

Note: I'm a Debian Developer but this bug report is being done as part
of my work at Amazon Linux.

Comment 1

[Anna Weine]

Hello,

Thanks for the report. We will take a look at the bug as soon as we can :)

Comment 2

[John Schanck [:jschanck]]

Attachment: Bug 1848183 - softoken: iterate hashUpdate calls for long inputs. r=rrelyea

Comment 3

[John Schanck [:jschanck]]

This affects softoken when used as a standalone library on systems where long has strictly greater rank than int. Applications that use softoken through NSS are not affected; the public NSS functions that call NSC_Digest and NSC_DigestUpdate, PK11_DigestOp and PK11_DigestKey, both take an unsigned int for their length argument.

I'm attaching a patch for NSC_Digest and NSC_DigestUpdate. After applying the patch, I get

$ ./nss_test_large_data
Digest: 785b0751fc2c53dc14a4ce3d800e69ef9ce1009eb327ccf458afe09c242c26c9
Digest: 8009037a4403ebb6cd30de9516ff1338aeb39f288cae3d8192afee05f9bcf48c

There are probably some similar problems with other PKCS11 functions, but I'll wait to file those until we're satisfied with the fix for the digest functions.

Comment 4

[John Schanck [:jschanck]]

https://hg.mozilla.org/projects/nss/rev/0e3085381a9512988bf79f7fa76eb517f9e076ee