306790 - FIPS power-up self test fails on 32-bit sparc with dbx runtime check options

Status: RESOLVED INVALID

(NSS :: Libraries, defect)

Description

[Julien Pierre]

The libpkix recently discovered that when they enable the dbx runtime checks,
signature verification fails on Sparc 32 bit, using the freebl library formerly
known as "hybrid" . I have reproduced the problem in a very simple test case :
just run bltest -F to run the FIPS self test.

When running bltest -F under dbx, the test will fail with a CK_RV of 48
(CKR_DEVICE_ERROR) . I traced the failure, and it happens in
sftk_fips_RSA_PowerUpSelfTest :

    if( ( rsa_status != SECSuccess ) ||
        ( PORT_Memcmp( rsa_computed_ciphertext, rsa_known_ciphertext,
                       FIPS_RSA_ENCRYPT_LENGTH ) != 0 ) )

The problem is that the computed ciphertext does not match the known ciphertext
! I believe there may be something wrong in our Sparc floating point code for 32
bit sparc that uses the 32x32 to 64 multiply.

Several tools choke on our Sparc 32 bit multiply code - not just dbx runtime
checks, but also purify .

FYI, the same problem does not exist in the 64 bit version of the floating point
code .

I discussed this problem with Nelson earlier this week and he mentioned that we
were given only one version of the multiply code - either the v8plus or the v9
version, and the other one was a port. If the v8plus code was ported from v9,
this might be a bug introduced during the port .

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

This is not a bug in NSS.  This has been found to be a problem with the dbx 
"check -all" option for v8+ binaries only.  It is not a problem for v8 or v9, 
but only for v8+.  

The suggested workaround is to use v9 builds rather than v8+ builds for 
this "check -all" testing.