1062903 - FIPS mode test fails on x32

Status: UNCONFIRMED

(NSS :: Libraries, defect, P4)

Description

[Daniel Schepler]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36

Steps to reproduce:

I've added USE_X32=1 to the Debian packaging for 2:3.17-1 (for some reason I don't see 3.17 in the version drop-down).  The library builds and gets signed just fine, but when the packaging runs the FIPS mode test, it gets an error.


Actual results:

LD_LIBRARY_PATH=debian/libnss3/usr/lib/x86_64-linux-gnux32:debian/libnss3/usr/lib/x86_64-linux-gnux32/nss debian/libnss3-tools/usr/bin/modutil -fips true -dbdir debian/tmp < /dev/null
ERROR: ld.so: object 'libfakeroot-sysv.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

WARNING: Performing this operation while the browser is running could cause
corruption of your security databases. If the browser is currently running,
you should exit browser before continuing this operation. Type 
'q <enter>' to abort, or <enter> to continue: 
A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot.
ERROR: Unable to switch FIPS modes.



Expected results:

FIPS mode enabled.

Comment 1

[Daniel Schepler]

Reproduced using nss_3.17.orig.tar.gz (i.e. no Debian patches applied) with commands:

make all MOZILLA_CLIENT=1 NSPR_INCLUDE_DIR=/usr/include/nspr NSPR_LIB_DIR=/usr/lib SOURCE_PREFIX=/tmp/nss/nss-3.17/dist SOURCE_MD_DIR=/tmp/nss/nss-3.17/dist DIST=/tmp/nss/nss-3.17/dist BUILD_OPT=1 NS_USE_GCC=1 OPTIMIZER="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -pipe -D_FORTIFY_SOURCE=2" LD_FLAGS='-Wl,-z,relro $(ARCHFLAG) $(ZDEFS_FLAG)' DSO_LDOPTS='-shared $(LDFLAGS)' NSS_USE_SYSTEM_SQLITE=1 NSS_ENABLE_ECC=1 CHECKLOC= USE_X32=1
cd ../dist
for l in libsoftokn3.so libfreebl3.so libnssdbm3.so; do LD_LIBRARY_PATH=lib bin/shlibsign -v -i lib/$l; done
mkdir -p debian/tmp
LD_LIBRARY_PATH=lib bin/modutil -create -dbdir debian/tmp < /dev/null
LD_LIBRARY_PATH=lib bin/modutil -fips true -dbdir debian/tmp < /dev/null

Comment 2

[Mike Hommey [:glandium]]

That might suggest something very wrong with NSS itself. You should try to figure out why that CKR_DEVICE_ERROR is being returned.

Comment 3

[Adam Borowski]

Works for me with 3.21

Comment 4

[Adam Borowski]

Attachment: disable Intel AES on x32

It looks like the failure happens only on certain Intel CPUs.  I have no idea which ones but so far all reported to fail were i7 models.

I've narrowed it down to intel_aes_*_worker() in freebl.  Alas, I have no knowledge of Intel crypto opcodes, thus I can't fix this myself.  But, as x32 is currently a little-used architecture, temporarily disabling the broken acceleration is an option (let's first make it correct, then think about making it fast).  Here's a patch that disables Intel AES there, guarded with appropriate #ifdefs so amd64 isn't touched.

Comment 5

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I think the problem is that the hard-coded struct offsets in intel-aes.s are wrong.  Here's the struct:

struct AESContextStr
{
    unsigned int   Nb;
    unsigned int   Nr;
    freeblCipherFunc worker;
    /* NOTE: The offsets of iv and expandedKey are hardcoded in intel-aes.s.
     * Don't add new members before them without updating intel-aes.s. */
    unsigned char iv[RIJNDAEL_MAX_BLOCKSIZE];
    PRUint32      expandedKey[RIJNDAEL_MAX_EXP_KEY_SIZE];
    freeblDestroyFunc destroy;
    void	      *worker_cx;
    PRBool	      isBlock;
    int           mode;
};

The assembly code assumes that `iv` is at offset 16, but if `worker` is a 32-bit pointer then it would be at offset 12.  Similarly, expandedKey (also hardcoded) would move from 40 to 36.

Assuming the x32 ABI passes pointers like 32-bit integers — i.e., in 32-bit registers leaving the upper half unspecified — then the assembly will also need to be fixed to not use the 64-bit registers as pointers (or to zero-extend them first), like in bug 1215681.

It would be nice not to introduce even more code duplication than there already is here (GNU/AT&T vs. Intel, ECB vs. CBC vs. CTR, 128 vs. 192 vs. 256, etc.).  I'm not sure why we're using hard-coded integers instead of macros; the comments in bug 459248 suggest that might have been needed for Solaris compatibility, but I don't know if that's still the case.

Comment 6

[Laurent Bigonville]

Hello,

Any news about this bug?

nss still FTBFS with the last version

Comment 7

[Dennis Jackson]

Hi Laurent,

Your other comment referred to an issue on amd64. Are you also building 32-bit NSS in FIPS mode?

Comment 8

[Laurent Bigonville]

Hello Dennis,

Yes indeed, I was just looking at the architectures where nss FTBFS in debian and seeing what could be done

https://buildd.debian.org/status/package.php?p=nss

Comment 9

[Laurent Bigonville]

Hello,

Any news from this?