941690 - Disable use of Hardware AES GCM acceleration support if platform (hardware of OS) does not support it

Status: RESOLVED INVALID

(NSS :: Libraries, defect)

Description

[Elio Maldonado]

A crash in intel_aes_gcmINI was reported on Bug 940794 and this bug is in response to the request made on https://bugzilla.mozilla.org/show_bug.cgi?id=940794#c7 to bring a Red Hat patch here. 

Hardware acceleration GCM support support Intel must be not only in the processor but also the operating system. On RHEL-5, for example, we had a case where the hardware support was there in the processor but due to an older version of Linux the support is not really there. We discovered this while running the bapi test suite.

bltest -T -m aes_gcm -d /builddir/build/BUILD/nss-3.14.3/mozilla/security/nss/tests/../cmd/bltest -1 7 -2 6
./cipher.sh: line 57: 10272 Illegal instruction     (core dumped) ${PROFTOOL} ${BINDIR}/bltest -T -m $PARAM -d $CIPHERTESTDIR -1 $inOff -2 $outOff
bltest -T -m aes_gcm -d /builddir/build/BUILD/nss-3.14.3/mozilla/security/nss/tests/../cmd/bltest -1 7 -2 7
./cipher.sh: line 57: 10275 Illegal instruction     (core dumped) ${PROFTOOL} ${BINDIR}/bltest -T -m $PARAM -d $CIPHERTESTDIR -1 $inOff -2 $outOff
cipher.sh: #15: AES GCM (Failed in/out offset pairs:  [0:0][0:1][0:2][0:3][0:4][0:5][0:6][0:7][1:0][1:1][1:2][1:3][1:4][1:5][1:6][1:7][2:0][2:1][2:2][2:3][2:4][2:5][2:6][2:7][3:0][3:1][3:2][3:3][3:4][3:5][3:6][3:7][4:0][4:1][4:2][4:3][4:4][4:5][4:6][4:7][5:0][5:1][5:2][5:3][5:4][5:5][5:6][5:7][6:0][6:1][6:2][6:3][6:4][6:5][6:6][6:7][7:0][7:1][7:2][7:3][7:4][7:5][7:6][7:7]) - Core file is detected - FAILED
.
The test would succeed but there was a core dump.

The fix was to perform check that build time environment variable DISABLE_HW_GCM=1 was is set  and do some checks at runtime to determine if we needed to disable HW GCM. 

The patch used by Red Hat is at ttps://bugzilla.redhat.com/attachment.cgi?id=778910
This problem is general and we must test for the support being there. The aforementioned patch be adapted to the current state of the sources.

Comment 1

[Elio Maldonado]

Typo, the patch is at https://bugzilla.redhat.com/attachment.cgi?id=778910

Comment 2

[Wan-Teh Chang]

I looked at the patch in comment 1.

With the patch for bug 940794 checked in, your patch doesn't
seem useful any more.

In the unlikely case that intel-gcm.s has bugs, I think the
existing workaround of NSS_DISABLE_HW_AES is sufficient.
We should not need to add an environment variable for every
new assembly code file.