Closed Bug 941690 Opened 11 years ago Closed 7 years ago

Disable use of Hardware AES GCM acceleration support if platform (hardware of OS) does not support it


(NSS :: Libraries, defect)

Not set


(Not tracked)



(Reporter: elio.maldonado.batiz, Unassigned)



A crash in intel_aes_gcmINI was reported on Bug 940794 and this bug is in response to the request made on to bring a Red Hat patch here. 

Hardware acceleration GCM support support Intel must be not only in the processor but also the operating system. On RHEL-5, for example, we had a case where the hardware support was there in the processor but due to an older version of Linux the support is not really there. We discovered this while running the bapi test suite.

bltest -T -m aes_gcm -d /builddir/build/BUILD/nss-3.14.3/mozilla/security/nss/tests/../cmd/bltest -1 7 -2 6
./ line 57: 10272 Illegal instruction     (core dumped) ${PROFTOOL} ${BINDIR}/bltest -T -m $PARAM -d $CIPHERTESTDIR -1 $inOff -2 $outOff
bltest -T -m aes_gcm -d /builddir/build/BUILD/nss-3.14.3/mozilla/security/nss/tests/../cmd/bltest -1 7 -2 7
./ line 57: 10275 Illegal instruction     (core dumped) ${PROFTOOL} ${BINDIR}/bltest -T -m $PARAM -d $CIPHERTESTDIR -1 $inOff -2 $outOff #15: AES GCM (Failed in/out offset pairs:  [0:0][0:1][0:2][0:3][0:4][0:5][0:6][0:7][1:0][1:1][1:2][1:3][1:4][1:5][1:6][1:7][2:0][2:1][2:2][2:3][2:4][2:5][2:6][2:7][3:0][3:1][3:2][3:3][3:4][3:5][3:6][3:7][4:0][4:1][4:2][4:3][4:4][4:5][4:6][4:7][5:0][5:1][5:2][5:3][5:4][5:5][5:6][5:7][6:0][6:1][6:2][6:3][6:4][6:5][6:6][6:7][7:0][7:1][7:2][7:3][7:4][7:5][7:6][7:7]) - Core file is detected - FAILED
The test would succeed but there was a core dump.

The fix was to perform check that build time environment variable DISABLE_HW_GCM=1 was is set  and do some checks at runtime to determine if we needed to disable HW GCM. 

The patch used by Red Hat is at ttps://
This problem is general and we must test for the support being there. The aforementioned patch be adapted to the current state of the sources.
I looked at the patch in comment 1.

With the patch for bug 940794 checked in, your patch doesn't
seem useful any more.

In the unlikely case that intel-gcm.s has bugs, I think the
existing workaround of NSS_DISABLE_HW_AES is sufficient.
We should not need to add an environment variable for every
new assembly code file.
Closed: 7 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.