492888 - RSA premaster secret error in JSS test on Niagara

Status: RESOLVED INVALID

(JSS Graveyard :: Library, defect)

Description

[Julien Pierre]

This happened in tinderbox on our Niagara machine :

http://tinderbox.mozilla.org/showlog.cgi?log=NSS/1242247550.1242258189.9237.gz&fulltext=1

============= SSL Ciphersuite JSS Server with Bypass Off and JSSE client 
 ./startJssSelfServ.sh /export/tinderlight/data/pool_64_OPT/mozilla/dist/SunOS5.10_64_OPT.OBJ/../xpclass.jar /export/tinderlight/data/pool_64_OPT/mozilla/tests_results/jss/pool.1 localhost 8476 bypassOff /opt/jdk/1.6.0_01/SunOS64/jre/bin/java -d64 
/opt/jdk/1.6.0_01/SunOS64/jre/bin/java -d64 -classpath /export/tinderlight/data/pool_64_OPT/mozilla/dist/SunOS5.10_64_OPT.OBJ/../xpclass.jar org.mozilla.jss.tests.JSS_SelfServServer /export/tinderlight/data/pool_64_OPT/mozilla/tests_results/jss/pool.1 passwords localhost false 8476 bypassOff verboseoff &
***FilePasswordCallback returns m1oZilla
JSS_SelfServServ localhost ready to accept connections on 8476

SSL Server is envoked using port 8476 
/opt/jdk/1.6.0_01/SunOS64/jre/bin/java -d64 -cp /export/tinderlight/data/pool_64_OPT/mozilla/dist/SunOS5.10_64_OPT.OBJ/../xpclass.jar org.mozilla.jss.tests.JSSE_SSLClient /export/tinderlight/data/pool_64_OPT/mozilla/tests_results/jss/pool.1 8476 localhost JSS 
using port: 8476
Testing Connection:localhost:8476
connect isBound
javax.net.ssl.SSLKeyException: RSA premaster secret error
	at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:97)
	at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:574)
	at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:197)
	at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:511)
	at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:449)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:817)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1029)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1056)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1040)
	at org.mozilla.jss.tests.JSSE_SSLClient.testSSLSocket(JSSE_SSLClient.java:432)
	at org.mozilla.jss.tests.JSSE_SSLClient.testCiphersuites(JSSE_SSLClient.java:195)
	at org.mozilla.jss.tests.JSSE_SSLClient.main(JSSE_SSLClient.java:579)
Caused by: java.security.InvalidKeyException: wrap() failed
	at sun.security.pkcs11.P11RSACipher.engineWrap(P11RSACipher.java:395)
	at javax.crypto.Cipher.wrap(DashoA13*..)
	at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:95)
	... 11 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_MECHANISM_INVALID
	at sun.security.pkcs11.wrapper.PKCS11.C_WrapKey(Native Method)
	at sun.security.pkcs11.P11RSACipher.engineWrap(P11RSACipher.java:391)
	... 13 more
JSSTEST_CASE 24 (SSL Ciphersuite JSS Server with Bypass Off and JSSE client): FAILED return value 1

I am worried about the "RSA premaster secret error" . This could affect FIPS.

Comment 1

[Julien Pierre]

Same problem in test case 25 :

============= SSL Ciphersuite JSS Server with Bypass On and JSSE client 
 ./startJssSelfServ.sh /export/tinderlight/data/pool_64_OPT/mozilla/dist/SunOS5.10_64_OPT.OBJ/../xpclass.jar /export/tinderlight/data/pool_64_OPT/mozilla/tests_results/jss/pool.1 localhost 8477 bypass /opt/jdk/1.6.0_01/SunOS64/jre/bin/java -d64 
/opt/jdk/1.6.0_01/SunOS64/jre/bin/java -d64 -classpath /export/tinderlight/data/pool_64_OPT/mozilla/dist/SunOS5.10_64_OPT.OBJ/../xpclass.jar org.mozilla.jss.tests.JSS_SelfServServer /export/tinderlight/data/pool_64_OPT/mozilla/tests_results/jss/pool.1 passwords localhost false 8477 bypass verboseoff &
***FilePasswordCallback returns m1oZilla
JSS_SelfServServ localhost ready to accept connections on 8477

SSL Server is envoked using port 8477 
/opt/jdk/1.6.0_01/SunOS64/jre/bin/java -d64 -cp /export/tinderlight/data/pool_64_OPT/mozilla/dist/SunOS5.10_64_OPT.OBJ/../xpclass.jar org.mozilla.jss.tests.JSSE_SSLClient /export/tinderlight/data/pool_64_OPT/mozilla/tests_results/jss/pool.1 8477 localhost JSS 
using port: 8477
Testing Connection:localhost:8477
connect isBound
javax.net.ssl.SSLKeyException: RSA premaster secret error
	at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:97)
	at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:574)
	at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:197)
	at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:511)
	at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:449)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:817)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1029)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1056)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1040)
	at org.mozilla.jss.tests.JSSE_SSLClient.testSSLSocket(JSSE_SSLClient.java:432)
	at org.mozilla.jss.tests.JSSE_SSLClient.testCiphersuites(JSSE_SSLClient.java:195)
	at org.mozilla.jss.tests.JSSE_SSLClient.main(JSSE_SSLClient.java:579)
Caused by: java.security.InvalidKeyException: wrap() failed
	at sun.security.pkcs11.P11RSACipher.engineWrap(P11RSACipher.java:395)
	at javax.crypto.Cipher.wrap(DashoA13*..)
	at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:95)
	... 11 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_MECHANISM_INVALID
	at sun.security.pkcs11.wrapper.PKCS11.C_WrapKey(Native Method)
	at sun.security.pkcs11.P11RSACipher.engineWrap(P11RSACipher.java:391)
	... 13 more
JSSTEST_CASE 25 (SSL Ciphersuite JSS Server with Bypass On and JSSE client): FAILED return value 1

These failures are repeatable and occur in every instance of this tinderbox.

Comment 2

[glen beasley]

This is not a JSS/NSS/NSPR issue. The Test that is failing is JSSE_SSLClient.java that is testing the JAVA's JSSE SSL. 

On Niagara running java 5 the test run successfully.

The issue is with java 6. 
Java 6 configured default provider is SunPKCS11-Solaris.

Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_MECHANISM_INVALID
    at sun.security.pkcs11.wrapper.PKCS11.C_WrapKey(Native Method)
    at sun.security.pkcs11.P11RSACipher.engineWrap(P11RSACipher.java:391)

Pool is installed with JDK 1.6.0_01 which is very old.

I am updating to the latest version of java 6, latest patch cluster, and will then test.

Comment 3

[glen beasley]

after I updated the latest patches, and jdk 1.6 update 13 I got the same error.

Then I got the same stack trace running this simple test from the jdk samples.

http://java.sun.com/javase/6/docs/technotes/guides/security/jsse/samples/sockets/client/SSLSocketClient.java

which works with java5 and doesn't with java6. 

I will leave this bug open to hopefully provider resolution but the problem is with jdk 6 or SunPKCS11-Solaris on niagara, or this actual box has an odd hardware failure..

Comment 4

[Julien Pierre]

Thanks for the investigation, Glen. Please get the Java and/or SCF guys in the loop to get to the bottom of this.

Comment 6

[glen beasley]

one of the fan's in this machine failed. the lab will be replacing. 
the lab tech has also updated the box solaris 10 u6_7b and the issue
is no longer reproducible. I had also logged in to several T2000 machines, with various solaris 10 patch levels, belonging to other teams, and was unable to reproduce the error. 

closing bug as hardware corruption issue.