315445 - Temp directory for AMD and Intel are the same, hence over writing each others files causing the tests to fail

Status: RESOLVED WONTFIX

(JSS Graveyard :: Tests, defect, P2)

Description

[Sandeep Konchady]

============= Setup DB
main: jss library loaded
org.mozilla.jss.crypto.AlreadyInitializedException
	at org.mozilla.jss.pkcs11.PK11Token.initPassword(PK11Token.java:258)
	at org.mozilla.jss.tests.SetupDBs.main(SetupDBs.java:58)
SetupDBs returned 1
JSSTEST_CASE 1 (Setup DB): FAIL

The issue is that we run the tests on 2 Solaris 10 x86 machines (mandela is Intel, Biarritz is AMD64).  The temporary directory and the result file are named based on the OS name (SunOS5.10_i86pc_OPT.OBJ for example). I have disabled the tests on mandela to fix this temporarily.

I guess that the fix would be to add the machine name to the temporary directory and result file names (example: mandela-SunOS5.10_i86pc_OPT.OBJ). See $testdir in all.pl.

Comment 1

[Julien Pierre]

Sandeep,

On Solaris 64-bit AMD64, the target directory is called SunOS5.10_i86pc_64_DBG.OBJ, instead of SunOS5.10_i86pc_OPT.OBJ for 32-bit x86 . At least that's the case for NSS with coreconf . JSS should follow the same model.
If it does not, then there is a bug .

Comment 2

[Christophe Ravel]

The issue is when testing the 32 bit version. The name is the same on Intel and AMD 64.
For the 64 bit version, there is no conflict since we don't have any 64 bit Intel machine.

We could have the same problem if we want to test on different Sparc processors.

This is why I am proposing to add the machine name to the objdir directory name for the tests only (the temporary directory for the certificate database and the result file name).

Comment 3

[Christophe Ravel]

Actually, the result files already have the hostname added at the begining:
biarritz-SunOS5.10_i86pc_DBG
biarritz-SunOS5.10_i86pc_OPT
mandela-SunOS5.10_i86pc_DBG
mandela-SunOS5.10_i86pc_OPT

We should do the same for the temporary directory that holds the certificate database.

Comment 4

[Julien Pierre]

Christophe,

Both when building and testing, the scripts are using the value of the USE_64 environment variable to set the proper OBJDIR directory name that contains _64_ or not . That's the way it's done in the NSS tests scripts, which uses coreconf, and the JSS test scripts should do the same. There is no need to append the machine name. That won't actually solve the problem on platforms that have multiple architectures, eg. if you build x86 and AMD64 on the same box, and then test them both on the same box.

Comment 5

[Christophe Ravel]

I guess that we are not talking about the same thing.

The issue is not with the USE_64 flag (that we use).
An AMD64 machine is can also run 32 bit binaries. And this is what we want to test.

When JSS TESTS are RUN, a certificate database is created.
If you start a JSS test (32 bit optimized) on an AMD64 machine, this directory name is currently SunOS5.10_i86pc_OPT.OBJ.
If you start a JSS test (32 bit optimized) on an Intel machine, this directory name is currently SunOS5.10_i86pc_OPT.OBJ.

So both tests are creating a certificate database in the same directory: SunOS5.10_i86pc_OPT.OBJ. If the 2 machines are running the tests at the same time, we run into access conflicts.


Note: If you start a JSS test (64 bit optimized) on an AMD64 machine, this directory name is currently SunOS5.10_i86pc_64_OPT.OBJ.

This is also easy to understand with the test results file. We want to have the tests results from the JSS 32 bit optimized run on AMD64 to be in a different file than the JSS 32 bit optimized run on Intel.
The JSS tests are already doing this by adding the machine name before the OBJDIR_NAME for the test results file.

examples:
biarritz-SunOS5.10_i86pc_DBG
biarritz-SunOS5.10_i86pc_OPT
mandela-SunOS5.10_i86pc_DBG
mandela-SunOS5.10_i86pc_OPT

Comment 6

[Julien Pierre]

Christophe,

Each time you run the NSS all.sh, a new directory is created under tests_results, of the form /export/home/nss/pkix/mozilla/tests_results/security/HOSTNAME.X . I don't see the OBJDIR being used anywhere.
Given these separate directories, there can never be any collisions. Can't the JSS test script follow the same model ?

It would help if you would give the fully-qualified pathname of the directory and files that are being written to multiple times.

Comment 7

[Christophe Ravel]

The jss test temporary files are in (example):
$ /share/builds/mccrel3/security/securitytip/builds/20051107.1/wozzeck_Solaris8/mozilla/security/jss/org/mozilla/jss/tests/SunOS5.10_i86pc_OPT.OBJ> ls
./              cert8.db        keystore.pfx    secmod.db
../             key3.db         libnssckbi.so*

We could use hostname.x. But our choice was to use the OBJDIR_NAME (see https://bugzilla.mozilla.org/show_bug.cgi?id=297573) because it is more informative about the test is (OS name and version, debug/optimize, 32/64 bit).
To fix the current issue, the proposition is to use hostname-OBJDIR_NAME as we already do for the test results. 

I'll let you choose the best option.

Comment 8

[Julien Pierre]

Christophe,

I looked at bug 297573 . I think the fix that was checked in is incomplete. It allows JSS tests to run in parallel only if the 2 different instances are on 2 different platforms. Multiple instances of the tests cannot run in parallel on the same platform, whether on the same machine or on a different machine, because the tests are writing to $(OBJDIR) under the source directories. IMO, only the build process should be allowed to write to the sources/$(OBJDIR) location, not the test execution . Further, if you want to rerun the tests, you currently need to run a make rule to clobber jss/tests, which erases all the test files from the previous run.

The quickest fix is probably to write test output to $(OBJDIR)/HOSTNAME.X directories . I'm guessing this will probably take the least effort to change . Look at the existing scripts in nss/tests to figure out HOSTNAME.X .

Comment 9

[Christophe Ravel]

Agreed. We need a better fix this time. Otherwise we will run again into troubles.

Comment 10

[Sandeep Konchady]

NSS has setup a good structure to allow tests to run in parallel on the same system for different configurations.  We should reuse the same structure for JSS too.  The benefit to this being
[1] Test results can be located in a single location for both NSS/JSS
[2] No reinventing the wheel

This seems logical to me and I have the following proposal

[1] Current NSS test results reside at mozilla/tests_results/security/ . JSS test results could reside in the same location, but within a subdirectory jss (mozilla/tests_results/security/jss).
[2] Individual test results reside in directories that are versioned incrementally as with NSS
[3] These individual test results are located in directories that are OS/Platform/Configuration specific. (EG: SunOS5.9_DBG.OBJ_scooter.1, SunOS5.9_64_DBG.OBJ_scooter.2)
[4] The result directory will contain only runtime specific objects (cert8.db      ,keystore.pfx,output.log,secmod.db,key3.db,libnssckbi.so, and passwords).
[5] The test files themselves (all.sh, init.sh, Makefile and perhaps cleanup.sh?) will reside in the same location as that of NSS (mozilla/security/jss/tests)
[6] Since we are not moving the tests themselves, these are backward compatible and one can run the JSS tests as they used to before.

Please review this proposal for running JSS tests in parallel on the same system and give your feedback.

Comment 11

[Christophe Ravel]

Attachment: Create db in mozilla/tests_results/jss/<hostname>.<version>

This patch implements the same behaviour as for NSS.
The database used for each test run is created in mozilla/tests_results/jss/<hostname>.<version>

<version> is a sequential number. The last number used is stored in mozilla/tests_results/jss/<hostname>

This fix should avoid having 2 hosts using (removing and creating) the same database from the same directory (see discussion above).

Comment 12

[glen beasley]

Comment on attachment 221758 [details] [diff] [review]
Create db in mozilla/tests_results/jss/<hostname>.<version>

looks good. I like the check for the old database, and error message.

Comment 13

[Christophe Ravel]

Committed on JSS trunk.

/cvsroot/mozilla/security/jss/org/mozilla/jss/tests/all.pl,v  <--  all.pl
new revision: 1.34; previous revision: 1.33

Marked target 4.2.3. But it should be 4.2.4 (yet to be created). Glen, could you create a 4.2.4 target ?

Comment 14

[Christophe Ravel]

Verified.
Mark as fixed in 4.2.4.

Comment 15

[Slavomir Katuscak]

This bug still appears in nightly testing.

Comment 16

[Christophe Ravel]

I confirm that the bug is fixed in JSS 4.2.4

Comment 27

[Slavomir Katuscak]

Reopening, because it seems to be the same problem as bug 337157 which is marked as duplicate of this bug.

============= Start Mozilla-JSS Secret Key Gen and Ciphers
/share/builds/components/jdk/1.5.0_09/Linux_amd64/jre/bin/java -d64 org.mozilla.jss.tests.JCASymKeyGen /share/builds/mccrel3/security/securityjes5/builds/20070520.1/redhat21as_RHEL21/mozilla/tests_results/jss/nssamdrhel3.4
SunJCE: SunJCE Provider (implements RSA, DES, Triple DES, AES, Blowfish, ARCFOUR, RC2, PBE, Diffie-Hellman, HMAC)
Mozilla-JSS: Provides Signature, Message Digesting, and RNG
Enter password for NSS FIPS 140-2 User Private Key
Enter password for NSS FIPS 140-2 User Private Key
Exception in thread "main" java.lang.ExceptionInInitializerError
	at javax.crypto.KeyGenerator.getInstance(DashoA12275)
	at org.mozilla.jss.tests.JCASymKeyGen.genSecretKey(JCASymKeyGen.java:102)
	at org.mozilla.jss.tests.JCASymKeyGen.main(JCASymKeyGen.java:323)
Caused by: java.lang.SecurityException: Cannot set up certs for trusted CAs
	at javax.crypto.SunJCE_b.<clinit>(DashoA12275)
	... 3 more
Caused by: java.lang.SecurityException: Signature classes have been tampered with
	at javax.crypto.SunJCE_b.d(DashoA12275)
	at javax.crypto.SunJCE_b.c(DashoA12275)
	at javax.crypto.SunJCE_q.run(DashoA12275)
	at java.security.AccessController.doPrivileged(Native Method)
	... 4 more
JCASymKeyGen returned 1
JSSTEST_CASE 22 (JCASymKeyGen successful): FAIL

Occured in nightly testing on nssamdrhel3 Linux2.4_x86_64_glibc_PTH_64_OPT securityjes5 20070520.1.

Comment 28

[BMO Automation]

JSS development has moved from the Mozilla community to the Dogtag PKI community. Please re-file this bug at https://github.com/dogtagpki/jss if it is still relevant. Thank you!