319501 - libpkix test scripts need to be enhanced

Status: RESOLVED FIXED

(NSS :: Test, defect, P1)

Description

[Julien Pierre]

Currently, libpkix has dozens of QA test that run and are invoked by all.sh, but they are summarized in the following categories in results.hmtl :

LIBPKIX Tests
Test Case	Result
RUNNING tests in pkix_pl_test 	Passed
RUNNING tests in pkix_test 	Passed
RUNNING performance tests in sample_apps 	Passed
ALL LIBPKIX TESTS COMPLETED SUCCESSFULLY 	Passed

There is no way to immediately tell which test fails. The tests scripts should be enhanced to show more details.

Comment 1

[Julien Pierre]

We are getting a lot of test failures on the PKIX branch now. But all we see in the report is something like "27 LIBPKIX tests failed". This is not good enough. I'm raising the priority of this bug to P1. We cannot ship NSS 3.12 without this being fixed. Richard, can you please work on this ? This has priority over the HTTP cert store.

In addition, I think everything that is in libpkix.log should be piped to output.log as well.

Comment 2

[Richard N. Freedman]

The failures have been determined to be failures in the CRL checking, but more investigation is still needed to fix the problem. Meanwhile, libpkix test scripts have been enhanced to show in results.html details of tests, including how many pass (out of how many) in each category.

The changes are all in mozilla/security/nss/tests/libpkix:
Checking in libpkix.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/Attic/libpkix.sh,v  <--  libpkix.sh
new revision: 1.1.2.9; previous revision: 1.1.2.8
done
Checking in common/libpkix_init.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/common/Attic/libpkix_init.sh,v  <--  libpkix_init.sh
new revision: 1.1.2.13; previous revision: 1.1.2.12
done
Checking in pkix_pl_tests/runPLTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_pl_tests/Attic/runPLTests.sh,v  <--  runPLTests.sh
new revision: 1.1.2.9; previous revision: 1.1.2.8
done
Checking in pkix_pl_tests/module/runPLTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_pl_tests/module/Attic/runPLTests.sh,v  <--  runPLTests.sh
new revision: 1.1.2.17; previous revision: 1.1.2.16
done
Checking in pkix_pl_tests/pki/runPLTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_pl_tests/pki/Attic/runPLTests.sh,v  <--  runPLTests.sh
new revision: 1.1.2.15; previous revision: 1.1.2.14
done
Checking in pkix_pl_tests/system/runPLTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_pl_tests/system/Attic/runPLTests.sh,v  <--  runPLTests.sh
new revision: 1.1.2.9; previous revision: 1.1.2.8
done
Checking in pkix_tests/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.6; previous revision: 1.1.2.5
done
Checking in pkix_tests/certsel/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/certsel/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.10; previous revision: 1.1.2.9
done
Checking in pkix_tests/checker/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/checker/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.7; previous revision: 1.1.2.6
done
Checking in pkix_tests/crlsel/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/crlsel/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.8; previous revision: 1.1.2.7
done
Checking in pkix_tests/params/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/params/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.12; previous revision: 1.1.2.11
done
Checking in pkix_tests/results/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/results/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.10; previous revision: 1.1.2.9
done
Checking in pkix_tests/store/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/store/Attic/runTests.sh,v  <--  runTests.shnew revision: 1.1.2.9; previous revision: 1.1.2.8
done
Checking in pkix_tests/top/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/top/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.24; previous revision: 1.1.2.23
done
Checking in pkix_tests/util/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/util/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.9; previous revision: 1.1.2.8
done
Checking in sample_apps/runPerf.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/sample_apps/Attic/runPerf.sh,v  <--  runPerf.sh
new revision: 1.1.2.13; previous revision: 1.1.2.12
done

Comment 3

[Richard N. Freedman]

Additional changes:

mozilla/security/nss/tests/libpkix/libpkix.sh
revision 1.1.2.10
date: 2006/03/29 21:59:37;  author: richard.freedman%sun.com;  state: Exp;  lines: +4 -2
Use an extra line to echo a blank line; echo does not interpret backslash-n
properly on all platforms.
****************************
mozilla/security/nss/tests/libpkix/common/libpkix_init.sh
revision 1.1.2.16
date: 2006/04/06 20:14:51;  author: richard.freedman%sun.com;  state: Exp;  lines: +2 -2
Once again include all output in log
----------------------------
revision 1.1.2.15
date: 2006/03/29 22:03:17;  author: richard.freedman%sun.com;  state: Exp;  lines: +8 -6
Use a shortened form of output for the HTML lines. With tracing on, use
shortened output in output log as well, for non-error lines.
----------------------------
revision 1.1.2.14
date: 2006/03/27 23:46:49;  author: richard.freedman%sun.com;  state: Exp;  lines: +3 -0
Fix calculations; output more detail information for pkix tests
****************************
mozilla/security/nss/tests/libpkix/pkix_tests/runTests.sh
revision 1.1.2.7
date: 2006/03/29 22:04:34;  author: richard.freedman%sun.com;  state: Exp;  lines: +1 -4
Some shell variables were not enclosed in braces. This may have been the
cause of some incorrect calculations.
****************************
mozilla/security/nss/tests/libpkix/pkix_tests/top/runTests.sh
revision 1.1.2.25
date: 2006/03/27 23:46:52;  author: richard.freedman%sun.com;  state: Exp;  lines: +7 -1
Fix calculations; output more detail information for pkix tests

Comment 4

[Richard N. Freedman]

The long pathnames are typically the result of prepending ${curdir} to a relative path. I have removed ${curdir}/ from a number of command lines, and the tests
run happily with their relative paths. The resulting output table is no longer the full width of a window.

Another set of changes:

Checking in libpkix/pkix_pl_tests/module/runPLTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_pl_tests/module/Attic/runPLTests.sh,v  <--  runPLTests.sh
new revision: 1.1.2.18; previous revision: 1.1.2.17
done
Checking in libpkix/pkix_pl_tests/pki/runPLTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_pl_tests/pki/Attic/runPLTests.sh,v  <--  runPLTests.sh
new revision: 1.1.2.16; previous revision: 1.1.2.15
done
Checking in libpkix/pkix_tests/certsel/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/certsel/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.11; previous revision: 1.1.2.10
done
Checking in libpkix/pkix_tests/crlsel/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/crlsel/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.9; previous revision: 1.1.2.8
done
Checking in libpkix/pkix_tests/params/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/params/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.13; previous revision: 1.1.2.12
done
Checking in libpkix/pkix_tests/results/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/results/Attic/runTests.sh,v  <--  runTests.sh
new revision: 1.1.2.11; previous revision: 1.1.2.10
done
Checking in libpkix/pkix_tests/top/runTests.sh;
/cvsroot/mozilla/security/nss/tests/libpkix/pkix_tests/top/Attic/runTests.sh,v <--  runTests.sh
new revision: 1.1.2.28; previous revision: 1.1.2.27
done

Comment 5

[Alexei Volkov]

integrated into the truck