Closed Bug 232894 Opened 20 years ago Closed 16 years ago

Automate the running of NIST PKITS tests in NSS nightly QA

Categories

(NSS :: Test, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bishakhabanerjee, Assigned: slavomir.katuscak+mozilla)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 3 obsolete files)

Run the set of NIST PKITS tests every night, with the rest of the NSS tests.

Patch follows.
Comment on attachment 140431 [details] [diff] [review]
patch to header to set dir for cert and CRL data for NIST tests

I rather doubt this patch would work on Win2k.
Bishakha,

Since we do not distribute the PKITS test data with NSS,
we cannot make all.sh invoke pkits.sh.
Patch that addresses Nelson's latest comments from email:

"1. We do not want to run wget and pull down a fresh copy of the
zip file, and unzip the PKITS data with every tinderbox run. 

We want to allow the user to specify an environment variable stating
where the files are.  If the files are there, then we don't want to
pull down a fresh copy.  If the files aren't there, then we pull down
a copy. "

It would not be a problem to download the data and unzip every time for
Tinderboxes (the time it takes to download and unzip is 3-4 minutes), but it
would store a copy of the data files in every Tinderbox run, and take up space
that we are borrowing from Anthony on the server. Hence, I went ahead and
implemented Nelson's suggestion. Here are the relevant lines of code:


############################## pkits_init ##############################
# local shell function to initialize this script 
########################################################################
pkits_init()
{
  SCRIPTNAME=pkits.sh

  if [ -z "${CLEANUP}" ] ; then
      CLEANUP="${SCRIPTNAME}"
  fi

  if [ -z "${INIT_SOURCED}" -o "${INIT_SOURCED}" != "TRUE" ]; then
      cd ../common
      . ./init.sh
  fi

  PKITSDIR=${HOSTDIR}/pkits

  mkdir -p ${PKITSDIR}

  if [ ! -d "${PKITS_DATA}" ]; then
      PKITS_DATA=${PKITSDIR}/PKITS_data
      mkdir -p ${PKITS_DATA}
      wget http://csrc.nist.gov/pki/testing/PKITS_data.zip \
          -O ${PKITS_DATA}/PKITS_data.zip
      cd ${PKITS_DATA}
      unzip -o PKITS_data
  fi

  certs=${PKITS_DATA}/certs
  crls=${PKITS_DATA}/crls

  if [ ! -d "${certs}" ]; then
      echo "download test data from "
      echo "http://csrc.nist.gov/pki/testing/PKITS_data.zip "
      echo "and save to PKITS_DATA"
      exit 1
  fi

  cd ${PKITSDIR}

  PKITSdb=${PKITSDIR}/PKITSdb

  PKITS_LOG=${PKITSDIR}/pkits.log #getting its own logfile

  if [ ! -d "${PKITSdb}" ]; then
      mkdir -p ${PKITSdb}
  else
      echo "$SCRIPTNAME: WARNING - ${PKITSdb} exists"
  fi

  echo "HOSTDIR" $HOSTDIR
  echo "PKITSDIR" $PKITSDIR
  echo "PKITSdb" $PKITSdb
  echo "PKITS_DATA" $PKITS_DATA
  echo "certs" $certs
  echo "crls" $crls

  echo nss > ${PKITSdb}/pw
  certutil -N -d ${PKITSdb} -f ${PKITSdb}/pw

  certutil -A -n TrustAnchorRootCertificate -t "C,C,C" -i \
      $certs/TrustAnchorRootCertificate.crt -d $PKITSdb
  crlutil -I -i $crls/TrustAnchorRootCRL.crl -d ${PKITSdb}
}

"2. Rather than not running the PKITS tests, it would be preferable
to run the pkits tests, but not allow PKITS failures to become tinderbox
QA failures until we've resolved more of the CRL related issues."

This may entail some amount of work. Before doing this (and I'm not averse to
doing this), I'd like to know the timeframe for the CRL and UTF8 issues to get
resolved. Is it going to be months, or a few weeks?
In any case, this would need to be fixed in a different script, "header". 

Are the changes to pkits.sh okay to check in?

Bishakha, please make patches using cvs diff -u, and attach them as patches
to the bug rather than putting them in the comment.  

Your newest code above contains these 

>  if [ ! -d "${PKITS_DATA}" ]; then 
>      PKITS_DATA=${PKITSDIR}/PKITS_data
>      mkdir -p ${PKITS_DATA}


I think that test is insufficient.  Imagine that on the first run,
it creates the directory, but the wget fails.  Then on the second run,
it will not attempt to redo the wget.

You need to test for more than the presence of the directory.  You also 
need to test for the presence of a  file in it.  I'd suggest you test 
for the presence of the last file to come out of the ZIP file.  
Attached patch patch v3 (obsolete) — Splinter Review
Hi Nelson,

We just discussed this, and I am attaching a patch of what I mentioned made
with a cvs diff -u.

Here, I am first checking to see if the top level dir, PKITS_DATA, is set in
the environment (this check is mostly for our test and Tinderbox systems (and I
am assuming that if taht is set to a networked dir, the underlying dirs would
all be there). 

If I do not find the directory, I go ahead and download the data using wget.
For instance, if you were using a system from home, or someone else wishing to
download all the NSS tests to run them, and who would not have this env
variable set, could also run this suite.

Now, if your machine does not have wget, I do a furthur check later on to check
for the cert and crl base directories, and exit if I do not find them.

About testing for the last file in the zipped data file, as opposed to testing
for the top-level directory, as I just mentioned to you, the number of default
tries to connect in "wget" is 25. I can always up the number of tries, but I am
fairly comfortable (even in the case of patchy network connections) that 25
tries would be more than fine to download one zipped datafile. Besides, for our
automated test and Tinderbox cases, we wouldn't even go that route, since we
have data on a network drive set as an environment variable.
Attachment #140431 - Attachment is obsolete: true
Attachment #142375 - Flags: review?(MisterSSL)
Comment on attachment 142375 [details] [diff] [review]
patch v3

other problems with this patch.  

1. If the user sets PKITS_DATA in his environment, but the directory does not
exist, 
  this patch will ignore the user's value of PKITS_DATA and create its own.  

2. If the directory exists, but the contents are incomplate, this script will
simply
fail in each subsequent run, without attempting to fill it again.

I believe that what you want to do is this (in pseudo code here):

  if (PKITS_DATA not defined in environment) {
     define PKITS_DATA to be ${PKITSDIR}/PKITS_data
  }
  if ($PKITS_DATA does not exist as a directroy) {
     create it;
  }
  if ($PKITS_DATA does not contain the certs and crls directories) {
     wget the zip file
     unzip it
  }
  if ($PKITS_DATA still does not contain the certs and crls directories) {
     complain and exit
  }
Attachment #142375 - Flags: review?(MisterSSL) → review-
Assignee: bishakhabanerjee → jason.m.reid
QA Contact: bishakhabanerjee → jason.m.reid
Assignee: jason.m.reid → nobody
QA Contact: jason.m.reid → test
Assignee: nobody → richard.freedman
OS: Windows 2000 → All
Priority: -- → P2
Hardware: PC → All
Target Milestone: --- → 3.12
tests/libpkix/common/libpkix_init_nist.sh checks for two environment
variables. If NIST_FILES_DIR is defined, tests using the NIST files will
be run, and will expect to find the files in that directory. If it is
NULL or not defined, those tests will be skipped.

If NIST_FILES_DIR is defined, there is an additional check for whether
PDVAL is defined. If both variables are defined, then a couple of additional
tests are run. Those tests use URIs found in AIA extensions to try to find
additional certs from an LDAP server, for path validation. Again, those tests
are skipped if either NIST_FILES_DIR or PDVAL is undefined.

When LIBPKIX's HttpCertStore is ready, more tests will be added under PDVAL,
with URIs pointing to HTTP sources as well as LDAP sources.

I am ready to check in a third set of tests, to be executed or skipped
according to whether OCSP is defined. These tests will access an OCSP server
(whose URI comes from an AIA extension). But the necessary files include
some certs whose creation (because of the AIAs) is beyond the present
capability of certutil, and the necessary server has to be running on
pride.red.iplanet.com. Obviously this is a stopgap solution, but is worth
including to capture the test for LIBPKIX's OCSP checking.

Who will have responsibility for coordinating the environment values of
NIST_FILES_DIR, PDVAL, and OCSP for the libpkix tests with the existence
of the necessary files and servers?
Assignee: richard.freedman → slavomir.katuscak
I'm going to prepare new patch for this bug, but first I want to write my idea how it should look:

1. if PKITS_DATA variable defined and data OK then goto 7
2. if PKITS_DATA variable defined and data not OK then complain (warning) and continue
3. define (or redefine) PKITS_DATA variable to ${PKITSDIR}/PKITS_data
4. mkdir PKITS_DATA
5. wget zip file and unzip it
6. if data not OK then complain (warning) and exit (skip PKITS tests)
7. PKITS testing

My idea of checking if data are OK is to list directory recursively and count lines (`ls -1R | wc -l` -eq 1573).

Nelson, please send me feedback, if this is OK with you, or some suggestions what to improve.
Blocks: 396601
Blocks: 419425
No longer blocks: 419425
Depends on: 419425
Slavo,  It sounds to me like your proposal will make this test script 
dependent on a test program that is NOT (or may not be) a part of the 
standard set of test programs available on every supported platform.  
I am referring to wget.  It is undesirable to increase the number of 
tools (test programs not built as part of NSS) required on every platform
in order to test NSS.  

The creation of a pkits data directory should be viewed as part of the test 
machine setup, just as installing the compilers, shells, etc. are also part 
of the machine setup.  Your script should merely determine if the data is 
available (at the directory named in some shell variable) and run PKITS if 
it is present, and do not run PKITS if the data is absent.
Attached patch Patch v4Splinter Review
Simplified to 2 steps:
1. Check if PKITS_DATA defined, if not then print message and skip PKITS testing.
2. Check if directory with PKITS_DATA exists, if not then print message and skip PKITS testing.

No error messages in case that PKITS_DATA is not defined or directory doesn't exists, only message in logfile.
Attachment #140430 - Attachment is obsolete: true
Attachment #142375 - Attachment is obsolete: true
Attachment #310201 - Flags: review?(nelson)
Attachment #142375 - Attachment description: patch → patch v3
Attachment #310201 - Attachment description: Patch. → Patch v4
Comment on attachment 310201 [details] [diff] [review]
Patch v4

Please make these very minor changes, then you can commit.

>-tests="cipher perf libpkix cert dbtests tools fips sdr crmf smime ssl ocsp merge"
>+tests="cipher perf libpkix cert dbtests tools fips sdr crmf smime ssl pkits ocsp merge"

Make pkits last in the list.


>+  if [ ! -d "${PKITS_DATA}" ]; then
>+      echo "${SCRIPTNAME}: PKITS data directory ${PKITS_DATA} doesn't exists, skipping."

change "exists" to "exist".
Attachment #310201 - Flags: review?(nelson) → review+
Checking in all.sh;
/cvsroot/mozilla/security/nss/tests/all.sh,v  <--  all.sh
new revision: 1.47; previous revision: 1.46
done
Checking in pkits/pkits.sh;
/cvsroot/mozilla/security/nss/tests/pkits/pkits.sh,v  <--  pkits.sh
new revision: 1.21; previous revision: 1.20
done
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: