436499 - Automate PKIX certificate chains testing.

Status: RESOLVED FIXED

(NSS :: Test, enhancement, P1)

Description

[Slavomir Katuscak]

Write testing script to generate certificate chains and verify certificates.

Comment 1

[Slavomir Katuscak]

Attachment: Testing script.

Testing script I use now, it's not yet integrated to testing suite (all.sh).

Comment 2

[Slavomir Katuscak]

Attachment: Example configurations.

Comment 3

[Slavomir Katuscak]

Configuration format for my testing script:

Creating entities (+ signing by other entities):

entity CA1
  type Bridge
  issuer Root1
    policy OID.1.0
    mapping OID.1.0:OID.2.0
    inhibit
  issuer Root2
    policy OID.2.0

entity - starts entity block, entity name as parameter, after block should follow empty line
type - type of entity (as parameter), can be Root, Intermediate, Bridge or EE. type is mandatory for every entity
issuer - one or more issuer blocks (none for Root type), issuer name is used as parameter, following line can contains policies, mapping or inhibit settings
policy - contains extension policy
mapping - policy mapping in format IDP:SDP (issuer domain policy, subject domain policy)
inhibit - inhibit any policy extension

When creating entities it's automatically created certificate DB, for Root type there is generated self-signed certificate, for other types certificate request which is signed by every issuer in block.

Creating cert DB + importing certs:

db All

db - create empty certificate DB, this DB is used as testing DB for following import and verify commands

import Army::C,,
import Bridge:Army::

import - import certificate to testing DB, format of parameter is entity:issuer:trustargs

Verifying certificate chains:

verify User:CA1
  cert CA1:CA2
  trust Root
  policy OID.1.0
  result pass

verify - starts verify block, first cert to verify as parameter (if needed more, you can set them in following cert lines), after block should follow empty line. cert format is entity:issuer
cert - add more certs to verify block, format is entity:issuer
trust - sets trust anchor to verify block (-t in vfychain), format is entity:issuer, or certificate path (if doesn't contain : character)
policy - policy extension used for testing (-o in vfychain)
result - expected result (pass or fail), to be also able to run negative tests

Comment 4

[Slavomir Katuscak]

For better understanding of configuration format look to example configurations in attachment.

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

This RFE needs better definition of the requirements.
In what ways do the tests envisioned in this RFE differ from the existing 
tests?

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

One requirement is (or should be) to use the new shareable cert DBs for this
testing.  

These tests undoubtedly also have specific requirements for various types 
of chains, featuring various types of extensions, necessary to support 
testing of bridge CA environments.  Those requirements should be enumerated 
in this bug.

Comment 7

[Slavomir Katuscak]

Reuqirements:

Write script to generate PKIX certificate chains from configuration file.
Define format of this configuration file.

* Generate certificates - supported features:
- 4 types of certificates - Root CA (self-signed), Intermediate CA (signed by one CA), Bridge CA (signed by more CAs), EE (end entity signed by one CA) 
- certificate policy extension
- policy mappings extension
- inhibit any policy extension

* Create certificate databases

* Import certificates to database - supported features:
- trust flags

* Verify of certificate chains - supported features:
- certificate policy extension
- trust anchor (from file or DB)
- positive/negative tests

Configuration format is defined in comment #3.

Comment 8

[Slavomir Katuscak]

(In reply to comment #6)
> One requirement is (or should be) to use the new shareable cert DBs for this
> testing.  

I plan to integrate testing script to nightly test suite (all.sh). In this case shareable certDB format is transparent feature which can be enabled/disabled outside of this script by setting environment variable (in this case in all.sh) and we don't need to specify it explicitly.

Comment 9

[Slavomir Katuscak]

Attachment: Testing script - to run via all.sh.

Modified to run via all.sh (need to set in TESTS variable, for example TESTS=chains).

In this version I used hardcoded config file name all.cfg (can be changed in future) - one config file can contain more testing scenarios (to have only one config file instead of many). Because of this I defined optional statement scenario to configuration file.

Usage:
scenario ScenarioName

Reason of this is that scenario name (if defined) will be used as prefix of test name, to have unique test names in logs.

Comment 10

[Slavomir Katuscak]

Attachment: New version.

Some improvements:

* Added PKCS#7 packages support
- created for every bridge, containing all bridge certificates

* Added AIA extension
- can be defining for every signed certificate (for every issuer)
- can contain link to certificate (.der) or PKCS#7 package (.p7)

- for .der certificate defined in format: aia entity:issuer 

entity CA2
  type Intermediate
  issuer Bridge
    aia Bridge:Navy

- for .p7 package defined in format: aia entity (without ':' character) 

entity CA1
  type Intermediate
  issuer Bridge
    aia Bridge

- for vfychain tests added optional option fetch, which enables fetching from AIA URL

verify EE2:CA2
  cert CA2:Bridge
  trust Navy:
  fetch
  result pass

- AIA URL needs to be defined (externally) as NSS_AIA_HTTP variable

NSS_AIA_HTTP="http://cindercone.red.iplanet.com/share/builds/mccrel3/security/test"

- if NSS_AIA_HTTP not defined, all vfychain tests containing fetch option are skipped

- path to copy AIA files need to be defined (externally) as NSS_AIA_PATH variable

export NSS_AIA_PATH="/share/builds/mccrel3/security/test"

- if NSS_AIA_PATH not defined, then files are not copied there and AIA tests can fail

* added testdb option to change testing DB
- before there was only createdb which deleted DB and created new one, testdb only set which DB to use and doesn't touch it
- can be set to no value, then no DB is used
- examples:

testdb EE1
testdb

* support for more config files
- list of config files is file scenarios (defined as CHAINS_SCENARIOS variable in function chains_init)
- this file contains names of all config files to process by chains.sh
- directory wher all those files are located should be the same as chains.sh

Steps to run chains tests after applying this patch:

export TESTS=chains (or add to all.sh)
export NSS_AIA_HTTP="http://cindercone.red.iplanet.com/share/builds/mccrel3/security/test"
export NSS_AIA_PATH="/share/builds/mccrel3/security/test"

Copy testing configuration files to mozilla/security/nss/tests/chains directory and create scenarios file there containing all this configurations (ls -1 *.cfg > configurations).

Comment 11

[Slavomir Katuscak]

Attachment: My testing configurations (scenarios).

This zip file contains my testing configurations (Christophe's configurations rewritten to new format). Unzip directly to mozilla/security/nss/tests/chains directory. Christophe, please select which of them we plan to test in our nightly testing, or prepare new in this format. 

When we are done with configurations, we can add chains.sh to all.sh to TESTS variable to run in nightly tests and Tinderboxes.

Comment 12

[Christophe Ravel]

Comments for security/nss/tests/chains/chains.sh

General remarks:
- You are using global variables for most of the functions. This is not very structured and should be avoided when possible. If you really need to use global variables, their usage should be clearly documented:
	* at the beginning of the script: list of global variables, their meaning and typical value
	* in each function: list of global variables used as input and as output.
- When executing a command (like certutil), make sure that the full command line appears in the log before the output of the actual command. You may need to create a function to do this for all commands and avoid to have to duplicate each command: first for the output and then for the execution. See "create_pkcs7" for an example of duplication to avoid and wrong order (execute then display command line).
	
Noise creation (dd if=/dev/urandom of=tests_noise bs=256 count=1)
- this code is duplicated in many function. You should create a function just for this.
- the location of the output is not defined (tests_noise). It should be created within the directory of the specific run for the test (like $test_results/machine.n).
- /dev/urandom may not exist on all the supported OSes. It doesn't exist on Windows for example.

create_cert_req:
- The additional user input (y\n -1\n y\n) is only if use the flag "-2". It should be ignored if you don't use this flag. But that's only because we don't have another flag that requires additional user input. Not a major issue, but not very elegant either.

import_cert:
- I am not sure we still need the option of importing ascii certs. I used the ascii format at the beginning of my experiments, but Nelson gave me a more efficient solution: keep the certs when they are signed by "sign_cert". I think your script uses this latter method.


Comments for test scenarios:
- remove bug425314.cfg, it is the same as bridge.cfg
- remove bug436599.cfg, it is included in bridgewithhalfaia.cfg
- rename the scenario in megabridge.cfg from megabridge to megabridge_3_2 (3 branches per bridge, 2 levels of bridges). We may want to add more of these later.

The order of the scenarios should go from simple to complex:
bridge.cfg
megabridge.cfg
extension.cfg
extension2.cfg
anypolicy.cfg
mapping.cfg
mapping2.cfg
aia.cfg
bridgewithaia.cfg
bridgewithhalfaia.cfg
bridgewithpolicyextensionandmapping.cfg


Also for testing chains, see with Alexei if we can leverage the existing test for libpkix.

Comment 13

[Slavomir Katuscak]

(In reply to comment #12)
> Comments for security/nss/tests/chains/chains.sh
> 
> General remarks:
> - You are using global variables for most of the functions. This is not very
> structured and should be avoided when possible. If you really need to use
> global variables, their usage should be clearly documented:
>     * at the beginning of the script: list of global variables, their meaning
> and typical value
>     * in each function: list of global variables used as input and as output.

This list could be relatively long, passing long list of parameters from one function to other is sometimes less elegant them using global ones. Especially when it's needed to pass it more times. Difference between shell and languages like C is that shell doesn't have local variables, so even if I process some variable inside nested function, it's value is visible also outside - and that is just what we want there.

> - When executing a command (like certutil), make sure that the full command
> line appears in the log before the output of the actual command. You may need
> to create a function to do this for all commands and avoid to have to duplicate
> each command: first for the output and then for the execution. See
> "create_pkcs7" for an example of duplication to avoid and wrong order (execute
> then display command line).

Done manually. Using special function would be only more complicated.

> Noise creation (dd if=/dev/urandom of=tests_noise bs=256 count=1)
> - this code is duplicated in many function. You should create a function just
> for this.
> - the location of the output is not defined (tests_noise). It should be created
> within the directory of the specific run for the test (like
> $test_results/machine.n).
> - /dev/urandom may not exist on all the supported OSes. It doesn't exist on
> Windows for example.

Used only 2 times, replaced by 'date >> ${NOISE_FILE} 2>&1' (NOISE_FILE is defined in common/init.sh).

> create_cert_req:
> - The additional user input (y\n -1\n y\n) is only if use the flag "-2". It
> should be ignored if you don't use this flag. But that's only because we don't
> have another flag that requires additional user input. Not a major issue, but
> not very elegant either.

Done.

 
> import_cert:
> - I am not sure we still need the option of importing ascii certs. I used the
> ascii format at the beginning of my experiments, but Nelson gave me a more
> efficient solution: keep the certs when they are signed by "sign_cert". I think
> your script uses this latter method.

It's good to have it for case of importing external certificates.
 
 
> Comments for test scenarios:
> - remove bug425314.cfg, it is the same as bridge.cfg
> - remove bug436599.cfg, it is included in bridgewithhalfaia.cfg
> - rename the scenario in megabridge.cfg from megabridge to megabridge_3_2 (3
> branches per bridge, 2 levels of bridges). We may want to add more of these
> later.

Done.

> The order of the scenarios should go from simple to complex:
> bridge.cfg
> megabridge.cfg
> extension.cfg
> extension2.cfg
> anypolicy.cfg
> mapping.cfg
> mapping2.cfg
> aia.cfg
> bridgewithaia.cfg
> bridgewithhalfaia.cfg
> bridgewithpolicyextensionandmapping.cfg
 
Done. Added also new scenarios.

> Also for testing chains, see with Alexei if we can leverage the existing test
> for libpkix.

Done.

Comment 14

[Slavomir Katuscak]

Attachment: Patch v4.

Testing script addressing most from comment 12.

Changes:
-chains.sh contains some functionality which was previously in libpkix.sh (libpkix_setup_db, libpkix_leak_test), support for importing existing certs
-removed chains tests from libpkix.sh 
-support for memory leak testing (integrated to memleak.sh)
-reorganized memleak.sh, added option NSS_MEMLEAK_TESTS to set which tests to run
-check for libpkix tests moved from all.sh to libpkix.sh
-new scenarios
-comments about global variables
-levels for any policy

Things to add later:
-all about PKIX variables (BUILD_LIBPKIX_TESTS, NSS_ENABLE_PKIX_VERIFY, PKIX_OBJECT_LEAK_TEST) - current tests works also without them


Steps to run ONLY chains tests:
export NSS_TESTS=chains
./all.sh

To enable ONLY chains memory leak tests:
export NSS_TESTS=memleak
export NSS_MEMLEAK_TESTS=chains
./all.sh

For AIA extension need to set also NSS_AIA_HTTP AND NSS_AIA_PATH (see comment 10).

Comment 15

[Christophe Ravel]

Comment on attachment 342590 [details] [diff] [review]
Patch v4.

Just 1 minor comment:

in chain.sh lines 513 to 519:
You use 'vfychain ${DB_OPT} -pp -vv ${FETCH_OPT} ${POLICY_OPT} ${VFY_CERTS} ${TRUST_OPT}' 3 times. You could have created a variable to avoid duplication.

Otherwise r+=christophe.

Comment 16

[Alexei Volkov]

Comment on attachment 342590 [details] [diff] [review]
Patch v4.

It is impossible to check all the things in the patch without spending days on it. I've only verified that mem leak functionality was properly imported from libpkix.sh and the script has ability to check existing chains. Since Christophe has review the main functionality of the patch, I'm giving it r+.

Comment 17

[Slavomir Katuscak]

Patch v4 checked into CVS.