265784 - SSL interoperability regression testing

Status: RESOLVED FIXED

(NSS :: Test, enhancement, P2)

Description

[Julien Pierre]

Recently, a patch was made that broke SSL in a way that was undetectable by
all.sh . The patch affected hashing with SSL3 cipher suites.

When all.sh runs, it tests an SSL client and server using the current NSS build.
Because the library was broken, both SSL client and server sides computed the
hash incorrectly. They both incorrectly computed it exactly the same way, and
thus were able to communicate in the broken NSS build.

The problem was only caught by manual testing, when I tried to connect to the
broken SSL server with a released Mozilla browser, and found that they didn't
interoperate in SSL3 mode.

It is very conceivable that this kind of problem could slip through a release,
especially as the bug only affected certain cipher suites.

We need to improve our testing.

a) we should use some previous, known-good, release of NSS , in the SSL test, to
make sure that the newest NSS release can still talk SSL against the old NSS
releases.

b) we may also want to extend the interoperability tests to non-NSS products.

These tasks may be done either through extending all.sh or through another
script altogether, but either way these tests need to be optional. It must
remain possible for a developer to test NSS against itself, without older
releases or other products. The interoperability tests could be run in either
the tinderbox or nightly QA.

Comment 1

[Julien Pierre]

Jason, would you mind taking a look at this problem ?

Comment 2

[glen beasley]

Note: Sandeep added JSS/JDK SSL tests to the JSS test suite.
http://lxr.mozilla.org/security/source/security/jss/org/mozilla/jss/tests/
The test has separate JSS client iterating through all common SSL ciphers 
talking to a separte JSSE server, and a JSSE client iterating through
all common ciphers to a JSS server. 
https://bugzilla.mozilla.org/show_bug.cgi?id=283383

Comment 3

[Jason Reid]

I reopened this bug after wrongly marking it resolved.

Comment 4

[Alexei Volkov]

Attachment: NSS interoperability tests suite with Apache+mod_ssl

See information document(config.txt) that is a part of the patch and added as a
separate attachment

Comment 5

[Alexei Volkov]

Attachment: Interoperability test suite info and apache server configuration document

Comment 6

[Alexei Volkov]

Attachment: NSS interoperability tests suite with Apache+mod_ssl

Comment 7

[Julien Pierre]

Reassigning to Alexei, since he has taken ownership of these new tests.

Comment 8

[Wan-Teh Chang]

Another good interoperability test is to use a Windows SSL client
that uses the same SSL library as Internet Explorer against
NSS's SSL server-side code (selfserv).

If OpenSSL has a SSL server test program (the equivalent of our
selfserv), it is better to use that than a full web server.

Comment 9

[Alexei Volkov]

Adding to the bug the summary of interoperability test suite design that Julien, Nelson and myself have been discussing on the meeting last Tuesday.

I've added numbers in front of every paragraph so it would be easy to write comments for.

---------------------------
Interoperability Test Suite

#1
Want to run interoperability tests that corresponds to the following
picture:


                       | 
      NSS build        |             Preconfigured Web servers 
   hosts               |                with enabled SSL
                       |
                       |           ------------------------
                    run tests     |                        |
     ------        against each   |        ---------       |
    | NSS  |    server in the box |       | Linux   |      |
    |tests |     <<<======>>>     |       |Apache   |      |
     ------            |          |        --------        |
                       |          |                        |
                    run tests     |                        |
     ------        against each   |        ---------       |
    | NSS  |    server in the box |       |Windows  |      |
    |tests |     <<<======>>>     |       | IIS     |      |
     ------            |          |        ---------       |
                       |          |                        |
                       |          |       -----------      |
                       |          |      |Platform X |     |
                       |          |      |Server   X |     |
                       |          |       -----------      |
                                  |                        |
                                   ------------------------


TEST CONFIGURATION OBJECTIVES:
#2
  Test NSS SSL coverage, authentication and crl functionalities with
a minimum required configuration on the NSS test side. 


WEB SERVER CONFIGURATION REQUIREMENTS:
#3
Web server X should be manually preconfigured. The following steps are
necessarily to run NSS tests in automated mode:

     * Create CA cert for the test system on a machine.
     * Create server cert for this web server.
     * Create two user certs that will be used for coverage,
     authentication and crl testing from client side.
     * Generate CRL which will have one of the created user certs revoked
     * Configure web SSL server with created server cert
     * Make all certs and crl files available for download
     * Install cgi-bin script which will allow reverse testing and results
     filtering
     * create a web page tstclnt will try to access, contents of1 which will be   
     used as an input for output filter that will determine client server SSL 
     communication success or failure.



NSS TEST SETUP REQUIREMENTS:
#4
Set an environment variable that will have space separated list of 
hosts(probably with ports, to allow to use non-default ones) that have installed web servers on it.



TEST RUN DETAILS:
#5
We can converge the picture and review testing against single server, since
all three test suites will be run against each server in direct and reverse
modes and each of the servers will have all needed for testing certs available
for download.

#6
Interoperability testing is consider to be turned on when user sets variable
with predefined name which is set to have a host name of web server.

#7
If interoperability testing is on, NSS client will download needed certs and crls
from server. 

#8
First download(downloading unrevoked client cert) is done by "tstclnt" command
without specifying a cert. Web server should be configured to allow such access
to cert files repository by either using "SSLVerifyClient=optional"(for apache
and any other servers where this option is available) or by starting additional
copy of web server (or additional zone) which will allow SSL connection without
a cert. 

#9
IMHO: this set up seems to me too complicated to archive ability to make just
one connection to to get a user cert. Instead I would just download a user cert
without using ssl by utilizing wget(in this case we will add one more command
into our set of required tools for NSS testing) or modify tstclnt to make
connection without using ssl(Nelson oppose to this approach).

-- Julien Commented --
I think making the change to tstclnt not to support SSL is fine if wget
doesn't fit the bill. Unless you need to submit a POST request, wget
should work .
---------
I've checked latest wget version. It looks like there is new option 
starting from 1.9.x which can be used for POST(called --post-data).
Anyway, does not look like POST will be needed. So, I'll use wget for it.

#10
All others certs will be downloaded through ssl connection and installed into dbs
located in server_<server name> and client_<server name> directories. If any of
certs missing, a failure will be declared and interoperability mode will be tuned off
for this web server.


   * Direct Test Run:
#11
Test run against web server using tstclnt with the following test suites:
#12
Cipher Coverage Suite:
All cipher suites from sslcov.txt file will be checked against a webserver
SSL engine.
#13
Authentication Suite:
Each protocol(ssl2/ssl3/tls) will be tested with and without cert/proper db pwd.
#14
CRL Suite:
One test that checks revocation of a second user cert on a web server side.

-- Julien Commented --
I would recommend we make the CRL test optional. We don't know to what
extent each server supports CRLs. Some may require most recent CRLs.
Perhaps there could be a config file on the web server that would be
downloaded by the NSS QA and would indicate whether to run CRL testing
with the revoked cert.
---------
For this I this we can use the same script to rase a flag that direct CRL tests are
disabled.

#15
Test client exit status in combination with webserver output parsing
will be used for checking the success/failure of a communication. A web server
have to have a webpage that the client will try to access. The page have to have
predefined string that the client script will grep for to determine the status
of a test. This additional check is needed for web server like IIS since these
servers do not break communication upon SSL handshake failure, but issue an error
page instead.

-- Julien Commented --
I suggest you include a file on the webserver that contains the expected
page for a success case.
This would be downloaded by the NSS QA scripts prior to running tests.
After that the filtering could be done based on the HTTP response
against that "gold" response file. If the result is anything else, or
there is an SSL error, then you know there is a failure.
-------------------------
I agree. This is what I wanted to express in paragraph that is before your last
comment.



   * Reverse Test Run:
#16
NOTE: In the next context word "webserver" is reserved for remote, manually
installed webserver(one of Apache, IIS, Oracle, etc). The list does not include NSS
selfserv.

#17
Test run against selfserv executed by submitting requests to webserver cgi-script
using tstclnt(cgi-bin capability should be enabled on each webserver). CGI script will
invoke a SSL client(call it R-client) on a webserver side that will connect testing
selfserv.

#18
The script should have the following capabilities:
        * properly construct R-client command line from
        script parameters.
        * run R-client and collect an output
        * interpret R-client output and respond with common
        for all web servers result.
        * report cipher list that is supported by R-client.

Cipher Coverage Suite:
The list of possible ciphers will be obtained from webserver by making request to
cgi script, before testing. Then all cipher suites will be tested against selfserv.

#19
Authentication Suite:
Each protocol(ssl2/ssl3/tls) will be tested with and without cert/proper db pwd
against selfserv started with 4 -r's parameters(request and require cert
on second handshake).

#20
CRL Suite:
One test that checks revocation of a second user cert on a web server side.

#21
The simple filters will be used to identify the status of a test, since a cgi 
script will take care of all platform specific differences in R-client outputs.

Comment 10

[Julien Pierre]

Comment on attachment 199867 [details] [diff] [review]
NSS interoperability tests suite with Apache+mod_ssl 

Removing review requests on obsolete patch.

Comment 11

[Julien Pierre]

Per our meeting today, lowering to P2.

Comment 12

[Alexei Volkov]

Attachment: latest version of test scripts

untar and apply nss_3.10_iopr.patch

Comment 13

[Alexei Volkov]

Attachment: working patch. Tests apache only.

This patch for nss - apache/openSSL interoperability(iopr) test only. Windows perl script still has a bug, which makes a test run hang.

To run the tests add IOPR_HOSTADDR_LIST=nsssvr.red.iplanet.com to your environment.

Patch includes server(apache or AIS) and client(nss) side scripts. Server scripts are located at mozilla/security/nss/tests/iopr/server_scr directory.

New files:
* cert_iopr.sh - loads iopr client config and certs to nss test side and generates additional user certs.
* ssl_iopr.sh  - performs ssl testing against servers listed in IOPT_HOSTADDR_LIST variable.
* server_scr - directory what has server configuration files and server side cgi script

Patch for tstclnt.c makes tstclnt print out transferred server responds without http header.

sslcov.txt changes description of the ciphers. This is used in revers testing, when selfserv is tested by openssl/wtstclnt.

Comment 14

[Alexei Volkov]

A revised description of interoperability tests. 


> Interoperability Test Suite
> 
> #1
> Want to run interoperability tests that corresponds to the following
> picture:
> 
> 
>                        | 
>       NSS build        |             Preconfigured Web servers 
>    hosts               |                with enabled SSL
>                        |
>                        |           ------------------------
>                     run tests     |                        |
>      ------        against each   |        ---------       |
>     | NSS  |    server in the box |       | Linux   |      |
>     |tests |     <<<======>>>     |       |Apache   |      |
>      ------            |          |        --------        |
>                        |          |                        |
>                     run tests     |                        |
>      ------        against each   |        ---------       |
>     | NSS  |    server in the box |       |Windows  |      |
>     |tests |     <<<======>>>     |       | IIS     |      |
>      ------            |          |        ---------       |
>                        |          |                        |
>                        |          |       -----------      |
>                        |          |      |Platform X |     |
>                        |          |      |Server   X |     |
>                        |          |       -----------      |
>                                   |                        |
>                                    ------------------------
> 
> 
> TEST CONFIGURATION OBJECTIVES:
> #2
>   Test NSS SSL coverage, authentication and crl functionalities with
> a minimum required configuration on the NSS test side. 
> 
> 
> WEB SERVER CONFIGURATION REQUIREMENTS:
> #3
> Web server X should be manually preconfigured. The following manual steps are
> necessarily to run NSS tests in automated mode:
> 
>      * Create CA cert for the test system on a machine.
>      * Create server cert for this web server.
>      * Create two user certs that will be used for coverage,
>      authentication and crl testing from client side.
>      * Generate CRL which will have one of the created user certs revoked
>      * Configure web SSL server with created server cert
>      * Make all certs and crl files available for download
>      * Install cgi-bin script which will allow reverse testing and results
>      filtering
>      * create a web page tstclnt will try to access, contents of1 which will be 
>      used as an input for output filter that will determine client server SSL 
>      communication success or failure.
> 
> 
> 
> NSS TEST SETUP REQUIREMENTS:
> #4
> Set an environment variable that will have space separated list of 
> hosts(probably with ports, to allow to use non-default ones) that have
> installed web servers on it.
> 
> 
> 
> TEST RUN DETAILS:
> #5
> We can converge the picture and review testing against single server, since
> all three test suites will be run against each server in direct and reverse
> modes and each of the servers will have all needed for testing certs available
> for download.
> 
> #6
> Interoperability testing is considered to be turned on when user sets variable
> with predefined name which is set to have a host name of web server.  
> 
> #7
> If interoperability testing is on, NSS client will download needed certs and
> crls from server. 
> 
#8
NSS tests will configure itself based on configuration of
webserver. For this test configuration file is downloaded from
webserver which describes what suites are available on a particular
webserver. File is called iopr_server.cfg and it should be available
from /iopr directory.

#9
Here is a example of iopr_server.cfg file that describes two
testing directories, one without client auth and one with it 

--------Beginning of config file ----------------------------------
# directory where p12 cert files are resides
certDir=/iopr
# CA cert file name
caCertName=TestCA

# CRL file name
caCrlName=TestCA

# List of user certs available for download and testing
userCertNames="TestUser510 TestUser511"

# List of revoked certs from list above
userRevokedCertNames="TestUser510"

# Reverse testing script
reverseRunCGIScript="/cgi-bin/client.cgi"

# Test directory name list
supportedTests="SslSingleHs SslSecondHs"

# Each directory name from list above can have four parameters
# listed below:

# SslSingleHs: ssl single handshake with out client cert auth
SslSingleHsDescr="ssl with single handshake without client cert auth"
SslSingleHsPort=443
SslSingleHsUrl=/iopr_test/test_pg.html
# NOAUTH, NOCOV, NOCLR (set in <dirname>Params) can be used to
# disable corresponding test suites.
SslSingleHsParam=NOAUTH

#
# SslSecondHs: ssl with secondary hs when accessing directory 
# that requires cert verification
SslSecondHsDescr="ssl with secondary hs when accessing directory that requires ce
rt verification"
SslSecondHsPort=443
SslSecondHsUrl=/iopr_test_2hs/test_pg.html
SslSecondHsParam=NOCOV

---------End of config file -------------------------------------
 

#10
config file and certs are downloaded using tstclnt with server auth
tuned off.
Here is the list of certs and crl user during a test run from NSS test side:

caCert - downloaded from a webserver
testCert1 - downloaded from a webserver
testCert2 - downloaded from a webserver
crl - downloaded from a webserver

serverCert - selfserv cert. generated after receiving ca cert
             from webserver.

Total is four static and one dynamic parameters.

>    * Direct Test Run:
> #11
> Test run against web server using tstclnt with the following test suites:
> #12
> Cipher Coverage Suite:
> All cipher suites from sslcov.txt file will be checked against a webserver
> SSL engine.
> #13
> Authentication Suite:
> Each protocol(ssl2/ssl3/tls) will be tested with and without cert/proper db
> pwd.
> #14
> CRL Suite:
> One test that checks revocation of a second user cert on a web server side.
> 
> For this I this we can use the same script to rase a flag that direct CRL tests
> are
> disabled.
> 
> #15
> Test client exit status in combination with webserver output parsing
> will be used for checking the success/failure of a communication. A web server
> have to have a webpage that the client will try to access. The page have to
> have
> predefined string that the client script will grep for to determine the status
> of a test. This additional check is needed for web server like IIS since these
> servers do not break communication upon SSL handshake failure, but issue an
> error
> page instead.
> 
>    * Reverse Test Run:
> #16
> NOTE: In the next context word "webserver" is reserved for remote, manually
> installed webserver(one of Apache, IIS, etc). The list does not include
> NSS
> selfserv.
> 
> #17
> Test run against selfserv executed by submitting requests to webserver
> cgi-script
> using tstclnt(cgi-bin capability should be enabled on each webserver). CGI
> script will
> invoke a SSL client(call it R-client) on a webserver side that will connect
> testing
> selfserv.
> 
> #18
> The script should have the following capabilities:
>         * properly construct R-client command line from
>         script parameters.
>         * run R-client and collect an output
>         * interpret R-client output and respond with common
>         for all web servers result.
>         * report cipher list that is supported by R-client.
>  
Cipher Coverage Suite:
All ciphers listed in sslcov.txt will be tested agains selfserv. 
Negative test results will not be reported if failure occured due to
attempt to run a cipher suite unsupported by R-client.

> 
> #19
> Authentication Suite:
> Each protocol(ssl2/ssl3/tls) will be tested with and without cert/proper db pwd
> against selfserv started with 4 -r's parameters(request and require cert
> on second handshake).
> 
> #20
> CRL Suite:
> One test that checks revocation of a second user cert on a web server side.
> 
> #21
> The simple filters will be used to identify the status of a test, since a cgi 
> script will take care of all platform specific differences in R-client outputs.
> 

Comment 15

[Alexei Volkov]

Comment on attachment 238536 [details] [diff] [review]
working patch. Tests apache only.

will split this patch for three patches: changes to existing files, new client side files, new server side files and attach them separetely

Comment 16

[Alexei Volkov]

Attachment: changes to the existent files

Comment 17

[Alexei Volkov]

Attachment: new files invoked by nss tests

with function comments.

Comment 18

[Alexei Volkov]

Attachment: server side scripts

Comment 19

[Alexei Volkov]

Attachment: changes to the existent files

Comment 20

[Alexei Volkov]

Attachment: new files invoked by nss tests

Comment 21

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 242105 [details] [diff] [review]
server side scripts

r=nelson, to go into nss/tests/iopr/<something>/

Comment 22

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 242564 [details] [diff] [review]
changes to the existent files

r=nelson for trunk

Comment 23

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 242569 [details] [diff] [review]
new files invoked by nss tests

r=nelson for trunk

Comment 24

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 242105 [details] [diff] [review]
server side scripts

Be sure to add the MPL license boilerplate to these files before checkin

Comment 25

[Alexei Volkov]

Comment on attachment 199868 [details]
Interoperability test suite info and apache server configuration document

This text has correct configuration info up to the point where says "* At this point we are still not ready to start server". Will update with a new version of the document

Comment 26

[Alexei Volkov]

Attachment: Integrated patch

Comment 27

[Alexei Volkov]

Number of things need to be done to make the rfe completed:
    * setup windows box to run interoperability tests with IIS
    * add ECC and DH/DHE interoperability

Maitenance: periodical openssl library update on existent interoperability web servers. 

Comment 28

[Alexei Volkov]

reassign to Slavo to compete this rfe.

Comment 29

[Alexei Volkov]

Attachment: SSL and OCSP interoperability patch for 3.11 branch (checked in)

Comment 30

[Alexei Volkov]

Attachment: SSL and OCSP interoperability patch for 3.11 branch

Comment 31

[Alexei Volkov]

Comment on attachment 266485 [details] [diff] [review]
SSL and OCSP interoperability patch for 3.11 branch

accidentally attached the patch twice. Obsoleting this one.

Comment 32

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 266483 [details] [diff] [review]
SSL and OCSP interoperability patch for 3.11 branch (checked in)

requesting second review for branch

Comment 33

[Alexei Volkov]

Checking attchment 266483 into 3.11 branch:

/cvsroot/mozilla/security/nss/cmd/ocspclnt/ocspclnt.c,v  <--  ocspclnt.c
new revision: 1.7.28.1; previous revision: 1.7
/cvsroot/mozilla/security/nss/cmd/tstclnt/tstclnt.c,v  <--  tstclnt.c
new revision: 1.42.2.4; previous revision: 1.42.2.3
/cvsroot/mozilla/security/nss/tests/all.sh,v  <--  all.sh
new revision: 1.19.2.4; previous revision: 1.19.2.3
/cvsroot/mozilla/security/nss/tests/cert/cert.sh,v  <--  cert.sh
new revision: 1.28.2.9; previous revision: 1.28.2.8
/cvsroot/mozilla/security/nss/tests/common/init.sh,v  <--  init.sh
new revision: 1.40.24.8; previous revision: 1.40.24.7
/cvsroot/mozilla/security/nss/tests/iopr/cert_iopr.sh,v  <--  cert_iopr.sh
new revision: 1.2.2.1; previous revision: 1.2
/cvsroot/mozilla/security/nss/tests/iopr/ocsp_iopr.sh,v  <--  ocsp_iopr.sh
new revision: 1.3.2.1; previous revision: 1.3
/cvsroot/mozilla/security/nss/tests/iopr/ssl_iopr.sh,v  <--  ssl_iopr.sh
new revision: 1.2.2.1; previous revision: 1.2
/cvsroot/mozilla/security/nss/tests/ocsp/ocsp.sh,v  <--  ocsp.sh
new revision: 1.1.2.1; previous revision: 1.1
/cvsroot/mozilla/security/nss/tests/ssl/ssl.sh,v  <--  ssl.sh
new revision: 1.61.2.12; previous revision: 1.61.2.11
/cvsroot/mozilla/security/nss/tests/ssl/sslcov.txt,v  <--  sslcov.txt
new revision: 1.5.144.4; previous revision: 1.5.144.3

Comment 34

[Slavomir Katuscak]

Things to do in comment 27.

Decreasing priority to P3 (based on priorities set on meeting with Nelson in
September).

Comment 35

[Nelson Bolyard (seldom reads bugmail)]

Readjusting priority back to P2 to be consistent with the priority definitions
used in NSS.  
A fix for this bug/RFE is desired by one of the companies that sponsor NSS.  
By the present definition of NSS priorities, that makes it at least a P2.  

Comment 36

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 266483 [details] [diff] [review]
SSL and OCSP interoperability patch for 3.11 branch (checked in)

The last reviewed patch for this bug was committed.  
Can someone summarize why this bug is still open?
What is needed to close it?

Comment 37

[Nelson Bolyard (seldom reads bugmail)]

Since the question in comment 36 has gone unanswered for 6 months, I'm 
Marking resolved/fixed.  If someone thinks this bug is not fixed, he can 
reopen it and tell us why.

Comment 38

[Slavomir Katuscak]

Reopening, there are still some things to be done (comment #27).

Comment 39

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, and this bug has priority 'P2'.
:beurdouche, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 40

[John Schanck [:jschanck]]

We have other interop tests now.