Open Bug 430054 Opened 17 years ago Updated 2 years ago

Testing scripts should be resistant to IOPR server problems.

Categories

(NSS :: Test, defect, P2)

Product:
NSS
Component:
Test
Type:
defect

Tracking

(Not tracked)

Milestone:
3.12.4

People

(Reporter: slavomir.katuscak+mozilla, Unassigned)

Details

Since last friday there are many Tinderboxes in orange state. Problem is in server for IOPR testing (dochinups.red.iplanet.com) which doesn't respond and our our scripts are trying to download and source config file from it. Linux: Installing certs for dochinups.red.iplanet.com:443: /iopr cert.sh: Initializing DB at /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com -------------------------- certutil -N -d /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com -f ../tests.pw.4922 cert.sh: #541: Initializing DB at /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com - PASSED cert.sh: Loading root cert module to Cert DB at /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com -------------------------- modutil -add RootCerts -libfile /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/dist/Linux2.6_x86_glibc_PTH_DBG.OBJ/lib/libnssckbi.so -dbdir /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com WARNING: Performing this operation while the browser is running could cause corruption of your security databases. If the browser is currently running, you should exit browser before continuing this operation. Type 'q <enter>' to abort, or <enter> to continue: Module "RootCerts" added to database. cert.sh: #542: Loading root cert module to Cert DB at /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com - PASSED Bad server certificate: -8172, Peer's certificate issuer has been marked as not trusted by the user. subject DN: CN=dochinups.red.iplanet.com,E=dochinups.red.iplanet.com-rsa@bogus.com,O=BOGUS NSS,L=Mountain View,ST=California,C=US issuer DN: CN=NSS IOPR Test CA 3684,O=BOGUS NSS,L=Mountain View,ST=California,C=US 0 cache hits; 1 cache misses, 0 cache not reusable 0 stateless resumes /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com/iopr_server.cfg: line 1: syntax error near unexpected token `newline' /export/tinderbox/Linux_2.6.9-42.ELsmp/mozilla/tests_results/security/dopushups.1/CA_iopr_dochinups.red.iplanet.com/iopr_server.cfg: line 1: `<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">' cert.sh: #543: Fail to source config file(ws: dochinups.red.iplanet.com) - FAILED Download (tstclnt) passed even if should fail, sourcing failed and is reported. Solaris: Installing certs for dochinups.red.iplanet.com:443: /iopr cert.sh: Initializing DB at /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/ciotat.1/CA_iopr_dochinups.red.iplanet.com -------------------------- certutil -N -d /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/ciotat.1/CA_iopr_dochinups.red.iplanet.com -f ../tests.pw.3193 cert.sh: #541: Initializing DB at /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/ciotat.1/CA_iopr_dochinups.red.iplanet.com - PASSED cert.sh: Loading root cert module to Cert DB at /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/ciotat.1/CA_iopr_dochinups.red.iplanet.com -------------------------- modutil -add RootCerts -libfile /export/tinderbox/SunOS_5.9/mozilla/dist/SunOS5.9_DBG.OBJ/lib/libnssckbi.so -dbdir /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/ciotat.1/CA_iopr_dochinups.red.iplanet.com WARNING: Performing this operation while the browser is running could cause corruption of your security databases. If the browser is currently running, you should exit browser before continuing this operation. Type 'q <enter>' to abort, or <enter> to continue: Module "RootCerts" added to database. cert.sh: #542: Loading root cert module to Cert DB at /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/ciotat.1/CA_iopr_dochinups.red.iplanet.com - PASSED Bad server certificate: -8172, Peer's certificate issuer has been marked as not trusted by the user. subject DN: CN=dochinups.red.iplanet.com,E=dochinups.red.iplanet.com-rsa@bogus.com,O=BOGUS NSS,L=Mountain View,ST=California,C=US issuer DN: CN=NSS IOPR Test CA 3684,O=BOGUS NSS,L=Mountain View,ST=California,C=US 0 cache hits; 1 cache misses, 0 cache not reusable 0 stateless resumes ./all.sh: syntax error at line 2: `newline or ;' unexpected Download (tstclnt) passed even if should fail, sourcing passed even if should fail (this can be caused by using default /bin/sh which is not POSIX shell, just now trying to run on bash). We have there 2 cases of this failure: 1. Linux case - all Tinderbox runs are orange. 2. Solaris case - only Upgrade DB runs are orange - it's caused because cert_iopr.sh is sourced from cert.sh and cert.sh is sourced from dbupgrade.sh. When sourcing in cert_iopr.sh fails, dbupgrade.sh is not finished and databases are not upgraded, but variable for DB type is already switched. This causes failures when using certificates in following tests. Three things to fix there: 1. Tstclnt should return non-zero value when file downloading fails. 2. Sourcing failure should be detected (maybe POSIX shell is the solution). 3. Dbupgrade.sh should continue even if loading IOPR config fails.
I do not agree that we want Tinderbox to be green when a major group of tests are not being run because a test server has failed. Can we have yet another color in the Tinderbox columns that means: "Not all the tests were performed, but those that were performed all passed." ?? If so, then I suggest we adopt another color (blue?) to mean that, and change the scripts to show that color under those conditions.
Nelson, I don't expect all tests to pass there. I want to have reported failure in downloading data from IOPR server (that will make Tinderbox orange) - because this is the problem and it's not reported as failure now. And I want to prevent situation when dbupgrade scripts doesn't finish (script exits when trying to source empty file), databases are not upgraded and SSL tests using cert DBs fails. In this case we don't need new color (I'm not sure if this is possible without modifications of Tinderbox code on mozilla side).
Changing severity from critical to normal, dochinups is working now and tests are not failing.
Severity: critical → normal
Priority: -- → P2
Target Milestone: 3.12.1 → 3.12.4

The bug assignee is inactive on Bugzilla, and this bug has priority 'P2'.
:beurdouche, could you have a look please?

For more information, please visit auto_nag documentation.

Assignee: slavomir.katuscak+mozilla → nobody
Flags: needinfo?(bbeurdouche)
Severity: normal → S3

We have modified the bot to only consider P1 as high priority, so I'm cancelling the needinfo here.

Flags: needinfo?(bbeurdouche)
You need to log in before you can comment on or make changes to this bug.