Closed Bug 400841 Opened 17 years ago Closed 4 months ago

Tstclnt rejects certificate in combination FIPS + PKIX

Categories

(NSS :: Libraries, defect, P5)

Tracking

(Not tracked)

RESOLVED INACTIVE

People

(Reporter: slavomir.katuscak+mozilla, Unassigned)

Details

Attachments

(1 file)

Tstclnt fails when client DB is in FIPS mode and PKIX is enabled:

tstclnt -p 8443 -h aquash.red.iplanet.com -c j -f -d /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/aquash.2/pkix/client_memleak -w nss < /export/tinderbox/SunOS_5.9/mozilla/security/nss/tests/memleak/sslreq.dat
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
tstclnt: write to SSL socket failed: This certificate is not valid.

I'm not sure if this combination is supported, we don't run those test as standard nightly tests, only in memory leak tests. I found this when resolving why memory leak tests takes much more time with PKIX enabled then without. The reason is that we use tstclnt to stop selfserv and when tstclnt fails, selfserv waits until tinderclean script (running to kill hanged Tinderbox processes) kills it. In ssl.sh we don't use tstclnt to shutdown server, we use kill command. 

Alexei, when we discussed which tests to run, you suggested to test only server in bypass against client in bypass. I set this only for ssl.sh but not for memleak.sh (there are needed some more changes). Let me know if this combination is supported.
Summary: Tstclnt fails in combination FIPS + PKIX. → Tstclnt rejects certificate in combination FIPS + PKIX
Nelson, we discussed this workaround before staff meeting. I didn't have time to check if it works, I only prepared patch.
Attachment #285900 - Flags: review?(nelson)
I just noticed another thing, in all strsclnt tests where client DB is in FIPS mode there is this error printed:

/usr/dist/pkgs/sunstudio_i386,v11.0/SUNWspro/prod/bin/dbx /export/tinderbox/SunOS_5.9/mozilla/dist/SunOS5.9_i86pc_DBG.OBJ/bin/strsclnt
memleak.sh: -------- DBX commands:
dbxenv follow_fork_mode parent
dbxenv rtc_mel_at_exit verbose
dbxenv rtc_biu_at_exit verbose
check -memuse -match 16 -frames 16
run -q -p 8443 -d /export/tinderbox/SunOS_5.9/mozilla/tests_results/security/aquash.1/client_memleak -w nss -c 1001 -C c aquash.red.iplanet.com

selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.

This bug is there also in both non-PKIX and PKIX mode.
Original problem described in description occurs only in PKIX mode.
Comment on attachment 285900 [details] [diff] [review]
Workaround (checked in)

r=nelson
This should be an accpetable workaround
Attachment #285900 - Flags: review?(nelson) → review+
Slavo, does your comment 2 mean that this problem is also seen in non-FIPS
tests, and that PKIX is the only condition necessary to see this error?
(In reply to comment #4)
> Slavo, does your comment 2 mean that this problem is also seen in non-FIPS
> tests, and that PKIX is the only condition necessary to see this error?

No.

For tstclnt (as in description) are necessary both FIPS and PKIX.
For strsclnt (as in comment 2) is necessery only FIPS.

Some important notes - why was this detected now and not before:

We don't tests client in FIPS mode in ssl.sh, only server. Why ? Is this not supported, or we just don't have tests for it ? Should we write tests for testing client in FIPS mode ?

Memory leak checking doesn't check return values of NSS tools and if strsclnt fails (comment 2) it's not detected. Test is marked as FAILED only when there is new leak found. It's OK in case that we tests the same tests in other scripts like ssl.sh (it's detected there, we don't need to check the same in memleak.sh), but this is not our case.
Workaround checked in:

Checking in memleak.sh;
/cvsroot/mozilla/security/nss/tests/memleak/memleak.sh,v  <--  memleak.sh
new revision: 1.19; previous revision: 1.18
done
Slavo, Nelson,

Do we know why tstclnt can't verify the certificate ? Is this due to the DB not having the trusted CA ? If so, using -o is acceptable. Otherwise, this warrants further investigation.

Slavo, have you tried to run this same test manually outside dbx to see if it worked ?
Yes, I verified it also manually. Also case from description was tested outside DBX.

bash-3.00$ export NSS_ENABLE_PKIX_VERIFY=1
bash-3.00$ tstclnt -p 8443 -h solaris-devx.czech.sun.com -c j -f -d /export/home/sven/nss/virgintip/mozilla/tests_results/security/solaris-devx.7/pkix/client_memleak -w nss < /export/home/sven/nss/virgintip/mozilla/security/nss/tests/memleak/sslreq.dat
tstclnt: write to SSL socket failed: This certificate is not valid.
bash-3.00$ unset NSS_ENABLE_PKIX_VERIFY
bash-3.00$ tstclnt -p 8443 -h solaris-devx.czech.sun.com -c j -f -d /export/home/sven/nss/virgintip/mozilla/tests_results/security/solaris-devx.7/pkix/client_memleak -w nss < /export/home/sven/nss/virgintip/mozilla/security/nss/tests/memleak/sslreq.dat
subject DN: CN=solaris-devx.czech.sun.com,E=solaris-devx.czech.sun.com@bogus.com,O=BOGUS NSS,L=Mountain View,ST=California,C=US
issuer  DN: CN=NSS Test CA,O=BOGUS NSS,L=Mountain View,ST=California,C=US
0 cache hits; 1 cache misses, 0 cache not reusable
HTTP/1.0 200 OK
Server: Generic Web Server
Date: Tue, 26 Aug 1997 22:10:05 GMT
Content-type: text/plain

GET /stop HTTP/1.0

EOF

Blocks: FIPS2008
No longer blocks: FIPS2008
Attachment #285900 - Attachment description: Workaround. → Workaround (checked in)
Severity: normal → S3

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Assignee: alvolkov.bgs → nobody
Severity: S3 → S4
Status: NEW → RESOLVED
Closed: 4 months ago
Priority: -- → P5
Resolution: --- → INACTIVE
You need to log in before you can comment on or make changes to this bug.