108250 - OCSP support of NSS not functional with AOL internal CA

Status: RESOLVED INVALID

(NSS :: Libraries, defect, P2)

Description

[Julien Pierre]

When enabling client auth and OCSP support in a debug build of iWS 6 using NSS
3.3, the server crashes when trying to verify the client cert. The stack is :
t@22 (l@1) terminated by signal BUS (Bus Error)
Current function is CERT_LockDB
dbx: can't find file 
"/share/builds/mccrel/nss/nsstip/builds/20010712.1/y2sun2_Solaris8/mozilla/secur
ity/nss/lib/certdb/certdb.c" -- see `help finding-files'
(dbx) current thread: t@22
=>[1] CERT_LockDB(handle = 0xdadadada), line 2326 in "certdb.c"
  [2] DestroyCertificate(cert = 0x9f9118, lockdb = 1), line 6193 in "pcertdb.c"
  [3] CERT_DestroyCertificate(cert = 0x9f9118), line 6233 in "pcertdb.c"
  [4] ssl3_HandleCertificate(ss = 0x9ef040, b = 0x9f3022 "^P", length = 0), line 
6535 in "ssl3con.c"
  [5] ssl3_HandleHandshakeMessage(ss = 0x9ef040, b = 0x9f292c "", length = 
1782U), line 6998 in "ssl3con.c"
  [6] ssl3_HandleHandshake(ss = 0x9ef040, origBuf = 0x9eff54), line 7114 in 
"ssl3con.c"
  [7] ssl3_HandleRecord(ss = 0x9ef040, cText = 0xfd541794, databuf = 0x9eff54), 
line 7379 in "ssl3con.c"
  [8] ssl3_GatherCompleteHandshake(ss = 0x9ef040, flags = 0), line 204 in 
"ssl3gthr.c"
  [9] ssl_GatherRecord1stHandshake(ss = 0x9ef040), line 1302 in "sslcon.c"
  [10] ssl_Do1stHandshake(ss = 0x9ef040), line 156 in "sslsecur.c"
  [11] ssl_SecureRecv(ss = 0x9ef040, buf = 0x75cea0 
"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\
n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" ..., len = 
8191, flags = 0), line 1013 in "sslsecur.c"
  [12] ssl_Recv(fd = 0x879fd0, buf = 0x75cea0, len = 8191, flags = 0, timeout = 
3000000U), line 1171 in "sslsock.c"
  [13] PR_Recv(fd = 0x879fd0, buf = 0x75cea0, amount = 8191, flags = 0, timeout 
= 3000000U), line 215 in "priometh.c"
  [14] DaemonSession::GetConnection(this = 0x8199a0), line 400 in 
"daemonsession.cpp"
  [15] DaemonSession::run(this = 0x8199a0), line 462 in "daemonsession.cpp"
  [16] Thread::run_(this = 0x8199a0), line 234 in "Thread.cpp"
  [17] ThreadMain(thisObject = 0x8199a0), line 226 in "Thread.cpp"
  [18] _pt_root(arg = 0x81bed0), line 214 in "ptthread.c"
(dbx)

Comment 1

[Julien Pierre]

This crash is not reproduced when using NSS 3.4 tip debug binaries, only with
the 3.3 debug binaries built at iPlanet.
With 3.4 the client cert is simply recognized as "expired". I'm doing my own
builds of 3.3 / 3.3.1 to figure out if the problem occurs on them too.

Comment 2

[Julien Pierre]

I tried with various versions of 3.2 and 3.3, including my own builds. They all
crashed the same way. Only the 3.4 tip build does not reproduce the crash.
However, the client still gets an "unknown SSL error -12269" while the server
reports SEC_ERROR_EXPIRED_CERTIFICATE.

Comment 3

[Stephane Saux]

That's because PSM doesn't handle error -12269, which is
SSL_ERROR_EXPIRED_CERT_ALERT

So the mechanism here is that the server side ocsp check found the cert to be
expired (isn't that determined by looking that the dates on the cert rather than
doing an OCSP check?  Has the cert you're trying to validate actually expired?)

The browser doesn't handle SSL_ERROR_EXPIRED_CERT_ALERT.  It does handle
SEC_ERROR_EXPIRED_CERTIFICATE, but that's when the server cert has expired. 
I'll file a bug against the browser.

Comment 4

[Stephane Saux]

Filed PSM bug 108626

Comment 5

[Julien Pierre]

Can't test this on 3.4 right now because server hangs with it. We are looking
for resolution of this bug in 3.4 .

Comment 6

[Julien Pierre]

I checked this with the latest NSS tip.
The server now crashes when coming up trying to enable OCSP.

Current function is PR_Assert
dbx: can't find file
"/h/blds-sca15a/export/builds/mccrel/nspr/v4.1.2/builds/20010621.1/sunos56/mozilla/nsprpub/pr/src/io/prlog.c"
-- see `help finding-files'
(dbx) 
current thread: t@1
  [1] __sigprocmask(0x0, 0xffbed680, 0x0, 0x0, 0x0, 0x0), at 0xfe8b9bf0
  [2] _resetsig(0xfe8bc510, 0x0, 0x0, 0x23318, 0xfe8ce000, 0x0), at 0xfe8ae620
  [3] _sigon(0x23318, 0xfe8d5990, 0x6, 0xffbed754, 0x23318, 0xffbed798), at
0xfe8add10
  [4] _thrp_kill(0x0, 0x1, 0x6, 0xfe8ce000, 0x1, 0xfe83a428), at 0xfe8b0e84
  [5] raise(0x6, 0x0, 0x0, 0xffffffff, 0xfe83a394, 0xfeae8f24), at 0xfe7c9b08
  [6] abort(0xfe836000, 0x27, 0xfe83d99c, 0xfe839c78, 0x943, 0x87251), at 0xfe7b5124
=>[7] PR_Assert(s = 0xfe625f40 "0", file = 0xfe625f44 "certdb.c", ln = 2371),
line 495 in "prlog.c"
  [8] CERT_SetStatusConfig(handle = 0x77d68, statusConfig = 0x87210), line 2371
in "certdb.c"
  [9] ocsp_InitStatusChecking(handle = 0x77d68), line 3676 in "ocsp.c"
  [10] CERT_EnableOCSPChecking(handle = 0x77d68), line 3713 in "ocsp.c"
  [11] InitNSSMain(), line 674 in "servnss.cpp"
  [12] WebServer::Run(), line 607 in "WebServer.cpp"
  [13] main(argc = 3, argv = 0xffbeec1c), line 77 in "main.cpp"
(dbx) 

Comment 7

[Robert Relyea]

This fix was checked in Thursday or Friday.
mozilla/security/nss/lib/certdb/certdb.c rev 1.22.
Make sure you have the latest checked out, and that you aren't linking with old
.o's (The file no longer has the assert you described).

bob

Comment 8

[Julien Pierre]

Bob, my NSS build was from thursday. I just did another one and the crash on
startup is gone.
OCSP still doesn't work right though, but I get different behavior now.
After marking the AOL CA as trusted in the web server, and presenting my
personal cert, the cert gets refused by the server, and the following is logged
in the server's log :

[14/Jan/2002:16:01:38] failure ( 3458): Error receiving connection
(SEC_ERROR_INADEQUATE_KEY_USAGE - Certificate key usage inadequate for attempted
operation.)
However, it does not crash the server, so I'm lowering the priority of this bug
to P2.

Comment 9

[Wan-Teh Chang]

Changed the QA contact to Bishakha.

Comment 10

[Wan-Teh Chang]

Set target milestone to NSS 3.5.

Comment 11

[Wan-Teh Chang]

Moved to NSS 3.6.

Comment 12

[Julien Pierre]

I tried it again with the latest OCSP fixes and the failure points to this 
stack.

=>[1] CERT_VerifyCert(handle = 0xa3ba0, cert = 0x299920, checkSig = 1, certUsage 
= certUsageStatusResponder, t = 1023336458000000LL, wincx = (nil), log = (nil)), 
line 1053 in "certvfy.c"
  [2] ocsp_CheckSignature(signature = 0x2ad90c, tbs = 0x2ad948, encodeTemplate = 
0xfe6e3310, handle = 0xa3ba0, certUsage = certUsageStatusResponder, checkTime = 
1023336458000000LL, lookupByName = 1, certIndex = 0x277ebc, pwArg = (nil), 
pSignerCert = 0xfcec12a4, issuer = 0x299920), line 2423 in "ocsp.c"
  [3] CERT_VerifyOCSPResponseSignature(response = 0x2abe60, handle = 0xa3ba0, 
pwArg = (nil), pSignerCert = 0xfcec12a4, issuer = 0x299920), line 2568 in 
"ocsp.c"
  [4] CERT_CheckOCSPStatus(handle = 0xa3ba0, cert = 0x2e05b8, time = 
1023336458626291LL, pwArg = (nil)), line 3343 in "ocsp.c"
  [5] CERT_VerifyCert(handle = 0xa3ba0, cert = 0x2e05b8, checkSig = 1, certUsage 
= certUsageSSLClient, t = 1023336458626291LL, wincx = (nil), log = (nil)), line 
1154 in "certvfy.c"
  [6] CERT_VerifyCertNow(handle = 0xa3ba0, cert = 0x2e05b8, checkSig = 1, 
certUsage = certUsageSSLClient, wincx = (nil)), line 1179 in "certvfy.c"
  [7] SSL_AuthCertificate(arg = 0xa3ba0, fd = 0x4dcee8, checkSig = 1, isServer = 
1), line 254 in "sslauth.c"
  [8] ssl3_HandleCertificate(ss = 0x240050, b = 0x493c24 "^P", length = 0), line 
6550 in "ssl3con.c"
  [9] ssl3_HandleHandshakeMessage(ss = 0x240050, b = 0x4934b4 "", length = 
1904U), line 7145 in "ssl3con.c"
  [10] ssl3_HandleHandshake(ss = 0x240050, origBuf = 0x2401f8), line 7261 in 
"ssl3con.c"
  [11] ssl3_HandleRecord(ss = 0x240050, cText = 0xfcec1798, databuf = 0x2401f8), 
line 7526 in "ssl3con.c"
  [12] ssl3_GatherCompleteHandshake(ss = 0x240050, flags = 0), line 203 in 
"ssl3gthr.c"
  [13] ssl_GatherRecord1stHandshake(ss = 0x240050), line 1256 in "sslcon.c"
  [14] ssl_Do1stHandshake(ss = 0x240050), line 145 in "sslsecur.c"
  [15] ssl_SecureRecv(ss = 0x240050, buf = 0x6639c0 
"\xfdc\x89@\xfdc\x89@\xfdc\x89@\xfdc\x89@\xfdc\x89@", len = 8191, flags = 0), 
line 968 in "sslsecur.c"
  [16] ssl_Recv(fd = 0x4dcee8, buf = 0x6639c0, len = 8191, flags = 0, timeout = 
3000000U), line 1199 in "sslsock.c"
  [17] PR_Recv(fd = 0x4dcee8, buf = 0x6639c0, amount = 8191, flags = 0, timeout 
= 3000000U), line 215 in "priometh.c"
  [18] DaemonSession::GetConnection(this = 0x327ed8), line 417 in 
"daemonsession.cpp"
  [19] DaemonSession::run(this = 0x327ed8), line 479 in "daemonsession.cpp"
  [20] Thread::run_(this = 0x327ed8), line 234 in "Thread.cpp"
  [21] ThreadMain(thisObject = 0x327ed8), line 226 in "Thread.cpp"
  [22] _pt_root(arg = 0x2c9398), line 214 in "ptthread.c"

This happens when trying to verify the signature of the OCSP response.
The client certificate I was using when logging to the web server was my own AOL 
corporate cert. The server had the AOL CA installed and trusted.
I think the problem may be with the AOL CA cert not having the correct bit for 
"responder". I traced the code deep down and the requiredKeyusage was 128, but 
the keyUsage on the cert was 6, which resulted in the inadequate key usage 
error. Bill, any idea on this ?

FYI, I also tried to trace the OCSP code with PSM on the same client cert. It 
went through the same OCSP code path as the web server, and in fact the OCSP 
validation of the cert failed too within PSM, with the invalid key usage error. 
What was less obvious is the fact that the certificate was just not marked 
verified for no uses in PSM, instead of having an error because of the OCSP 
verification failure.

I'm getting more convinced that this is a configuration problem with our root CA 
rather than with the NSS code .

Comment 13

[bill]

I double-checked the CMS configuration and verified that CMS is supposed to be
using the CA's signing cert to sign OCSP responses.  I also viewed the CA
signing cert and verified that the "CRLSign" extension is set.

If we are SURE that the ocsp signing cert is incorrect we could always get a new
CRLsign cert but I'm not totally convinced yet.

    Certificate: 
        Data: 
            Version:  v3
            Serial Number: 0x20001E6
            Signature Algorithm: SHA1withRSA - 1.2.840.113549.1.1.5
            Issuer: <snip>
            Validity: 
                <snip>
            Subject: CN=Intranet Certificate Authority,OU=AOL
Technologies,O=America Online Inc,L=Mountain View,ST=CA,C=US
            Subject Public Key Info: 
                Algorithm: RSA - 1.2.840.113549.1.1.1
                Public Key: 
                    Exponent: 65537
                    Public Key Modulus: (1024 bits) :
                        E2:EF:5F:2C:76:43:89:4B:1A:5F:B3:E5:F8:AA:6F:8B:
                        2F:81:4D:67:FF:BD:4A:0F:63:2E:C4:DC:85:F6:9E:2C:
                        49:26:20:FF:00:17:E4:88:88:69:DE:FD:83:57:E0:A3:
                        11:19:11:AA:D6:DC:BC:EF:B3:D2:15:2E:54:C6:6E:7C:
                        BF:D9:B9:C3:46:D3:09:05:84:E5:53:5C:48:ED:84:85:
                        9A:0E:3B:3D:16:07:5C:F0:B3:79:AB:9A:10:A5:BC:C1:
                        A4:D1:78:4C:06:E5:64:41:FC:05:25:63:26:EB:EF:0C:
                        C7:6E:54:A1:8C:CE:54:57:B6:1F:92:DA:B2:12:4B:8D
            Extensions: 
                Identifier:  CRLDistributionPoints - 2.5.29.31
                    Critical: no 
                    Value: 
                        30:44:30:42:A0:40:A0:3E:86:3C:68:74:74:70:3A:2F:
                        2F:77:77:77:31:2E:75:73:2D:68:6F:73:74:69:6E:67:
                        2E:62:61:6C:74:69:6D:6F:72:65:2E:63:6F:6D:2F:63:
                        67:69:2D:62:69:6E:2F:43:52:4C:2F:47:54:45:52:6F:
                        6F:74:2E:63:67:69
                Identifier: Subject Key Identifier - 2.5.29.14
                    Critical: no 
                    Key Identifier: 
                        29:DB:B2:2D:83:7E:7F:8B:23:BB:C2:CC:66:B9:39:E8:
                        29:F3:02:86
                Identifier:  CertificatePolicies - 2.5.29.32
                    Critical: no 
                    Value: 
                        30:5D:30:46:06:0A:2A:86:48:86:F8:63:01:02:01:05:
                        30:38:30:36:06:08:2B:06:01:05:05:07:02:01:16:2A:
                        68:74:74:70:3A:2F:2F:77:77:77:2E:62:61:6C:74:69:
                        6D:6F:72:65:2E:63:6F:6D:2F:43:50:53:2F:4F:6D:6E:
                        69:52:6F:6F:74:2E:68:74:6D:6C:30:13:06:03:2A:03:
                        04:30:0C:30:0A:06:08:2B:06:01:05:05:07:02:01
                Identifier: Authority Key Identifier - 2.5.29.35
                    Critical: no 
                    Authority Name: <snip>
                    Serial Number: 0x1A3
                Identifier: Private Key Usage: - 2.5.29.16
                    Critical: no 
                    Validity:
                        <snip>
                Identifier: Key Usage: - 2.5.29.15
                    Critical: yes 
                    Key Usage: 
                        Key CertSign 
                        Crl Sign 
                Identifier: Basic Constraints - 2.5.29.19
                    Critical: no 
                    Is CA: yes 
                    Path Length Constraint: 1

Comment 14

[Julien Pierre]

Bill,

Yes, CMS indeed is using this certificate to sign the OCSP response in the OCSP 
responder.
The problem occurs when an OCSP client application (eg. a web server, or a 
browser), making an OCSP request to the responder. It receives the OCSP response 
and sees that it is signed by the AOL Intranet CA cert. But it considers the 
signature invalid because that AOL Intranet CA cert has the incorrect key usage. 
The specific code that fails is in ocsp.c :

 2422       rv = CERT_VerifyCert(handle, signerCert, PR_TRUE, certUsage, 
checkTime,
 2423                            pwArg, NULL);

(dbx) p certUsage
certUsage = certUsageStatusResponder
(dbx) p signerCert->keyUsage
signerCert->keyUsage = 6U
(dbx) p signerCert->nickname
signerCert->nickname = 0x279848 "Intranet Certificate Authority - GTE 
Corporation"
(dbx) p signerCert->subjectKeyID
signerCert->subjectKeyID = {
    type = siBuffer
    data = 0x279790 
")\xdb\xb2-\x83~^?\x8b#\xbb\xc2\xccf\xb99\xe8)\xf3^B\x86\xda\xda\xda\xdaCN=Intra
net Certificate Authority, OU=AOL Technologies, O=America Online Inc, L=Mountain 
View, ST=CA, C=US"
    len  = 20U
}
(dbx) 

To be completely specific and go to the deepest level in the stack, this 
"certUsageStatusResponder" maps to bit 7 of the key usage, or 128. The failure 
occurs inside CERT_VerifyCert, in CERT_CheckKeyUsage, when a binary AND is done 
between the requiredUsage and the cert->keyUsage :

 1152       if ( ( cert->keyUsage & requiredUsage ) != requiredUsage ) {
 1153           return(SECFailure);
 1154       }

(dbx) p cert->keyUsage
cert->keyUsage = 6U
(dbx) p requiredUsage
requiredUsage = 128U
(dbx) 

So I would say there is definitely something wrong with the AOL Intranet CA 
cert. The "status responder" bit needs to be set on it. "Key certsign" and 
"CRLSign" are different bits. Once you fix that and make a new CA cert, OCSP 
should start working with our corporate certs.

Comment 15

[Julien Pierre]

*** Bug 136459 has been marked as a duplicate of this bug. ***

Comment 16

[Julien Pierre]

Bill fixed the key usage in the cert so it works now. This wasn't an NSS bug.
Marking INVALID.