SSL server crashes under stress with client auth

VERIFIED FIXED in 3.4

Status

NSS
Libraries
P1
normal
VERIFIED FIXED
16 years ago
16 years ago

People

(Reporter: Julien Pierre, Assigned: Ian McGreer)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(7 attachments)

(Reporter)

Description

16 years ago
Running NES with 3.4 tip + patch 68245 from Ian in defect 122907.
The server is in multithread mode. The SSL socket requires client auth.
I am pounding on it with a client doing full handshakes with client auth enabled.

The server very quickly dumps core with the following stack :

(dbx) re
current thread: t@50
  [1] pthread_mutex_lock(0x6130255, 0x10000, 0x0, 0xfe8ce000, 0xfe8da46c,
0xfc961d78), at 0xfe8ab6c0
=>[2] PR_Lock(lock = 0x6130255), line 183 in "ptsynch.c"
  [3] nssPKIObject_AddRef(object = 0x728dd0), line 89 in "certdecode.c"
  [4] nssCertificate_AddRef(c = 0x728dd0), line 71 in "certificate.c"
  [5] add_cert_to_cache(td = 0x9b098, cert = 0x728dd0), line 728 in "tdcache.c"
  [6] nssTrustDomain_AddCertsToCache(td = 0x9b098, certs = 0xfcc61068, numCerts
= 1U), line 762 in "tdcache.c"
  [7] NSSTrustDomain_FindBestCertificateBySubject(td = 0x9b098, subject =
0x805518, timeOpt = 0x585bb0, usage = 0xfcc611c0, policiesOpt = (nil)), line 607
in "trustdomain.c"
  [8] NSSCertificate_BuildChain(c = (nil), timeOpt = 0x585bb0, usage =
0xfcc611c0, policiesOpt = (nil), rvOpt = 0xfcc611b4, rvLimit = 2U, arenaOpt =
(nil), statusOpt = 0xfcc611b0), line 363 in "certificate.c"
  [9] CERT_FindCertIssuer(cert = 0x8f6220, validTime = 1013052095630748LL, usage
= certUsageSSLClient), line 420 in "certvfy.c"
  [10] CERT_VerifyCertChain(handle = 0x9b098, cert = 0x8f6220, checkSig = 1,
certUsage = certUsageSSLClient, t = 1013052095630748LL, wincx = (nil), log =
(nil)), line 713 in "certvfy.c"
  [11] CERT_VerifyCert(handle = 0x9b098, cert = 0x8f6220, checkSig = 1,
certUsage = certUsageSSLClient, t = 1013052095630748LL, wincx = (nil), log =
(nil)), line 1149 in "certvfy.c"
  [12] CERT_VerifyCertNow(handle = 0x9b098, cert = 0x8f6220, checkSig = 1,
certUsage = certUsageSSLClient, wincx = (nil)), line 1190 in "certvfy.c"
  [13] SSL_AuthCertificate(arg = 0x9b098, fd = 0x2cac18, checkSig = 1, isServer
= 1), line 270 in "sslauth.c"
  [14] ssl3_HandleCertificate(ss = 0x62a338, b = 0x8f6c96 "^P", length = 0),
line 6561 in "ssl3con.c"
  [15] ssl3_HandleHandshakeMessage(ss = 0x62a338, b = 0x8f6a2c "", length =
618U), line 7157 in "ssl3con.c"
  [16] ssl3_HandleHandshake(ss = 0x62a338, origBuf = 0x62f99c), line 7273 in
"ssl3con.c"
  [17] ssl3_HandleRecord(ss = 0x62a338, cText = 0xfcc61794, databuf = 0x62f99c),
line 7538 in "ssl3con.c"
  [18] ssl3_GatherCompleteHandshake(ss = 0x62a338, flags = 0), line 204 in
"ssl3gthr.c"
  [19] ssl_GatherRecord1stHandshake(ss = 0x62a338), line 1300 in "sslcon.c"
  [20] ssl_Do1stHandshake(ss = 0x62a338), line 156 in "sslsecur.c"
  [21] ssl_SecureRecv(ss = 0x62a338, buf = 0x432c60 "GET / HTTP/1.0^M\n^M\n",
len = 8191, flags = 0), line 1038 in "sslsecur.c"
  [22] ssl_Recv(fd = 0x2cac18, buf = 0x432c60, len = 8191, flags = 0, timeout =
2000000U), line 1191 in "sslsock.c"
  [23] PR_Recv(fd = 0x2cac18, buf = 0x432c60, amount = 8191, flags = 0, timeout
= 2000000U), line 215 in "priometh.c"
  [24] DaemonSession::GetConnection(this = 0x42ebb8), line 400 in
"daemonsession.cpp"
  [25] DaemonSession::run(this = 0x42ebb8), line 462 in "daemonsession.cpp"
  [26] Thread::run_(this = 0x42ebb8), line 234 in "Thread.cpp"
  [27] ThreadMain(thisObject = 0x42ebb8), line 226 in "Thread.cpp"
  [28] _pt_root(arg = 0x432b08), line 214 in "ptthread.c"
(Reporter)

Updated

16 years ago
Priority: -- → P1
Target Milestone: --- → 3.4
(Reporter)

Comment 1

16 years ago
FYI, this is all running OK when using NSS 3.3, so it is a regression.

Comment 2

16 years ago
Julien, could you print *lock in frame 2 and *object in frame 3?
(Assignee)

Comment 3

16 years ago
Taking bug.  I believe I have already located the problem.  I can reproduce this
with strsclnt using multiple threads.
Assignee: wtc → ian.mcgreer

Comment 4

16 years ago
Ian, does it make sense to run your testcase in daily QA? 
(Assignee)

Comment 5

16 years ago
Sonja-

I'd like to find out if we can reproduce Julien's test within our QA, that is,
on the server side.  I found the same problem by running strsclnt against a
selfserv requiring client auth, when I didn't have a client cert.  This wouldn't
be a good test, because I did it with 1000 connections, causing 1000 failures. 
It was actually by accident that I ran it, but I got a crash when I did.
(Assignee)

Comment 6

16 years ago
Created attachment 68373 [details] [diff] [review]
the mother of all patches

This patch attempts to address three separate problems that showed up when
stressing NSS 3.4 in a multithreaded environment:

1)  The function nssTrustDomain_AddCertsToCache attempts to maintain uniqueness
of certificate pointers within the NSS world.  That is, there should be only
one pointer for any cert.  There is logic in the implementation for detecting
that certA is already in the cache when an attempt is made to add certB, where
certA == certB.  This logic depends on being able to convince the caller to
switch to certA.  It was not working for the two callback functions used during
traversal, because these functions only had access to a local copy of their
cert pointer, and were unable to report back to their caller that a different
cert was already in the cache.	This patch provides fixes for both of the
traversal callbacks, relying on the caller to destroy the certB reference,
while passing certA down to further callbacks.	This is OK because the caller
is always retrieve_cert, and it always destroys the cert it passes in.

2) Both the cache and temp store systematically mishandle cert references they
return.  They do a search, come up with a matching list, *release their locks*,
and then create references to the matching certs and return them.  Obviously,
the order is incorrect, references need to be created before the lock is
released.

3) The temp store was not ensuring uniqueness of its cert entries.  It must not
allow the same cert to be added twice.	I had thought nssHash_Add would report
a failure in this case, but it does not, so the entry must be looked for
explicitly.

Julien, could you try this when you get a chance?

And Bob, could you review it?

There is one other issue that concerns me.  Like I noted in item 1, the cache
is able to maintain complete uniqueness of cert pointers.  The temp store
cannot, because of a limitation in the NSSCryptoContext API.  The function
NSSCryptoContext_ImportCertificate takes a local copy of a cert pointer, and
cannot say, "Okay, I imported the cert, but really I already had a different
cert pointer so here's a reference to that one, forget about the one you sent
me."  If necessary, I think we can work around this, by doing the following
sequence:

NSSCryptoContext_ImportCertificate(context, cert);
newCert = NSSCryptoContext_FindCertByEncoding(context, &cert->encoding);
NSSCertificate_Destroy(cert);
cert = newCert;

This would only need to happen in CERT_NewTempCertificate.  In fact, I think it
is probably necessary.	That limitation is something we should address in 4.0,
though.
(Reporter)

Comment 7

16 years ago
Ian,

I updated my NSS tree and applied your patch. The server still dumps core
immediately. The stack is the same.

Wan-Teh, as far as the lock, it appears there is memory corruption going on :

(dbx) frame 2
Current function is PR_Lock
  183       rv = pthread_mutex_lock(&lock->mutex);
(dbx) print *lock
dbx: cannot access address 0xdadadada
(dbx) 
(Reporter)

Comment 8

16 years ago
I have confirmed that the problem only occurs with client auth. When I set the
server not to require client auth, and pound with the client without client
auth, there is no core dump and the server stays up.
Summary: SSL server crashes under stress → SSL server crashes under stress with client auth
(Assignee)

Comment 9

16 years ago
Can you describe exactly what your test does?  I was unable to get selfserv to
crash.  Specifically, what does the server's cert db look like, and what does
the client's cert db look like?
Summary: SSL server crashes under stress with client auth → SSL server crashes under stress
(Assignee)

Comment 10

16 years ago
oops, lost Julien's change to the summary, adding back
Summary: SSL server crashes under stress → SSL server crashes under stress with client auth
(Assignee)

Comment 11

16 years ago
I'm doing:

 selfserv -D -n localhost.nyc.rr.com -p 8443 -w nss -t 100 -d ext_server -r -r -r -r

and

[k@24-164-150-83 localhost.7]$ strsclnt -q -n ExtendedSSLUser -p 8443 -d
ext_client -c 10000 -w nss -t 100 localhost 
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: 0 cache hits; 1 cache misses, 0 cache not reusable
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: 9999 cache hits; 2 cache misses, 0 cache not reusable

without any problem.  That's 100 threads on the server, with 10000 client
connections on 100 client threads, with client auth forced on both handshakes. 
So I'm really not sure how to reproduce this.
(Reporter)

Comment 12

16 years ago
Ian, as far as the test goes :

I have an SSL server socket set to client auth required. The server's database
looks like this :

(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/work/B1/SunOS5.6_DBG.OBJ/alias{79}
certutil -L -d . 
Server-Cert                                                  u,u,u
clientcacert                                                 CT,, 
(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/work/B1/SunOS5.6_DBG.OBJ/alias{80}


(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/work/B1/SunOS5.6_DBG.OBJ/alias{81}
certutil -L -d . -n Server-Cert
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 5 (0x5)
        Signature Algorithm: PKCS #1 MD5 With RSA Encryption
        Issuer: CN=Certificate Shack, O=TestCentral, C=US
        Validity:
            Not Before: Tue Nov 06 20:14:07 2001
            Not After: Mon Feb 06 20:14:07 2006
        Subject: CN=*.mcom.com, O=TestCentral, C=US
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    c9:c1:c2:bd:37:c1:08:bb:c2:be:de:b3:5c:1f:e5:
                    a8:0f:ec:0a:50:8a:2c:7c:2b:39:1b:12:9c:2c:0a:
                    10:bd:e8:09:6b:d2:f6:dd:4e:84:2a:ef:e1:78:31:
                    b8:dc:9e:1c:60:a6:d8:f0:5e:fd:70:df:14:c8:8f:
                    14:88:ad:2f:92:35:bf:e0:e2:d4:48:f6:9f:08:ac:
                    ab:10:c7:86:5d:c5:ee:ab:85:1a:7e:95:fa:e9:0d:
                    3a:28:79:bd:03:94:94:8c:d1:28:3c:88:e0:f0:fc:
                    6d:d8:c5:b0:24:50:1f:b9:a6:c2:97:03:ae:d7:ad:
                    84:aa:f6:c5:78:9d:b2:51
                Exponent: 65537 (0x10001)
        Signed Extensions:
            Name:
                Certificate Type
            Data: <SSL Server>

            Name:
                Certificate Key Usage
            Data:
                03:02:05:20

    Fingerprint (MD5):
        D4:1D:8C:D9:8F:00:B2:04:E9:80:09:98:EC:F8:42:7E
    Fingerprint (SHA1):
        DA:39:A3:EE:5E:6B:4B:0D:32:55:BF:EF:95:60:18:90:AF:D8:07:09

    Signature Algorithm: PKCS #1 MD5 With RSA Encryption
    Signature:
        84:09:1e:ff:7d:a7:3f:20:a8:1b:85:6d:5f:00:c9:9a:86:a0:
        90:69:82:36:60:8b:0c:f1:79:62:b1:6c:3e:80:c2:69:1c:24:
        47:9c:08:78:6d:19:6a:0b:07:6a:f6:e6:56:b7:3e:e3:51:69:
        8e:fe:f4:46:d6:44:03:a4:8b:10:45:8f:db:b6:7f:cc:46:7f:
        66:02:e8:b9:56:7f:44:c4:03:ff:6b:43:1f:fa:4a:59:c1:48:
        76:8c:48:c4:3a:99:6f:82:96:e6:8e:da:67:c5:3c:00:67:68:
        34:db:dc:b2:93:4a:af:e0:08:b8:44:88:74:75:86:62:0b:2b:
        34:af
    Certificate Trust Flags:
        SSL Flags:
            User
        Email Flags:
            User
        Object Signing Flags:
            User

(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/work/B1/SunOS5.6_DBG.OBJ/alias{82}
certutil -L -d . -n clientcacert
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: PKCS #1 MD5 With RSA Encryption
        Issuer: CN=Certificate Shack, O=TestCentral, C=US
        Validity:
            Not Before: Tue Nov 06 20:13:08 2001
            Not After: Mon Feb 06 20:13:08 2006
        Subject: CN=Certificate Shack, O=TestCentral, C=US
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    dd:0b:21:ec:70:63:27:4f:43:2b:48:74:b2:b8:b1:
                    d2:88:bb:25:5b:5b:f7:7e:35:5d:42:72:9e:71:d6:
                    36:47:17:a9:15:29:0a:c8:0a:48:f0:ca:f8:f7:10:
                    21:37:28:9f:cd:d4:e7:1f:3d:2f:00:a3:83:63:44:
                    f1:be:ea:1b:33:0d:d1:00:bc:0e:65:91:c8:aa:7b:
                    d0:27:9d:6d:ac:62:f4:90:e7:ac:e4:c0:80:01:ef:
                    37:3f:89:8b:31:a6:dd:a8:e4:d1:37:c7:c3:6e:45:
                    ce:db:94:60:2a:27:4f:b2:a2:56:f7:ff:db:a4:02:
                    f8:99:dc:db:a8:d6:12:c5
                Exponent: 65537 (0x10001)
        Signed Extensions:
            Name:
                Certificate Type
            Data: <SSL CA,S/MIME CA,ObjectSigning CA>

            Name:
                Certificate Basic Constraints
            Critical:
                True
            Data: Is a CA with a maximum path length of 10.

            Name:
                Certificate Key Usage
            Data:
                03:02:02:04

    Fingerprint (MD5):
        D4:1D:8C:D9:8F:00:B2:04:E9:80:09:98:EC:F8:42:7E
    Fingerprint (SHA1):
        DA:39:A3:EE:5E:6B:4B:0D:32:55:BF:EF:95:60:18:90:AF:D8:07:09

    Signature Algorithm: PKCS #1 MD5 With RSA Encryption
    Signature:
        db:1e:af:69:8c:fe:25:93:87:88:fd:0f:7e:9b:cc:84:d2:e2:
        d2:5a:17:a4:2c:df:f3:98:64:52:00:cf:13:35:45:99:5a:ef:
        e2:5b:f9:e5:30:dd:5f:3f:b0:ba:6b:f1:c3:8c:b5:be:c3:14:
        0f:ca:19:c8:47:29:7c:ce:3e:20:18:c4:a7:f1:5c:33:7f:74:
        83:6b:74:39:b7:5f:37:dd:50:1d:df:b8:60:e4:f9:0e:78:66:
        b1:01:4f:b3:14:1f:03:e5:33:4e:c4:be:5e:fb:78:19:40:fd:
        cd:8e:e1:a0:ee:22:ca:e2:2b:36:8d:f9:fd:2d:ef:0b:00:f1:
        04:84
    Certificate Trust Flags:
        SSL Flags:
            Valid CA
            Trusted CA
            Trusted Client CA
        Email Flags:
        Object Signing Flags:

The SSL client is configured to present a given certificate stored in its
database. The client DB looks like this :

(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/internal/B1/SunOS5.6_DBG.OBJ/tests/httptests{60}
certutil -d suite1/certs/client/ -L
beta                                                         u,pu,u
alpha                                                        u,pu,u
gamma                                                        u,pu,u
cacert                                                       CTu,Cu,Cu
(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/internal/B1/SunOS5.6_DBG.OBJ/tests/httptests{61}


The client cert being presented in this test is "alpha".

(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/internal/B1/SunOS5.6_DBG.OBJ/tests/httptests{62}
certutil -d suite1/certs/client/ -n alpha -L
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
        Signature Algorithm: PKCS #1 MD5 With RSA Encryption
        Issuer: CN=Certificate Shack, O=TestCentral, C=US
        Validity:
            Not Before: Tue Nov 06 20:13:35 2001
            Not After: Mon Feb 06 20:13:35 2006
        Subject: E=alpha@testcentral.com, CN=Franzl Alpha, UID=alpha, OU=People,
O=TestCentral, C=US
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    be:94:74:78:3f:44:d0:9c:da:88:c8:a7:21:41:81:
                    b5:fc:4b:88:f9:7e:df:52:c2:fa:7b:83:82:84:90:
                    f4:ec:dc:d1:c2:f6:08:8d:c9:8e:4b:85:2d:1a:bf:
                    a1:3e:35:8f:79:24:2c:4a:59:71:b0:8e:86:dd:93:
                    18:b1:00:b8:25:e9:13:df:2b:cd:c6:c7:83:f7:84:
                    a1:a8:b0:15:07:4a:c6:3a:fc:7d:d5:ee:37:6e:f7:
                    fb:32:23:37:4b:60:18:a5:b6:e8:3e:e8:7d:85:69:
                    df:68:85:f8:9b:5f:38:16:cb:b4:14:9b:03:35:33:
                    51:45:9f:62:36:9b:df:43
                Exponent: 65537 (0x10001)
        Signed Extensions:
            Name:
                Certificate Type
            Data: <SSL Client>

            Name:
                Certificate Key Usage
            Data:
                03:02:05:a0

    Fingerprint (MD5):
        D4:1D:8C:D9:8F:00:B2:04:E9:80:09:98:EC:F8:42:7E
    Fingerprint (SHA1):
        DA:39:A3:EE:5E:6B:4B:0D:32:55:BF:EF:95:60:18:90:AF:D8:07:09

    Signature Algorithm: PKCS #1 MD5 With RSA Encryption
    Signature:
        93:2f:cf:0b:fb:5e:b8:97:2a:c5:c1:a5:11:fb:16:43:fc:b3:
        31:27:aa:fd:8b:db:06:38:db:78:87:f9:ad:2f:e4:9a:35:0e:
        29:4f:3e:cb:f5:d0:92:fb:29:78:1e:f4:e1:af:c6:86:6f:f7:
        ae:2c:52:d0:5e:a7:0f:18:8f:70:33:6b:d5:1a:e1:3d:ec:85:
        50:5c:2b:e2:e4:b6:a0:35:1c:85:79:05:80:ce:1b:c9:3e:a8:
        01:f0:4c:23:34:d2:18:76:57:f7:f4:34:f6:a5:ef:66:ec:a7:
        04:36:a8:89:be:51:5e:58:3b:04:44:37:ef:55:0f:eb:b4:5a:
        79:81
    Certificate Trust Flags:
        SSL Flags:
            User
        Email Flags:
            Valid Peer
            User
        Object Signing Flags:
            User

(strange)/export/home/jpierre/60/SunOS_5.8_depend/ns/server/internal/B1/SunOS5.6_DBG.OBJ/tests/httptests{63}

(Assignee)

Comment 13

16 years ago
This is essentially the same setup I have.  In fact, my setup should be *more*
stressful (no pun intended), because I'm using the extended SSL configuration. 
Both the client and server have certs that are 3 deep in the chain.

I cranked up the both the number of client and server threads to 100, and still
nothing crashes.  Can you think of any reason why the webserver would take a
different path than selfserv?
(Reporter)

Comment 14

16 years ago
I just tried with selfserv. I got a core dump as well.

Here is the stack :

(dbx) where
current thread: t@88
  [1] pthread_mutex_lock(0xdadadada, 0x10000, 0x0, 0xff0ee000, 0xff0fa19c,
0xfd831d78), at 0xff0cb6c0
=>[2] PR_Lock(lock = 0xdadadada), line 183 in "ptsynch.c"
  [3] nssPKIObject_AddRef(object = 0x103690), line 89 in "certdecode.c"
  [4] nssCertificate_AddRef(c = 0x103690), line 71 in "certificate.c"
  [5] add_cert_to_cache(td = 0x7a540, cert = 0x103690), line 728 in "tdcache.c"
  [6] nssTrustDomain_AddCertsToCache(td = 0x7a540, certs = 0xfd9ce6c0, numCerts
= 1U), line 762 in "tdcache.c"
  [7] NSSTrustDomain_FindBestCertificateBySubject(td = 0x7a540, subject =
0x27ac90, timeOpt = 0x95f40, usage = 0xfd9ce818, policiesOpt = (nil)), line 607
in "trustdomain.c"
  [8] NSSCertificate_BuildChain(c = (nil), timeOpt = 0x95f40, usage =
0xfd9ce818, policiesOpt = (nil), rvOpt = 0xfd9ce80c, rvLimit = 2U, arenaOpt =
(nil), statusOpt = 0xfd9ce808), line 363 in "certificate.c"
  [9] CERT_FindCertIssuer(cert = 0x27b530, validTime = 1013123479534947LL, usage
= certUsageSSLClient), line 420 in "certvfy.c"
  [10] CERT_VerifyCertChain(handle = 0x7a540, cert = 0x27b530, checkSig = 1,
certUsage = certUsageSSLClient, t = 1013123479534947LL, wincx = (nil), log =
(nil)), line 713 in "certvfy.c"
  [11] CERT_VerifyCert(handle = 0x7a540, cert = 0x27b530, checkSig = 1,
certUsage = certUsageSSLClient, t = 1013123479534947LL, wincx = (nil), log =
(nil)), line 1149 in "certvfy.c"
  [12] CERT_VerifyCertNow(handle = 0x7a540, cert = 0x27b530, checkSig = 1,
certUsage = certUsageSSLClient, wincx = (nil)), line 1190 in "certvfy.c"
  [13] SSL_AuthCertificate(arg = 0x7a540, fd = 0x58a88, checkSig = 1, isServer =
1), line 270 in "sslauth.c"
  [14] mySSLAuthCertificate(arg = 0x7a540, fd = 0x58a88, checkSig = 1, isServer
= 1), line 277 in "selfserv.c"
  [15] ssl3_HandleCertificate(ss = 0x1eebb8, b = 0x1e7606 "^P", length = 0),
line 6561 in "ssl3con.c"
  [16] ssl3_HandleHandshakeMessage(ss = 0x1eebb8, b = 0x1e739c "", length =
618U), line 7157 in "ssl3con.c"
  [17] ssl3_HandleHandshake(ss = 0x1eebb8, origBuf = 0x2cf2e4), line 7273 in
"ssl3con.c"
  [18] ssl3_HandleRecord(ss = 0x1eebb8, cText = 0xfd9cee64, databuf = 0x2cf2e4),
line 7538 in "ssl3con.c"
  [19] ssl3_GatherCompleteHandshake(ss = 0x1eebb8, flags = 0), line 204 in
"ssl3gthr.c"
  [20] SSL_ForceHandshake(fd = 0x58a88), line 372 in "sslsecur.c"
  [21] handle_connection(tcp_sock = 0x58a88, model_sock = 0x58548, requestCert =
4), line 937 in "selfserv.c"
  [22] jobLoop(a = (nil), b = (nil), c = 4), line 455 in "selfserv.c"
  [23] thread_wrapper(arg = 0x8b5e8), line 423 in "selfserv.c"
  [24] _pt_root(arg = 0x8f328), line 214 in "ptthread.c"
(dbx) where
current thread: t@88
  [1] pthread_mutex_lock(0xdadadada, 0x10000, 0x0, 0xff0ee000, 0xff0fa19c,
0xfd831d78), at 0xff0cb6c0
=>[2] PR_Lock(lock = 0xdadadada), line 183 in "ptsynch.c"
  [3] nssPKIObject_AddRef(object = 0x103690), line 89 in "certdecode.c"
  [4] nssCertificate_AddRef(c = 0x103690), line 71 in "certificate.c"
  [5] add_cert_to_cache(td = 0x7a540, cert = 0x103690), line 728 in "tdcache.c"
  [6] nssTrustDomain_AddCertsToCache(td = 0x7a540, certs = 0xfd9ce6c0, numCerts
= 1U), line 762 in "tdcache.c"
  [7] NSSTrustDomain_FindBestCertificateBySubject(td = 0x7a540, subject =
0x27ac90, timeOpt = 0x95f40, usage = 0xfd9ce818, policiesOpt = (nil)), line 607
in "trustdomain.c"
  [8] NSSCertificate_BuildChain(c = (nil), timeOpt = 0x95f40, usage =
0xfd9ce818, policiesOpt = (nil), rvOpt = 0xfd9ce80c, rvLimit = 2U, arenaOpt =
(nil), statusOpt = 0xfd9ce808), line 363 in "certificate.c"
  [9] CERT_FindCertIssuer(cert = 0x27b530, validTime = 1013123479534947LL, usage
= certUsageSSLClient), line 420 in "certvfy.c"
  [10] CERT_VerifyCertChain(handle = 0x7a540, cert = 0x27b530, checkSig = 1,
certUsage = certUsageSSLClient, t = 1013123479534947LL, wincx = (nil), log =
(nil)), line 713 in "certvfy.c"
  [11] CERT_VerifyCert(handle = 0x7a540, cert = 0x27b530, checkSig = 1,
certUsage = certUsageSSLClient, t = 1013123479534947LL, wincx = (nil), log =
(nil)), line 1149 in "certvfy.c"
  [12] CERT_VerifyCertNow(handle = 0x7a540, cert = 0x27b530, checkSig = 1,
certUsage = certUsageSSLClient, wincx = (nil)), line 1190 in "certvfy.c"
  [13] SSL_AuthCertificate(arg = 0x7a540, fd = 0x58a88, checkSig = 1, isServer =
1), line 270 in "sslauth.c"
  [14] mySSLAuthCertificate(arg = 0x7a540, fd = 0x58a88, checkSig = 1, isServer
= 1), line 277 in "selfserv.c"
  [15] ssl3_HandleCertificate(ss = 0x1eebb8, b = 0x1e7606 "^P", length = 0),
line 6561 in "ssl3con.c"
  [16] ssl3_HandleHandshakeMessage(ss = 0x1eebb8, b = 0x1e739c "", length =
618U), line 7157 in "ssl3con.c"
  [17] ssl3_HandleHandshake(ss = 0x1eebb8, origBuf = 0x2cf2e4), line 7273 in
"ssl3con.c"
  [18] ssl3_HandleRecord(ss = 0x1eebb8, cText = 0xfd9cee64, databuf = 0x2cf2e4),
line 7538 in "ssl3con.c"
  [19] ssl3_GatherCompleteHandshake(ss = 0x1eebb8, flags = 0), line 204 in
"ssl3gthr.c"
  [20] SSL_ForceHandshake(fd = 0x58a88), line 372 in "sslsecur.c"
  [21] handle_connection(tcp_sock = 0x58a88, model_sock = 0x58548, requestCert =
4), line 937 in "selfserv.c"
  [22] jobLoop(a = (nil), b = (nil), c = 4), line 455 in "selfserv.c"
  [23] thread_wrapper(arg = 0x8b5e8), line 423 in "selfserv.c"
  [24] _pt_root(arg = 0x8f328), line 214 in "ptthread.c"
(dbx) 

My selfserv command was :

selfserv -D -n Server-Cert -p 2000 -w enterprise -t 100 -d . -r -r -r -r

On the client side, I used the NES client on Win2K :

D:\60\WINNT_5.0_depend\ns\server\internal\B1\WINNT4.0_DBG.OBJ\tests\httptests>httptest
-h strange:2000 -e 3600 -s -P -L 30 -p 40 -H 1 -g COMMON -g SSL -x acl/aclS01d

That test presents a client cert "alpha" when requested.
I am attaching the dbs for both the client and server.
(Reporter)

Comment 15

16 years ago
Created attachment 68416 [details]
key database for server, password enterprise
(Reporter)

Comment 16

16 years ago
Created attachment 68417 [details]
cert database for server
(Reporter)

Comment 17

16 years ago
Created attachment 68419 [details]
cert database for client
(Reporter)

Comment 18

16 years ago
Created attachment 68420 [details]
key database for client, password enterprise
(Reporter)

Comment 19

16 years ago
oops. client key3.db password is httptest, not enterprise
(Reporter)

Comment 20

16 years ago
Created attachment 68437 [details]
ssltap output with strsclnt as client
(Reporter)

Comment 21

16 years ago
Created attachment 68438 [details]
ssltap output with httptest as client
(Reporter)

Comment 22

16 years ago
Somehow I can only reproduce the crash when using httptest as the client.
I have tried using ssltap to see what each client was doing differently, and
everything looks very similar. It must be a timing issue.
(Assignee)

Comment 23

16 years ago
Bob, can you review the patch?  I'd like to get it into tonight's builds. 
Julien has only been able to reproduce the bug on one machine, so if we get it
in the builds we can try more machines tomorrow.
(Reporter)

Comment 24

16 years ago
I just reproduced it on a second machine, on windows with selfserv.
I also reproduced it on my ultra10 with Solaris8 with both selfserv and NES.
I always have to use httptest as the client though, strsclnt won't do for some
reason.
(Assignee)

Comment 25

16 years ago
checked in patch to get it in tomorrow's builds.  There was an additional
one-liner needed, and it seems to have eliminated the crash Julien was seeing.

Comment 26

16 years ago
Ian, what is the one-liner that eliminated the crash Julien
was seeing?  Have we understood the difference between our
strsclnt and Julien's httptest client?  Just curious.
(Reporter)

Comment 27

16 years ago
Wan-Teh,

Ian and I spent the better part of the day discussing this online on IRC,
sending files, etc.

The difference ended up that strsclnt was doing restart handshakes, and I was 
doing full handshakes in httptest.

ssltap didn't show a difference because I took the trace of the first session of
each client, which is always a full handshake.

On strsclnt the corresponding option to disable the SSL cache is "-N". Once Ian
used that, he reproduced the crash very quickly, and fixed it with the one-liner.

I have been able to run the server for a while without crashing so far, except
that my Sparc machine ran out of memory after about 8000 sessions due to the
memory leak.

Ian said he got past this crash but ran into another, so there may be another
defect coming up soon - or perhaps not.

In any event, I consider this defect to be fixed.


(Reporter)

Comment 28

16 years ago
Marking fixed due to Ian's checkins.
Status: NEW → RESOLVED
Last Resolved: 16 years ago
Resolution: --- → FIXED
(Reporter)

Comment 29

16 years ago
Marking VERIFIED.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.