Although the SSLGetClientAuthData callback can return SECFailure, NSS handles that by sending an empty client certificate (in TLS) or the no_certificate alert (in SSL 3.0). http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/nss/lib/ssl/ssl3con.c&rev=1.136&mark=5525,5530,5570,5574,5576,5578#5525 The SSLGetClientAuthData callback cannot cause ssl3_HandleCertificateRequest to return SECFailure. In some cases, it is not desirable to send an empty client certificate to a server that requests a client certificate, because the server may try to be helpful, return an HTML error page, and set a "missing or invalid client certificate" state on the HTTP "session". In those cases, we would like to abort the handshake immediately, ask the user to select a client certificate, and retry the SSL handshake with the selected client certificate, so that the HTTP "session" is never marked with the "missing or invalid client certificate" state. The attached patch is one way to fix this without changing the API. It allows the SSLGetClientAuthData callback to use the error code to communicate to NSS whether the handshake should be aborted. Note: this problem can also be solved by having the SSLGetClientAuthData callback return SECWouldBlock and calling SSL_RestartHandshakeAfterCertReq later. Unfortunately, SSL_RestartHandshakeAfterCertReq is broken.
You mention an attached patch, but I don't see any.
Created attachment 430663 [details] [diff] [review] Patch Here is the patch. The idea is that the SSLGetClientAuthData callback sets the error code to a special value before returning SECFailure to signal to ssl3_HandleCertificateRequest that the client auth should not proceed. I use SSL_ERROR_HANDSHAKE_FAILURE_ALERT in this patch, but perhaps we should add a new error code like SSL_ERROR_ABORT_CLIENT_AUTH. I found that I can solve this problem by simply having the SSLGetClientAuthData callback return SECWouldBlock. That'll cause PR_Read to return -1 with PR_WOULD_BLOCK_ERROR (for renegotiation). So I'm going to mark this bug INVALID.