216695 - SSL client auth fails if CA certificate is renewed in client but not in server

Status: NEW

(NSS :: Libraries, defect, P3)

Description

[Julien Pierre]

Nelson ran into the following problem recently :

After getting the renewed intranet CA certificate, he could no longer login to
the IMAP server with SSL client auth.

Here is what's happening :

The IMAP SSL server is configured to request client auth. The acceptable CA
certificate is the old intranet CA, marked trusted for client auth.

When Nelson logged in, his SSL client (Mozilla) submitted a certificate chain
that included his leaf certificate as well as the new intranet CA certificate.

The new CA certificate is in all respects identical to the old one except for
the validity date and the signature. In particular the public key in the new
certificate is identical to the public key in the old certificate - this is the
classic renewal case.

Because of this, Nelson's leaf certificate can chain to either the new CA
certificate or the old CA certificate.

The problem is this :

When the IMAP server receives Nelson's certificate chain, it creates temporary
certificates for each certificate - the leaf cert and the new CA certificate.
Neither certificate exists in the server's certificate database. The
certificates chain to a known root, but the root is not trusted for client auth
by the server. Therefore Nelson's chain is rejected.

Here is what should be happening :
When verifying Nelson's certificate chain, the server should start from the
leaf, and then ask the question "what's the trusted issuer certificate for this
certificate" ?
Instead, it finds the newest issuer certificate, using the function
CERT_FindCertIssuer . Unfortunately, that new certificate is untrusted because
it came from Nelson's certificate chain.

The fix for this is most likely in the certificate verification code,
CERT_VerifyCertChain . The algorithm needs to be aware that there may be
multiple issuer certificates, rather than only one, and should try the different
options. It should pick the "most trusted" issuer. Of course this in itself is
somewhat ambiguous since it depends what the caller is trying to do, for
instance what usage(s) he is trying to verify the cert against.
CERT_VerifyCertChain doesn't know that. Only the top-level CERT_VerifyCert
function knows.

This is a case where multiple paths in the cert chain verification are possible
and where they should all be explored. We currently only pick one and it's
incorrect.

The temporary work around for this is that when a CA cert is renewed, all SSL
servers doing client auth need to be upgraded with the newest CA certificate and
trust it.

Comment 1

[Julien Pierre]

Note that the client can also workaround the problem by deleting the new
intermediate CA and not sending it in the SSL client auth handshake.
But it is easier to fix on the server side.

Comment 2

[Julien Pierre]

The problem is even more severe than previously thought. There are more issues
in the certificate chain verification code.

This problem happens in two cases :

1) Nelson has a cert issued on 8/8/2003

He uses Mozilla to log in to the IMAP SSL server which trusts the issuer valid
from 6/1/2001 until 6/1/2004 .

He can't login because Mozilla sent the new issuer cert with a validity date
from 8/7/2003 to 8/7/2004, which the server doesn't trust.

2) I log in with my cert, issued on 7/23/2003 .

I use Mozilla to log in to the IMAP SSL server which trusts the issuer valid
from 6/1/2001 until 6/1/2004 .

I can't login.

It turns out that Mozilla _also_ sent the newer issuer cert, even though my cert
doesn't chain to it - its issuing date doesn't fall within the new issuer
validity date of  8/7/2003 to 8/7/2004 !!!!

The server also wasn't able to correctly distinguish Mozilla's error - it also
thinks my cert chains to that new issuer cert, which isn't trusted, and chokes
because of lack of trust ...

This appears to be due to a bug in CERT_FindCertIssuer. Somehow that function
isn't making appropriate use of the certificate issuing date.
CERT_FindCertIssuer takes a subject cert, a usage, and a date of verification .
I suspect the date of verification is being incorrectly used when building the
chains, which results in the newer issuing cert incorrectly being picked ...

Comment 3

[Julien Pierre]

We discussed this in a meeting monday. It was suggested that the correct fix is
to install the new CA cert in the server.

This sounds good in theory, but I question how practical it is today.

If we don't make NSS smarter about cert chain validation, this is the burden
that CA and web server administrators are going to have to go through :

Either :
1) as soon as a new CA cert is issued, all servers that use it must pick up the
new CA cert from the CA and trust it.

While this is the right thing to do, there is no protocol for automatically
doing this, or any other standard facility built into netscape servers to do it.
This means in practice the CA administrator must track down every server that
uses the CA cert. This isn't practical. Even if a list is maintained, the
process of updating the cert will take a long time.

If a client gets the new CA cert before a server gets it and trusts it, we run
into this problem, where half the users can't login because they have the new CA
cert.

2) when the new CA cert is issued, it is only given to the servers and not the
clients who request new certs. This would have to be done through ACLs. The CA
would hold off giving the new CA cert to clients until it is reasonably certain
that the servers have been updated with the CA cert.

All it takes is one S/MIME message "accidentally" signed with the newest CA cert
and disseminated to a mailing list to get back to case 1), where the users will
no longer be able to login.

Neither solution looks satisfactory to me.
I prefer #3 where :

3) we patch NSS to transparently handle this case.

Servers pick up the new NSS with this patch.
Whenever the CA administrator renews the CA cert (and that doesn't mean only
this time, for this particular deployment), he has plenty of time to notify the
servers that use the CA cert - basically, up until the overlap period of the two
CA certs is over.

This would mean, in this case, that there would be until 6/1/2004 to update the
CA cert on all the servers, and nothing would break in between.

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

The problem I encountered, which lead to this bug report, was caused 
because a new CA cert was issued for a CA, well in advance of the 
expiration of the CA's present cert, and the CA began to ship that new CA 
cert to users who get new client auth certs immediately after it was 
issued.  

I think the new CA cert should have been distributed to servers BEFORE it 
was distributed to new clients.  I'd suggest that the new CA cert not be
distributed to clients until at least several weeks after the new cert is 
issued.

As for distributing new intermediate CA certs, there is a mime type for 
distributing CA certs.  web browsers should be able to visit a web page
with a simple URL to get the new cert.  A multipart MIME email message
should suffice to distribute the new CA cert to email users.  I don't 
know whether our current and future standalone email clients (e.g. 
thunderbird) properly handle those MIME types.  We should ensure that 
they do.

Comment 5

[Julien Pierre]

Nelson :

In other words, you would advocate the second option I listed in comment 3.

Distributing certs to browsers is not a problem, it happens fairly transparently.

What I pointed out is that no good means exists today for a server to
automatically fetch and trust the new CA cert. Some sort of subscription model
would be needed. Nothing like that exists in CMS / NES and other servers, AFAIK.
Steve, please correct me if I'm wrong on this.

Unless we find some solution to this problem, what's going to keep happening is
the status quo - some users will be able to log in to a server, while other
won't, just because there is a new CA cert out there. Most users will have no
clue why. It will continue to be a very complicated problem to diagnose, since
the server administrators may not be aware that a new CA cert has been issued,
and they have no way of telling from any server log that a new CA cert is being
sent by their users. From a supportability point of view, this situation is a
nightmare.

On the other hand, if we do 3), and nobody installs the new CA cert, the server
will continue to work for all users when the old CA expires, and then stop
working for everybody at once after that date. That is a lot easier of a problem
to diagnose and fix.

Comment 6

[Terry Hayes]

I'd like to make one correction to Julien's description of the problem.  He 
says:

  It turns out that Mozilla _also_ sent the newer issuer cert, even though my
  cert doesn't chain to it - its issuing date doesn't fall within the new
  issuer validity date of  8/7/2003 to 8/7/2004 !!!!

This implementation is actually correct.  There is no requirement that the 
validity period of a certificate falls completely within the validity period 
of the (current) issuing certificate.

The relying party (the server) should either trust the newer cert (it's trust 
database has been updated), or should find the older certificate in the 
database and use that.  I believe this is Julien's suggested change to the NSS 
verification code.

Notice that allowing validity times to extend outside of the issuer's validity 
period makes trust database updates easier.  The server administrator should 
simply delete the old version of the trusted certificate, and install the new 
one.

Comment 7

[Julien Pierre]

Terry, that's true, I wasn't aware when I originally wrote the bug that this
chain was indeed valid and a full overlap of the validity of certs in the chain
wasn't required.

The problem still remains that the server rejected a valid chain in this case,
even though it should have been able to handle it by matching the leaf to its
trusted cert in the DB by virtue of the signature.

If we aren't going to fix the NSS code, another workaround exists for the CA,
which is to issue a new CA cert that is not yet valid. That CA cert can then be
disseminated to servers and marked trusted, in addition to the current one (and
not as a replacement in this case).

If clients get ahold of the new CA cert, it won't harm anything, since the
clients won't send the new CA cert until it actually becomes valid. Unless the
clients configure their system's date in the future, they would continue to be
able to login to all servers, whether those servers had the new CA cert or not.

However this isn't a true workaround for the problem, but rather modifying the
problem, since originally both the old and new CA cert had overlapping periods
starting immediately at the time of issuance of the new cert.