634074 - Cannot validate valid certificate chain when looping/cross-signed certs are involved

Status: NEW

(NSS :: Libraries, defect, P5)

Description

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Attachment: Screen shot of bad certificate chain

I can use https://secure.eff.org without errors or warnings in other browsers (IE9, Opera). On one system, Minefield and Firefox can navigate the site without errors or warnings. On another one, Firefox will show the page at https://secure.eff.org and then pop up a modal dialog (multiple times) saying that the certificate is not trusted, ever time the site is accessed. Clicking on links will take us to the untrusted certificate error page.

Inspecting the certificate in Firefox's certificate viewer on the bad system shows a very strange and long certificate chain that is much different than what is shown in other browsers and/or the good system, and different from what the server actually sent (as seen in Wireshark). Screenshot attached.

Comment 1

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This seems to be due to some kind of profile corruption. I will investigate the cause of the corruption and why the corruption isn't detected by the cert chain validation code.

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I have run into this same problem now on Amazon.com, with similar symptoms to bug 613977.

Comment 3

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Sorry, the bug I mentioned in Comment 2 is actually bug 619563.

Comment 4

[Johnny Stenback (:jst)]

Brian, what do you think the security rating should be for this bug?

Comment 5

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This seems less likely to be an exploitable security vulnerability than I suspected, but I am still investigating.

I suspect there is something wrong with the cert database entries for "AddTrust External CA Root" and/or "UTN - DataCorp SGC".

The EFF has a certificate for secure.eff.org and another wildcard certificate for *.eff.org. The favicon for https://secure.eff.org is https://robin.eff.org/favicon.ico and the trouble seems to occur there. robin.eff.org sends four certs in its cert chain as shown at the bottom of the attached image.

The cert chain building code in NSS does not stop building the cert chain after adding AddTrust External CA Root. Instead, it keeps building a cert chain alternating between the last cert provided (UTN - DataCorp SGC) and the root CA (AddTrust External CA Root) until it hits its maximum of 20 certs in a chain. I am investigating why this happens.

I have observed exactly the same behavior with the same certs (AddTrust External CA and UTN - DataCorp SGC) on amazon.com, but only on one occasion.

Comment 6

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

It seems like in a previous release the Addtrust cert was subordinate to the UTI - DataCorp SGC cert, and in current releases the UTI - DataCorp SGC cert is subordinate to the Addtrust cert. It also seems like it is possible for the user to get his profile into a state where the self-issued Addtrust cert and the UTI-DataCorp-issued Addtrust cert are both available. This state confuses NSS. 

In my profile, I have these CA certs, where "SKI" is "Subject Key Identifier" and "AKI" is "Certificate Authority Key Identifier":

AddTrust External CA Root
in Built-in Token Object
Issued by: AddTrust External CA Root
SKI: adbd987a34b426f7fac42654ef03bde024cb541a
AKI: adbd987a34b426f7fac42654ef03bde024cb541a

AddTrust External CA Root
in Software Security Device
Issued by: UTN - DataCorp SGC
SKI: adbd987a34b426f7fac42654ef03bde024cb541a
AKI: 5332d1b3cf7ffae0f1a05d854e92d29e451db44f 

UTN - DataCorp SGC
Issued by: UTN - DataCorp SGC
SKI: 5332d1b3cf7ffae0f1a05d854e92d29e451db44f
AKI: not provided

The website provides a different "UTN - DataCorp SGC" certificate in its cert chain:

UTN - DataCorp SGC
Issued by: AddTrust External CA Root
SKI: 5332d1b3cf7ffae0f1a05d854e92d29e451db44f
AKI: adbd987a34b426f7fac42654ef03bde024cb541a

This fourth server-provided certificate is expecting to get chained to the first AddTrust cert, but instead we chain it to the second cert. The second cert isn't a root and the it is expecting to get chained to the third certificate, but instead we are chaining it to the fourth certificate. The fourth certificate isn't a root so we chain it back to the second certificate. This creates an infinite loop. NSS breaks the loop after 20 certs have been added to the chain and returns an error message since it never reached the end of the chain.

I can observe the same trucated infinite cert chain by going to Tools -> Options -> Advanced -> View Certificates, under "The USERTRUST Network" and look at the chain in the Details tab.

Comment 7

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Attachment: The cert8.db file from the affected profile

This should block because it at least affects two very high profile sites: amazon.com and eff.org. On amazon.com, it basically prevents stylesheets and/or images from loading on Amazon's HTTPS site. When the problem occurs, it makes both sites virtually unusable. It may also drive users to the "Add Exception" UI for amazon.com (via the modal dialog) which is bad.

Create a new profile and then replace its cert8.db file with the one I am attaching to reproduce the problem. Then navigate to https://robin.eff.org. To reproduce on amazon.com, try signing out and then signing in and do a hard refresh; the styling of the page will be completely jacked.

Comment 8

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Because of the way certs are put in a shared (cross-site) cache in NSS, even if the user doesn't have two different AddTrust certs stored on his computer, an website may still be able to (accidentally or maliciously) DoS websites that use these certs: User navigates to a website that supplies a complete cert chain including one pair of certs in the SSL handshake and in another window/tab navigates to another website that supplies the other two certs in the SSL handshake. For example, this could be used as a denial of service attack on Firefox users, preventing them from logging in or ordering from Amazon.com.

Comment 9

[Rob Stradling]

Brian, I suspect that this is a duplicate of bug 479508, for which Nelson wrote an (incomplete) patch nearly 2 years ago.  It would be great to see that bug fixed!

Comment 10

[Wan-Teh Chang]

Rob: you are right one.  My code inspection last night
also ended up in nssCertificateArray_FindBestCertificate.
You can duplicate my code inspection by following the
NSSCertificate_BuildChain call with the CERT_MAX_CERT_CHAIN
argument:
http://mxr.mozilla.org/security/ident?i=NSSCertificate_BuildChain

To a human being, it's clear that when multiple issuer
certs are available, a root cert, or more generally a
trust anchor, should be favored.  So Nelson's patch in
bug 479508 is the right approach.

A risk of that approach is that nssCertificateArray_FindBestCertificate
is used by not only find_cert_issuer.  Some other callers
of nssCertificateArray_FindBestCertificate may not favor
a trust anchor.

Shall we continue the technical discussion in bug 479508,
if we can't simply mark this bug as a duplicate?

Comment 11

[Rob Stradling]

(In reply to comment #10)
> Shall we continue the technical discussion in bug 479508,

Wan-Teh: IIUC, Nelson also mentioned another approach to fix/workaround this buggy behaviour:
"I suspect the only real solution to this is for PSM to switch to always 
use libPKIX.  Bug 479393 requests just that."

I suggest that efforts should be concentrated on whichever of these 2 bugs is likely to be the simplest/quickest to resolve.

BTW, does "P1 blocker" mean that Mozilla want this buggy behaviour fixed in time for FF4.0 ?

Comment 12

[Wan-Teh Chang]

Brian: please run a quick experiment to see if libpkix
can build this certificate chain successfully.  Please
set the environment variable NSS_ENABLE_PKIX_VERIFY to 1,
and then run Firefox.  Does that solve the problem?

Rob: in Bugzilla, the severity/importance level 'blocker'
does not mean it will block a release.  The 'blocker'
level is defined as:
    blocker	Blocks development and/or testing work
https://bugzilla.mozilla.org/page.cgi?id=fields.html#importance

I lowered the severity/importance level to 'major'.

Comment 13

[Johnathan Nightingale [:johnath]]

Based on the problem description, I don't think we'd hold release for this, but it is something we'd like to see fixed. Marking it .x, but if a safe, reviewed patch comes along in time, please nominate it for approval.

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

This is clearly (IMO) a dup of bug 479508, and (IMO) is not worthy of being 
security sensitive, just as bug 479508 is not security sensitive.

Has anyone here tried setting the environment variable NSS_ENABLE_PKIX_VERIFY 
to 1, to see what that does?

Comment 15

[Robert Relyea]

Inifinite loops are allowed under pkix. Because of our processing code, CA's have avoided them in the past.

PKIX tries more than one path, looking for the patch that works properly, So I join wtc and Nelson in asking what happens if you set NSS_ENABLE_PKIX_VERIFY?

It may be time to think about shift PKIX to default in the next set of releases (probably not 4.0:).

bob

Comment 16

[Doug Turner (:dougt)]

fennec can pick this up when the platform gets it.

Comment 17

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

See bug 635384. With NSS_ENABLE_PKIX_VERIFY=1, the behavior is even worse because we execute an infinite loop in pkix_BuildForwardDepthFirstSearch when attempting to access https://robin.eff.org

Comment 18

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to comment #14)
> This is clearly (IMO) a dup of bug 479508, and (IMO) is not worthy of being 
> security sensitive, just as bug 479508 is not security sensitive.

I agree, but I cannot clear the sensitive flag. I filed it as sensitive because I initially suspected a different cause.

Comment 19

[Johnny Stenback (:jst)]

Opening bug up per previous comments.

Comment 20

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This is not a regression from 3.6 as it happens in 3.6.12 too. If the cert is an EV cert then Firefox may crash with a stack overflow (not verified yet) because of bug 635384.

Comment 21

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

From Bug 639315: "check https://pki.treas.gov for some certificate bundles to import to see this issue."

Comment 23

[Kai Engert (:KaiE:)]

I confirm that
- NSS_ENABLE_PKIX_VERIFY=1 
- attached cert8.db
- visit https://robin.eff.org/

results in an endless loop.

Actually, not just a loop, but an endless recursion.
I'll attach a stacktrace with 333 frames.

Comment 24

[Kai Engert (:KaiE:)]

Attachment: pkix recursion stack

Comment 25

[Kai Engert (:KaiE:)]

Attachment: zip archive, certs sent by server

let's make sure we are able to debug this problem, even if the config of server robin.eff.org changes.

attaching a zip file that contains 5 certs sent by the server

dumped with nss tool vfyserv


        Subject: C=US/postalCode=94110, ST=California, L=San Francisco/street=454 Shotwell St, O=Electronic Frontier Foundation, OU=Comodo PremiumSSL Wildcard, CN=*.eff.org
        Subject: C=GB, ST=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO High Assurance Secure Server CA
        Subject: C=GB, ST=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO Certification Authority
        Subject: C=US, ST=UT, L=Salt Lake City, O=The USERTRUST Network, OU=http://www.usertrust.com, CN=UTN - DATACorp SGC
        Subject: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root

Comment 27

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

With the patch to NSS in bug 764393, this should be fixed without requiring the switch to libpkix. (We need to verify this.)

Wan-Teh pushed the beta of NSS with that fix to mozilla-inbound today.

Comodo has requested that we fix this bug in the mozilla-beta and mozilla-aurora channels and I agree that we should do so, as the fix is pretty simple and it fixes a serious problem.. But, I recommend that we wait at least a week to make sure Nightly does not experience any regressions.

Comment 28

[Kai Engert (:KaiE:)]

Do you intend to use this bug for "landing NSS 3.13.6 into mozilla-central and branches"?
If yes, the subject should probably mention it for clarity.
I understand you want to release NSS 3.13.6 next week.

Comment 29

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I filed a new bug for the landing of the NSS release because that's what we usually do, and because the NSS release makes several changes unrelated to this bug. That bug is bug 780009.

Comment 30

[Lukas Blakk [:lsblakk] use ?needinfo]

So what still needs to be done here now that we've updated to NSS 3.13.6?

Comment 31

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Rob/Robin, could you please verify that the latest build of Firefox 15 beta fixes the issue to your satisfaction? We will be shipping the final beta before Firefox 15 release soon.

http://www.mozilla.org/en-US/firefox/all-beta.html

Thanks for your help!

Comment 32

[Rob Stradling]

(In reply to Brian Smith (:bsmith) from comment #31)
> Rob/Robin, could you please verify that the latest build of Firefox 15 beta
> fixes the issue to your satisfaction?

Verified on Linux and Windows 7.  Thanks!

Comment 33

[Lukas Blakk [:lsblakk] use ?needinfo]

Thanks Rob, adjusting status flags accordingly.

Comment 34

[Kathleen Wilson]

Is this bug considered to be completely fixed? 
If yes, can we close this bug?

Comment 35

[Kai Engert (:KaiE:)]

(In reply to Kathleen Wilson from comment #34)
> Is this bug considered to be completely fixed? 

It depends on how you interpret this bug.

If the bug is meant to fix the special category of loops where a cross signed root is involved, then this bug is believed to be fixed.

On the other hand, if this bug rather means "fix all possible loops", then no, the bug isn't fixed. Bob said, a "cross cert to cross cert loop will require full pkix".

Comment 36

[jtaglia]

Still not working for us with Firefox 18...

Comment 37

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This is fixed in Firefox 31 in general by the switch to mozilla::pkix (bug 915930). For some cases, it was fixed in NSS by a patch that Wan-Teh wrote long ago.

The only remaining work is to write an automated test that proves that we can handle this case.

Comment 38

[Martin Rex]

I noticed this very problem (with a loop in CrossCerts from UTN-DATACorpSGC and AddTrustExternalCARoot) today in the CertifcatePatrol plugin (2.0.14.1-signed) in Firefox 38.0.1.

The looping Cert chain was reported/visualized for the _old_ Certificate in the CertificatePatrol plugin for the site adblockplus.org (for a server certificate that was valid from 09-Apr-2014 to 10-Apr-2015).  The current/new server certificate does not have this problem.

Is this a problem in the CertificatePatrol plugin, or is the problem not really fixed in the underlying Firefox code?

Comment 39

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

It could be that the certificate viewer is doing the wrong thing (I suspect GetChain). Would you please file a new bug with the relevant certificates attached? Thanks.

Comment 40

[Martin Pecka]

I've found out this bug has much more critical consequences. With the bad UTN-DATACorpSGC certificate in the cert store, when I try to login to startssl.com using my client certificate, Firefox enters an endless loop and gets completely frozen. Even the multiprocess nightly. Removing this cert stops the freezing from happening.

Comment 41

[Kathleen Wilson]

(In reply to Martin Pecka from comment #40)
> I've found out this bug has much more critical consequences. With the bad
> UTN-DATACorpSGC certificate in the cert store, when I try to login to
> startssl.com using my client certificate, Firefox enters an endless loop and
> gets completely frozen. Even the multiprocess nightly. Removing this cert
> stops the freezing from happening.

Good to know. We plan to remove that root in Firefox 44 -- Bug #1208461

Comment 42

[Rob Stradling]

(In reply to Martin Pecka from comment #40)
> Removing this cert stops the freezing from happening.

Martin, which cert did you remove to stop the freezing from happening?  Was it the self-signed "UTN - DATACorp SGC" built-in root (that bug #1208461 will be removing soon), or was it one of the "UTN - DATACorp SGC"<--->"AddTrust External CA Root" cross-certificates?

Comment 43

[Martin Pecka]

(In reply to Rob Stradling from comment #42)
> Martin, which cert did you remove to stop the freezing from happening?  Was
> it the self-signed "UTN - DATACorp SGC" built-in root (that bug #1208461
> will be removing soon), or was it one of the "UTN - DATACorp
> SGC"<--->"AddTrust External CA Root" cross-certificates?

Rob, I removed the UTN certificate signed by Addtrust. The self-signed UTN hasn't been touched.

Comment 44

[Rob Stradling]

Thanks Martin.  That's what I thought.

Kathleen, I don't think bug #1208461 will actually have any effect on any remaining cross-cert looping bugs.

Comment 45

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

This is an NSS bug. TLS certificate verification in Firefox won't have this problem because mozilla::pkix knows how to handle looping/cross-signing. (Note that this can still affect Firefox because there are places where the affected NSS functions will still be called (e.g. when libssl builds the client certificate chain if requested by the server and selected by the user)).

Comment 46

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.