453887 - check for out-of-business CA's that we still might be shipping

Status: RESOLVED DUPLICATE of bug 534274

(CA Program :: CA Certificate Root Program, task)

Description

[chris hofmann]

In the defcon talk https://www.defcon.org/html/defcon-16/dc-16-speakers.html#Dingledine 

the claim was made that "firefox ships 200 some odd certificates, many of which are CAs that are no longer in existances, and some of which goverenment agencies or rouge organizations might be trying to snatch up."

do we have a process in place to review the certs and CAs and disable?

if there is a good response and process in place we should try to add some clearity on this issue.

Comment 1

[Gervase Markham [:gerv]]

We have stats on which CAs are used in our report from NetCraft. We could certainly use that to generate candidates for consideration.

Gerv

Comment 2

[chris hofmann]

If we could complete this review before we ship firefox 3.1 and identify any bad ones we might have an opportunity yank certs from the bad CAs and not have to go through nasty/harder revocation.

Comment 3

[Gervase Markham [:gerv]]

I attempted to go through the list on the Authorities tab in the advanced Security preferences in my copy of Firefox and match it up with the Netcraft data. (Now, someone else trying this could of course substitute johnath's free data for the Netcraft stuff, although they'd need to do a little work to get the list of CAs.) I hit the following snags:

- I can't copy and paste the list out of Firefox;

- The list in my Firefox is "polluted" by CAs I've chosen personally to trust, 
  and there is no marking on the list to tell which those are;

- I don't quite understand why, if we are shipping only roots, there is a 
  two-level hierarchy; where do the two levels come from?

- The names in Netcraft's list of authorities don't seem to exactly match those 
  displayed in Firefox.

So, questions: can anyone explain the source of the text in the list in Firefox better? Do we define it or is it in the certificates? If we define it, is there a mapping somewhere? Is there anywhere to get a clean ASCII/UTF-8 version of the list of CA names? certdata.txt? 

Johnath's scanner doesn't capture the entire certificate chain - would it need to in order to be useful for this? And can anyone who has access to Netcraft tell me where exactly it's getting the text it uses to label each row of data?

Thanks :-)

Gerv

Comment 4

[Johnathan Nightingale [:johnath]]

(In reply to comment #3)
> Johnath's scanner doesn't capture the entire certificate chain - would it need
> to in order to be useful for this? And can anyone who has access to Netcraft
> tell me where exactly it's getting the text it uses to label each row of data?

In order to be maximally useful to you, yes, I think it would have to.  I will try to find some time this week to cook that together and start running another crawl, but it would be next week at the absolute earliest before we'd have any useful data.

On the other hand, I could probably produce a list of every (direct) issuer I've seen, which might help you at least confirm the liveliness of those CAs issuing directly off the root.  That might be sufficiently arduous that you're better off waiting for me to rewrite the crawler, but it's an option, if you'd like.

Comment 5

[Gervase Markham [:gerv]]

I think waiting seems like a good option :-)

Gerv

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Does this bug need to be "security sensitive"?  Seems to me more like 
ordinary spring house cleaning, or perhaps "bring out your dead". :)

I'm adding Kai to the CC list, since he is probably best qualified to 
answer the questions of Comment 3, but I will take a stab at them here.

The "Authorities" tab in cert manager is NOT merely a list of root CAs. 
It is a list of all CA certs, grouped by the Organization name in the cert's
Issuer name.  The higher/outer level of the hierarchy is that: the Issuer 
Organization name.  This grouping causes a root cert to be grouped together
with the CA certs that are issued directly by the root.  Since the root's
issuer name is also the issuer name of its top level subordinate CA certs.
This grouping tends to cause all the roots and top level subordinate CAs 
from a single CA organization to be grouped together.  Hence we see all of
Verisign's roots and top-level subordinate CA certs that bear Verisign's
brand name in the organization grouped together (for example).

The second level of the hierarchy shows the Subject Common Name in each CA cert.

I don't know what names appear in the Netcraft list you mentioned.

When you talk about a clean list of CA names, you must define what names
you want.  Subject names?  All name attributes? or only some?

Comment 7

[Johnathan Nightingale [:johnath]]

(In reply to comment #6)
> When you talk about a clean list of CA names, you must define what names
> you want.  Subject names?  All name attributes? or only some?

So, our intent is to find likely candidates for obsolescence by starting with a list of all CAs and then using something like my crawler data to eliminate CAs that are provably still issuing. Let's assume that I modify the crawler to collect all intermediates up to the root as part of the crawl.  Would you agree, then, that running a query afterwards to extract the subject CN from all (top-level) roots I scrape would be a good list to work from?  Is that an easy list to produce for our list of CAs?

(In reply to comment #6)
> Does this bug need to be "security sensitive"?  Seems to me more like 
> ordinary spring house cleaning, or perhaps "bring out your dead". :)

I agree that it doesn't need to be hidden.  The report cited in comment 0 is public, as is our list of CAs.  Unhiding.

Comment 8

[Chris Palmer]

I just got off the phone with Brandon Sterne from Mozilla security. He and I agreed that this bug should be resolved soon. The main thing is, the browser will believe any CA's claim about any site --- not just the correct CA's claim about the site. So even though SuperGreatBank uses Verisign to sign its site cert, Firefox users have to hope that Weird Old Dead CA, LLC is not used to sign the cert of an impostor bank site.

Luckily, Windows/IE already resolved their version of this bug (in Vista and later), so when Mozilla removes these old dead CA certs, you won't be causing any *new* problems for users.

Additionally, there is Sotirov, et al.'s "MD5 Considered Harmful Today" attack:

http://www.win.tue.nl/hashclash/rogue-ca/

In which they got a trusted CA to sign a cert (using the MD5 with RSA signing algo). Since MD5 is broken, they were able to re-use the signature to sign a totally evil intermediate CA cert, and hilarity ensues.

So, it'd be nice for Mozilla to deprecate and eventually disable the use of MD5 with RSA anywhere in the certificate signature chain (not just at the root).

In addition to phasing out MD5, you'll want to be on top of integrating support for the new hash algorithm(s) coming down the pike:

http://csrc.nist.gov/groups/ST/hash/sha-3/

The sooner browsers support these functions (for TLS/SSL data integrity checking as well as for certificate signing), the better for everyone.

Comment 9

[Johnathan Nightingale [:johnath]]

Chris, you're describing 3 or 4 different bugs here.

This bug is about weeding old CAs out of the root store, which is not really something that needs advocacy or hypotheticals. Certainly anyone who works on security will find it is a hard proposition to disagree with in the abstract. The question is finding resource to perform the review, and a good set of criteria for exclusion.

(In reply to comment #8)
> Luckily, Windows/IE already resolved their version of this bug (in Vista and
> later), so when Mozilla removes these old dead CA certs, you won't be causing
> any *new* problems for users.

This characterization seems pretty inflammatory, doesn't it?  Are you suggesting that the current situation has caused problems historically?

> Additionally, there is Sotirov, et al.'s "MD5 Considered Harmful Today" attack:
> 
> http://www.win.tue.nl/hashclash/rogue-ca/
> 
> In which they got a trusted CA to sign a cert (using the MD5 with RSA signing
> algo). Since MD5 is broken, they were able to re-use the signature to sign a
> totally evil intermediate CA cert, and hilarity ensues.
> 
> So, it'd be nice for Mozilla to deprecate and eventually disable the use of MD5
> with RSA anywhere in the certificate signature chain (not just at the root).

Yes, this has been discussed extensively both within the mozilla community and between browsers. It's not really part of this bug though, given that the cert was issued by an active CA, right?  Bug 471539 is tracking md5 retirement, and bug 472823 is discussing the implications for digest auth, for instance.

> In addition to phasing out MD5, you'll want to be on top of integrating support
> for the new hash algorithm(s) coming down the pike:
> 
> http://csrc.nist.gov/groups/ST/hash/sha-3/
> 
> The sooner browsers support these functions (for TLS/SSL data integrity
> checking as well as for certificate signing), the better for everyone.

SHA-3 isn't scheduled to be an official recommendation until 2012, but I'm quite certain that our crypto community would welcome your help if you're interested in implementing some of the candidates. In the meantime, though, we've supported the SHA-2 family for some time, against which I'm not aware of any practical attacks.  In either event, though, it's probably also a topic for another bug?

Comment 10

[Chris Palmer]

(In reply to comment #9)

> Chris, you're describing 3 or 4 different bugs here.

They all seem closely related to me, so I felt ok mentioning them here. I apologize for polluting the thread.

> This bug is about weeding old CAs out of the root store, which is not really
> something that needs advocacy or hypotheticals. Certainly anyone who works on
> security will find it is a hard proposition to disagree with in the abstract.
> The question is finding resource to perform the review, and a good set of
> criteria for exclusion.

In my experience, it actually does need advocacy, unfortunately. Obviously, you already understand these issues -- and that puts you in a very small circle of knowledgeable people (even among security professionals and skilled developers).

> > Luckily, Windows/IE already resolved their version of this bug (in Vista and
> > later), so when Mozilla removes these old dead CA certs, you won't be causing
> > any *new* problems for users.
> 
> This characterization seems pretty inflammatory, doesn't it?  Are you
> suggesting that the current situation has caused problems historically?

I didn't mean to say anything inflammatory, and I don't think I did. I do apologize if I unintentionally did.

What I meant was that, in a sense, Microsoft is playing interference for you here: When Mozilla winnows its CA cert store, you won't be introducing new incompatibilities ("Firefox says this site is not authenticated! What!") that Microsoft has not already introduced.

By design, the current model is that browsers do trust all CAs in their root cert stores to vouch for the identity of all sites the user visits. I don't know of any specific instance of that trust being abused (other than Sotirov's demo), but it could have happened.

Here's a Microsoft document discussing criteria CAs must meet to get in the Windows CA store. Perhaps it would be helpful when you are deciding which CA certs to remove:

http://technet.microsoft.com/en-us/library/cc751157.aspx

Comment 11

[Alex Stamos]

> Chris, you're describing 3 or 4 different bugs here.
> This bug is about weeding old CAs out of the root store, which is not really
> something that needs advocacy or hypotheticals. Certainly anyone who works on
> security will find it is a hard proposition to disagree with in the abstract.
> The question is finding resource to perform the review, and a good set of
> criteria for exclusion.
> (In reply to comment #8)
> > Luckily, Windows/IE already resolved their version of this bug (in Vista and
> > later), so when Mozilla removes these old dead CA certs, you won't be causing
> > any *new* problems for users.
> This characterization seems pretty inflammatory, doesn't it?  Are you
> suggesting that the current situation has caused problems historically?
> > Additionally, there is Sotirov, et al.'s "MD5 Considered Harmful Today" attack:
> > 
> > http://www.win.tue.nl/hashclash/rogue-ca/
> > 
> > In which they got a trusted CA to sign a cert (using the MD5 with RSA signing
> > algo). Since MD5 is broken, they were able to re-use the signature to sign a
> > totally evil intermediate CA cert, and hilarity ensues.
> > 
> > So, it'd be nice for Mozilla to deprecate and eventually disable the use of MD5
> > with RSA anywhere in the certificate signature chain (not just at the root).
> Yes, this has been discussed extensively both within the mozilla community and
> between browsers. It's not really part of this bug though, given that the cert
> was issued by an active CA, right?  Bug 471539 is tracking md5 retirement, and
> bug 472823 is discussing the implications for digest auth, for instance.
> > In addition to phasing out MD5, you'll want to be on top of integrating support
> > for the new hash algorithm(s) coming down the pike:
> > 
> > http://csrc.nist.gov/groups/ST/hash/sha-3/
> > 
> > The sooner browsers support these functions (for TLS/SSL data integrity
> > checking as well as for certificate signing), the better for everyone.
> SHA-3 isn't scheduled to be an official recommendation until 2012, but I'm
> quite certain that our crypto community would welcome your help if you're
> interested in implementing some of the candidates. In the meantime, though,
> we've supported the SHA-2 family for some time, against which I'm not aware of
> any practical attacks.  In either event, though, it's probably also a topic for
> another bug?

Comment 12

[Johnathan Nightingale [:johnath]]

(In reply to comment #10)
> What I meant was that, in a sense, Microsoft is playing interference for you
> here: When Mozilla winnows its CA cert store, you won't be introducing new
> incompatibilities ("Firefox says this site is not authenticated! What!") that
> Microsoft has not already introduced.

Oh, I understand now.  No need to apologize, that was my misinterpretation, sorry about that.  Yes, I agree, unless we remove a CA they haven't, it certainly helps to know that those CAs were already on the way out elsewhere.

> Here's a Microsoft document discussing criteria CAs must meet to get in the
> Windows CA store. Perhaps it would be helpful when you are deciding which CA
> certs to remove:
> 
> http://technet.microsoft.com/en-us/library/cc751157.aspx

Yeah, Microsoft has been spending a lot of time cleaning up their lists lately and it's great to see. As we are able to follow suit, I think we all agree that it will be a good thing.

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

Advocacy belongs in Mozilla's news groups and mailing lists, not in bugs.
Discussions of Mozilla's CA certificate policy occur in the newsgroup
news://news.mozilla.org:119/mozilla.dev.security.policy

Having said that, I will add that I did not find the remark about IE to
be inflammatory or advocating.  I understood it to be informing us that IE 
no longer supports certain old CAs in Vista, and so Mozilla can also remove 
support for those CAs without any worries of driving users of FF on Vista 
away from FF towards IE.  That's interesting, but the major issue is the 
state of the CA, not the platforms that support it.  

I believe Johnathan's point was that we must exercise as much diligence 
and care in the removal of CAs from the list as we do for the addition of
CAs to the list.  There's a well-defined (and somewhat lengthy) process 
by which those things happen.  The decisions for individual CAs are made
publicly, in the aforementioned newsgroup.  

This bug serves as a place for people to name root CAs that they believe 
are no longer operating, which can serve as a starting point for the 
evaluation process to begin.

Comment 14

[Alex Stamos]

Sorry about the spam comment, tab and enter don't work so well in the text box...

> > Chris, you're describing 3 or 4 different bugs here.
> > Discussions of Mozilla's CA certificate policy occur in the newsgroup

Perhaps there needs to be a separate bug for "create and publicly document the CA trust standard for Mozilla"?  I'm sure there can be good discussion on the newsgroup but it would be nice to see the decision making process documented by those with the power to make such decisions.

Some things to think about:
 -Should government agencies have CAs in the default store?

 -Should new domain-verified CAs be added in the future?  It's a little late to remove vendors that provide domain-verification at this point, but lots of people have shown how attacks like temporarily hijacking somebody's email or DNS cache poisoning can lead to the long-term security issue of a falsely issued SSL cert.

 -Should Mozilla set a date after which MD5 signature checking in the SSL chain will no longer be performed? This would be a technological change as well as a policy question.

 -Should Mozilla strengthen the policy or technological protections against default-CAs signing non-FQDNs?  For example, we have a root-CA-signed cert for "mail", which is quite useful when attacking internal enterprise networks.

Comment 15

[Eddy Nigg (StartCom)]

Alex, you really should move these question to the newsgroup. A vital group will be glad to join the discussion there. All points you raised here are actively discussed and debated at the newsgroup. Decisions are not made here.

In the meantime you might want to have a look at the Mozilla CA Policy and other documents. Leaving some links to follow for you:

http://www.mozilla.org/projects/security/certs/policy/
https://wiki.mozilla.org/CA:Problematic_Practices
https://wiki.mozilla.org/CA:Recommended_Practices

Comment 16

[Kai Engert (:KaiE:)]

Re comment 3 and comment 6, I think Gerv wants a full list of all CA names we currently ship with NSS?

cd mozilla/security/nss/lib/ckfw/builtins/
grep -i CKA_LABEL certdata.txt | sort | uniq  | sed 's#CKA_LABEL UTF8 ##' | sed 's#^\"##' | sed 's#\"$##' |grep -v ^# > /tmp/calist

This command greps the labels out of the certdata.txt file, sorts and removes the unnecessary portions from the lines.

I'm attaching the result.

Gerv, I think these strings are the nicknames we use for the certs.
If those names are not what you see in other lists, when we might use a different approach.
I could run a series of commands that dumps all certificates as human readable, and extract the fields you want. Just let me know (preferably by email for quicker response time).

Comment 17

[Kai Engert (:KaiE:)]

Attachment: listing of all current CA nicknames

Comment 18

[Johnathan Nightingale [:johnath]]

Attachment: 69 "Known Active" top-level issuers

My crawl with intermediates completed a while ago, but the DB isn't very efficient right now, and complex queries take an obscene amount of time, so I cut this one short.  

Once I get some time to optimize it, I'll run again looking for any I missed, but in the meantime, here are 69 issuers we should not remove.

Comment 19

[Kathleen Wilson]

The complete list of included certificate authorities is available from the "List of all included root certificates" link on http://www.mozilla.org/projects/security/certs/. This page is generated from a spreadsheet that I maintain and use. My version of the spreadsheet also has contact information, and dates of most recent audit reports.

I have also filed bug #534274 for removing/disabling legacy roots that I could not find recent audit reports for.

So I think this bug can be closed. Is there anything I'm missing?

Comment 20

[David E. Ross]

Are you sure that your efforts have caught all certificate authorities that are no longer in business but whose root and subscriber certificates have not yet expired?  

I would also be concerned about certificate authorities that have been purchased or merged into other entities.

Comment 21

[Johnathan Nightingale [:johnath]]

(In reply to comment #19)
> The complete list of included certificate authorities is available from the
> "List of all included root certificates" link on
> http://www.mozilla.org/projects/security/certs/. This page is generated from a
> spreadsheet that I maintain and use. My version of the spreadsheet also has
> contact information, and dates of most recent audit reports.
> 
> I have also filed bug #534274 for removing/disabling legacy roots that I could
> not find recent audit reports for.
> 
> So I think this bug can be closed. Is there anything I'm missing?

I agree, this can be closed. I'm not sure what assurance David is looking for in comment 20 beyond "I have filed a bug to remove roots without recent audits", but since I know you've got contact information for every root now, I think we're fine here. If, at some future point, another root is identified as being defunct, nothing stops us from opening another bug.

Comment 22

[Kathleen Wilson]

I have filed bug #534274 for removing/disabling legacy roots that I could
not find recent audit reports for.

I have also filed bug #549701 for two other legacy root certificates that I need to follow up on.

I am closing this bug as a duplicate of bug #534274, because that's where the remaining effort for this issue is being tracked.