Last Comment Bug 643056 - Detecting collusion between vendors and CAs: Revocation isn't enough and everyone knows it
: Detecting collusion between vendors and CAs: Revocation isn't enough and ever...
Status: VERIFIED INCOMPLETE
: meta
Product: Core
Classification: Components
Component: Security (show other bugs)
: unspecified
: All All
: -- critical with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: David Keeler [:keeler] (use needinfo?)
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-03-18 17:50 PDT by Jacob Appelbaum
Modified: 2011-04-01 11:02 PDT (History)
27 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description Jacob Appelbaum 2011-03-18 17:50:55 PDT
User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.134 Safari/534.16
Build Identifier: 

I'm planning to send this email out to the EFF ssl observatory list but I wanted to give security@mozilla.com a heads up first.


BEGIN EMAIL:

"Detecting collusion between vendors and CAs: Revocation isn't enough
and everyone knows it"

This email is probably going to be somewhat controversial and so I'd
like to caution people that I base my claims entirely on evidence linked
withing this email. Please come to your own conclusions and consider
that I mean no disrespect to any of the involved projects - this is a
very serious issue that needs to be addressed.

tl;dr:

I think a CA was compromised and important high value certificates were
issued to unknown attackers.

Certificate revocation occurs for a number of reasons and the process of
revocation is largely worthless. CRL and OCSP are both totally flawed
processes for revocation. An attacker who is able to perform a SSL/TLS
MITM is easily able to thwart the CRL and the OCSP process. It is total
garbage in every browser except Chrome when interfacing with HSTS
enabled sites.

I watch a few software projects carefully and recently, I noticed a
commit diff of extreme interest - revision 78478, Thu Mar 17 00:48:21
2011 UTC:

http://src.chromium.org/viewvc/chrome/trunk/src/net/base/x509_certificate.cc?view=markup&pathrev=78478

In the function bool X509Certificate::IsBlacklisted() we see something
rather interesting:

{0x07,0x7a,0x59,0xbc,0xd5,0x34,0x59,0x60,0x1c,0xa6,0x90,0x72,0x67,0xa6,0xdd,0x1c},
{0x04,0x7e,0xcb,0xe9,0xfc,0xa5,0x5f,0x7b,0xd0,0x9e,0xae,0x36,0xe1,0x0c,0xae,0x1e},
{0xd8,0xf3,0x5f,0x4e,0xb7,0x87,0x2b,0x2d,0xab,0x06,0x92,0xe3,0x15,0x38,0x2f,0xb0},
{0xb0,0xb7,0x13,0x3e,0xd0,0x96,0xf9,0xb5,0x6f,0xae,0x91,0xc8,0x74,0xbd,0x3a,0xc0},{0x92,0x39,0xd5,0x34,0x8f,0x40,0xd1,0x69,0x5a,0x74,0x54,0x70,0xe1,0xf2,0x3f,0x43},
{0x92,0x39,0xd5,0x34,0x8f,0x40,0xd1,0x69,0x5a,0x74,0x54,0x70,0xe1,0xf2,0x3f,0x43},
{0xd7,0x55,0x8f,0xda,0xf5,0xf1,0x10,0x5b,0xb2,0x13,0x28,0x2b,0x70,0x77,0x29,0xa3},
{0xf5,0xc8,0x6a,0xf3,0x61,0x62,0xf1,0x3a,0x64,0xf5,0x4f,0x6d,0xc9,0x58,0x7c,0x06},

We can reduce those to the following easy to read serial numbers:

047ecbe9fca55f7bd09eae36e10cae1e
d8f35f4eb7872b2dab0692e315382fb0
b0b7133ed096f9b56fae91c874bd3ac0
9239d5348f40d1695a745470e1f23f43
9239d5348f40d1695a745470e1f23f43 <-dupe
d7558fdaf5f1105bb213282b707729a3
f5c86af36162f13a64f54f6dc9587c06

The first line is stated in a comment to be a test "Not a real
certificate. For testing only" and this implies that the rest appear to
be sixteen byte serial numbers that are now blocklisted. Lines five and
six appear to be a mistake as they are identical. I suspect there will
be an update to this function; it may be the case that a certificate in
need of blocklisting was left out.

This alone is not very compelling and without the leaf certificate,
we're really not able to learn very much about the certificates that
were blocked. The CRL used for revocation is generally present in the
leaf certificate handed to a client - in theory, this means that an
attacker is unable to tamper with the CRL distribution point without
breaking the digital signature. While we lack the CRL distribution point
in this function, the project that I announced yesterday, crlwatch, was
specifically written to assist in finding who issued, and potentially
revoked the serial numbers in question.

About twelve hours (Thursday, March 17, 2011 | 13:00) after the above
patch was pushed into source control - Google announced an important
Chrome Update:
http://googlechromereleases.blogspot.com/2011/03/stable-and-beta-channel-updates_17.html

This also is mostly uninteresting until we notice that this is not
isolated to Google. Mozilla pushed out two patches of interest:
http://hg.mozilla.org/mozilla-central/rev/f6215eef2276
http://hg.mozilla.org/mozilla-central/rev/55f344578932

The first patch includes the following serial numbers in a struct called
nsSerialBinaryBlacklistEntry:
\x00\x92\x39\xd5\x34\x8f\x40\xd1\x69\x5a\x74\x54\x70\xe1\xf2\x3f\x43
\x00\xd8\xf3\x5f\x4e\xb7\x87\x2b\x2d\xab\x06\x92\xe3\x15\x38\x2f\xb0
\x72\x03\x21\x05\xc5\x0c\x08\x57\x3d\x8e\xa5\x30\x4e\xfe\xe8\xb0
\x00\xb0\xb7\x13\x3e\xd0\x96\xf9\xb5\x6f\xae\x91\xc8\x74\xbd\x3a\xc0
\x00\xe9\x02\x8b\x95\x78\xe4\x15\xdc\x1a\x71\x0a\x2b\x88\x15\x44\x47
\x00\xd7\x55\x8f\xda\xf5\xf1\x10\x5b\xb2\x13\x28\x2b\x70\x77\x29\xa3
\x04\x7e\xcb\xe9\xfc\xa5\x5f\x7b\xd0\x9e\xae\x36\xe1\x0c\xae\x1e
\x00\xf5\xc8\x6a\xf3\x61\x62\xf1\x3a\x64\xf5\x4f\x6d\xc9\x58\x7c\x06

The second patch adds two more serial numbers to that same struct:
\x39\x2a\x43\x4f\x0e\x07\xdf\x1f\x8a\xa3\x05\xde\x34\xe0\xc2\x29
\x3e\x75\xce\xd4\x6b\x69\x30\x21\x21\x88\x30\xae\x86\xa8\x2a\x71

Once again, we can reduce those to the following easy to read serial
numbers:

009239d5348f40d1695a745470e1f23f43
00d8f35f4eb7872b2dab0692e315382fb0
72032105c50c08573d8ea5304efee8b0
00b0b7133ed096f9b56fae91c874bd3ac0
00e9028b9578e415dc1a710a2b88154447
00d7558fdaf5f1105bb213282b707729a3
047ecbe9fca55f7bd09eae36e10cae1e
00f5c86af36162f13a64f54f6dc9587c06
392a434f0e07df1f8aa305de34e0c229
3e75ced46b693021218830ae86a82a71

According to the commit, these serial numbers were added to address a
secret Mozilla bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=642395

Attempts to access this bug results in the following error:
"You are not authorized to access bug #642395. To see this bug, you must
first log in to an account with the appropriate permissions."

The full Mozilla change set list is here:
http://hg.mozilla.org/mozilla-central/rev/f6215eef2276

It appears to have been made in this changeset:

changeset:   63436:f6215eef2276
user:        Kaie Engert <kaie@kuix.de>
date:        Thu Mar 17 14:40:13 2011 -0700
summary:     Bug 642395 - change to handling of bad certificates.
r=rrelyea, bsmith. a=sayrer

This shows that Mozilla has blocklisted ten certificate serial numbers
and Google has so far blocklisted eight certificates, one of which is a
test and another that is a duplicate. Mozilla calls them bad, Google
calls it a blacklist. Interesting timing too! Looks coordinated!

The unified list of serials without uniqueness or sorting is:

047ecbe9fca55f7bd09eae36e10cae1e
d8f35f4eb7872b2dab0692e315382fb0
b0b7133ed096f9b56fae91c874bd3ac0
9239d5348f40d1695a745470e1f23f43
9239d5348f40d1695a745470e1f23f43 <-dupe
d7558fdaf5f1105bb213282b707729a3
f5c86af36162f13a64f54f6dc9587c06

009239d5348f40d1695a745470e1f23f43
00d8f35f4eb7872b2dab0692e315382fb0
72032105c50c08573d8ea5304efee8b0
00b0b7133ed096f9b56fae91c874bd3ac0
00e9028b9578e415dc1a710a2b88154447
00d7558fdaf5f1105bb213282b707729a3
047ecbe9fca55f7bd09eae36e10cae1e
00f5c86af36162f13a64f54f6dc9587c06
392a434f0e07df1f8aa305de34e0c229
3e75ced46b693021218830ae86a82a71

It's clear that there is some overlap here as we see both lists include
identical entries. In total, when we make a sorted list with unique
entries we find the following serial numbers:

009239d5348f40d1695a745470e1f23f43
00b0b7133ed096f9b56fae91c874bd3ac0
00d7558fdaf5f1105bb213282b707729a3
00d8f35f4eb7872b2dab0692e315382fb0
00e9028b9578e415dc1a710a2b88154447
00f5c86af36162f13a64f54f6dc9587c06
047ecbe9fca55f7bd09eae36e10cae1e
392a434f0e07df1f8aa305de34e0c229
3e75ced46b693021218830ae86a82a71
72032105c50c08573d8ea5304efee8b0
9239d5348f40d1695a745470e1f23f43
b0b7133ed096f9b56fae91c874bd3ac0
d7558fdaf5f1105bb213282b707729a3
d8f35f4eb7872b2dab0692e315382fb0
f5c86af36162f13a64f54f6dc9587c06

This gives us fifteen serial numbers worthy of blocking between both
browsers. I suspect they should unify their list. Of course, this brings
us to the next question - Why would Mozilla have one list and Google has
another?

As a slight conjecture tangent: I'm betting on a split disclosure -
Mozilla was probably given a list that is important to them and Google
was likely in the same position.

This returns me to the reason for creating crlwatch yesterday - I want
to find someone who knowingly revoked those special certificates.
Because anyone looking lacks the leaf certificates in question, we have
to look in a round about manner. Thanks to crlwatch, I now have a nearly
canonical list of all CRLs. I parsed the CRLs into human readable text
and searched for the above serial numbers.

This was the result:
Looking for 009239d5348f40d1695a745470e1f23f43 in parsed CRLs...
Looking for 00b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
Looking for 00d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
Looking for 00d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
Looking for 00e9028b9578e415dc1a710a2b88154447 in parsed CRLs...
Looking for 00f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
Looking for 047ecbe9fca55f7bd09eae36e10cae1e in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
047ECBE9FCA55F7BD09EAE36E10CAE1E
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
047ECBE9FCA55F7BD09EAE36E10CAE1E
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
047ECBE9FCA55F7BD09EAE36E10CAE1E
Looking for 392a434f0e07df1f8aa305de34e0c229 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
392A434F0E07DF1F8AA305DE34E0C229
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
392A434F0E07DF1F8AA305DE34E0C229
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
392A434F0E07DF1F8AA305DE34E0C229
Looking for 3e75ced46b693021218830ae86a82a71 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
3E75CED46B693021218830AE86A82A71
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
3E75CED46B693021218830AE86A82A71
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
3E75CED46B693021218830AE86A82A71
Looking for 72032105c50c08573d8ea5304efee8b0 in parsed CRLs...
Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
9239D5348F40D1695A745470E1F23F43
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
9239D5348F40D1695A745470E1F23F43
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
9239D5348F40D1695A745470E1F23F43
Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
B0B7133ED096F9B56FAE91C874BD3AC0
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
B0B7133ED096F9B56FAE91C874BD3AC0
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
B0B7133ED096F9B56FAE91C874BD3AC0
Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
D7558FDAF5F1105BB213282B707729A3
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
D7558FDAF5F1105BB213282B707729A3
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
D7558FDAF5F1105BB213282B707729A3
Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
D8F35F4EB7872B2DAB0692E315382FB0
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
D8F35F4EB7872B2DAB0692E315382FB0
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
D8F35F4EB7872B2DAB0692E315382FB0
Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
F5C86AF36162F13A64F54F6DC9587C06
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
F5C86AF36162F13A64F54F6DC9587C06
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
F5C86AF36162F13A64F54F6DC9587C06

Huzzah! It appears that we've found a few matches!

Here are the three matching files:
https://github.com/ioerror/crlwatch/blob/master/crl-parsed/2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt
https://github.com/ioerror/crlwatch/blob/master/crl-parsed/2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt
https://github.com/ioerror/crlwatch/blob/master/crl-parsed/5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt

Matching entries in the list look like this:

    Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
        Revocation Date: Mar 15 20:15:38 2011 GMT

An interesting note is that this date is a bit earlier than the above
patches! The CA knew to revoke it and patches were worked into software
a a few days later. I wonder how long the CA knew before they revoked it?

All three of the CRLs in question belong to the same CA:

issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST
Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware

This CA url is here:
http://www.usertrust.com/about.html

This appears to be a reseller or something similar for Comodo when you
inspect the trust chain for their site:
CN = COMODO High Assurance Secure Server CA

We appear to have no initial matches for the following serials from the
data that I gathered earlier this morning:

009239d5348f40d1695a745470e1f23f43
00b0b7133ed096f9b56fae91c874bd3ac0
00d7558fdaf5f1105bb213282b707729a3
00d8f35f4eb7872b2dab0692e315382fb0
00e9028b9578e415dc1a710a2b88154447
00f5c86af36162f13a64f54f6dc9587c06

Those serial numbers appear to not match, right? Nope. Mozilla appears
to have a really serious bug in their secret bug patching process. If
you remove the prefix of 00 from those serial numbers and search for the
sixteen byte values, we find what we'd expect:

Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
9239D5348F40D1695A745470E1F23F43
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
9239D5348F40D1695A745470E1F23F43
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
9239D5348F40D1695A745470E1F23F43
Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
B0B7133ED096F9B56FAE91C874BD3AC0
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
B0B7133ED096F9B56FAE91C874BD3AC0
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
B0B7133ED096F9B56FAE91C874BD3AC0
Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
D7558FDAF5F1105BB213282B707729A3
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
D7558FDAF5F1105BB213282B707729A3
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
D7558FDAF5F1105BB213282B707729A3
Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
D8F35F4EB7872B2DAB0692E315382FB0
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
D8F35F4EB7872B2DAB0692E315382FB0
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
D8F35F4EB7872B2DAB0692E315382FB0
Looking for e9028b9578e415dc1a710a2b88154447 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
E9028B9578E415DC1A710A2B88154447
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
E9028B9578E415DC1A710A2B88154447
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
E9028B9578E415DC1A710A2B88154447
Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt:    Serial Number:
F5C86AF36162F13A64F54F6DC9587C06
2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt:    Serial Number:
F5C86AF36162F13A64F54F6DC9587C06
5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt:    Serial Number:
F5C86AF36162F13A64F54F6DC9587C06

Here's sample from one of those CRLs:
    Serial Number: D7558FDAF5F1105BB213282B707729A3
        Revocation Date: Mar 15 20:15:26 2011 GMT

It looks like Mozilla made a rather weird way of structuring their
blocklisting serial struct!

Ironically, the Mozilla patch also leaks the CA name:
http://hg.mozilla.org/mozilla-central/rev/f6215eef2276#l1.57

I think that this is evidence of a rather serious event. If I had to
make a bet, I'd wager that an attacker was able to issue high value
certificates, probably by compromising USERTRUST in some manner, this
was discovered sometime before the revocation date, the certificate was
revoked, the vendors notified, the patches were shipped, and binary
builds kicked off - people are probably still updating and thus many
people are vulnerable to the failure that is the CRL and OCSP method for
revocation. Even after they update, I'm guessing users are in trouble because the list isn't the same. Mozilla should force OCSP verification by default as part of their next release, I think.

I've cc'ed Comodo, Google, and Mozilla for comment - I sure would love
to get their side of the story here. It sounds like people are at risk
and while everyone looks like they're trying their best, it looks like
we REALLY need to find a better way to deal with issuing certificates, leaf certificate revocation, and provide a more transparent method for alerting people to these kinds of serious mishaps.

All the best,
Jacob

Reproducible: Always




I think this is eligible for a bug bounty.
Comment 1 Jacob Appelbaum 2011-03-18 18:05:01 PDT
I forgot to mention that if you remove the 00 prefix, sort and only list unique elements, the list is actually ten certificates long:
047ecbe9fca55f7bd09eae36e10cae1e
392a434f0e07df1f8aa305de34e0c229
3e75ced46b693021218830ae86a82a71
72032105c50c08573d8ea5304efee8b0
9239d5348f40d1695a745470e1f23f43
b0b7133ed096f9b56fae91c874bd3ac0
d7558fdaf5f1105bb213282b707729a3
d8f35f4eb7872b2dab0692e315382fb0
e9028b9578e415dc1a710a2b88154447
f5c86af36162f13a64f54f6dc9587c06
Comment 2 Boris Zbarsky [:bz] (still a bit busy) 2011-03-18 18:15:24 PDT
Jacob, what's the bug you're reporting on our end?  We appreciate the heads-up on your planned mail, obviously!
Comment 3 chris hofmann 2011-03-18 18:33:04 PDT
I'll suggest that we change the title and make this bug become:

> Mozilla should force OCSP verification by default as part of their next 
> release, I think.

and start the investigation on how we might pull that off in firefox 5.
Comment 4 Jacob Appelbaum 2011-03-18 18:36:01 PDT
I'm on the phone with Dan at the moment, so perhaps he'll add more to this bug report.

Basically the bug is that Comodo is doing split disclosures and is compromised. I guess all of their certs should be marked as invalid for the given three CRLs I listed. Also, that the lists of certs isn't the same between Mozilla and chrome.
Comment 5 Boris Zbarsky [:bz] (still a bit busy) 2011-03-18 18:40:35 PDT
> Basically the bug is that Comodo is doing split disclosures

I don't think they are.  For example, every single entry in Chrome's list is in our list.  It does look like Chrome needs to update their list some more...

Past that, I'll defer to Dan.
Comment 6 Daniel Veditz [:dveditz] 2011-03-18 20:28:45 PDT
We also requested a test cert and received
  72:03:21:05:C5:0C:08:57:3D:8E:A5:30:4E:FE:E8:B0

Ignoring the test certs it looks like Google and Mozilla were given the same initial list, at least if you assume the Chrome duplicate is a mistake and was intended to be e9:02:8b:95:78:e4:15:dc:1a:71:0a:2b:88:15:44:47

The two serial numbers from the second Mozilla patch were not sent to us until after the Chrome update was released. Presumably Google also have these SNs and will issue a future update.

The strange initial null byte stripping in our patch is explained by Bob Relyea in bug 642395 comment 41, which I'll repeat here since that bug is inaccessible at the moment

   Serial numbers are unsigned integers. DER INTS are all signed, so when
   you encode an unsigned integer with DER you need to make sure the leading
   bit is not zero. This CA issues serial numbers which are 16 bytes long.
   If the lead byte starts with a zero (0x00-0x7f) then the serial number is
   16 bytes long, If the lead starts with a 1 (0x80-0xff) then a zero is
   added as padding.

   Different uses of these values tend to decode the resulting bytes
   differently, so it's usually best to canonicalize the bytes before
   comparing them. the easiest way of doing that is to strip the leading bytes.

You might ask why hardcode the leading 00 into our table of SNs if we're just going to strip it? The table was generated from the raw certs using a pre-existing tool that encodes with an initial null. Given the haste of the release it seemed best to leave the tool alone and not risk hand editing, and then re-hand-editing when we got the two new certs and regenerated the table.

Although our update servers are not amongst the certs in question, anyone in a position to deploy those certs could easily block updates if they became aware we were addressing the issue. I would be grateful if you could wait to go public until users have a chance to get the update next week.
Comment 7 Jacob Appelbaum 2011-03-18 21:42:12 PDT
Ok. It's good to understand why the certs aren't in sync - I've been in touch with Google and they're appraised of this issue. It's bad to know that they're not in sync because the attack had some ability to generate new certs after the first set of certs was disclosed. I'm glad to hear that it may not be a split disclosure but it sure seems like Comodo should get an internet death sentence for this specific SSL reseller. This CA can't be trusted unless they can prove the key material wasn't compromised. That seems like a REALLY hard thing to prove given what I've been able to uncover.

I won't go public until next week but I strongly urge you guys to enable mandatory OCSP *warning* at the very least before you ship the next Firefox. This is a security *disaster* if the certs issued are really what I believe them to be - it would allow for remote code execution on all Firefox clients when they query for an update. All an attacker has to do is reject OCSP/CRL queries with a HTTP 500 and MITM any attempts to upgrade any extensions. This will lead to native code execution, no?

Read this blog post to understand the full ramifications:
http://www.imperialviolet.org/2011/03/18/revocation.html

I've said for *years* that relying on SSL alone for package and code verification in Firefox is not adequate. It seems like this event is a prime example of a near worst case scenario - if the attacker has specific people in mind, they're absolutely screwed. Mozilla needs to address this and not just by covering for a few certificates that Comodo has stated are compromised.
Comment 8 Daniel Veditz [:dveditz] 2011-03-18 23:12:27 PDT
(In reply to comment #7)
> This is a security *disaster* if the certs issued are really what I believe
> them to be - it would allow for remote code execution on all Firefox clients
> when they query for an update.

The certs in question do not include the update servers for Firefox or extensions (aus2|aus3.mozilla.org and versioncheck.addons.mozilla.org)

> This will lead to native code execution, no?

If they had gotten the right certs; in this case we appear to have gotten lucky. Firefox 4 is safer: you'd have to obtain a spoofed intermediate cert in addition to the leaf cert. Not yet good enough.
Comment 9 Jacob Appelbaum 2011-03-18 23:48:14 PDT
(In reply to comment #8)
> (In reply to comment #7)
> > This is a security *disaster* if the certs issued are really what I believe
> > them to be - it would allow for remote code execution on all Firefox clients
> > when they query for an update.
> 
> The certs in question do not include the update servers for Firefox or
> extensions (aus2|aus3.mozilla.org and versioncheck.addons.mozilla.org)
> 

Oh? I thought they did get AMO?

> > This will lead to native code execution, no?
> 
> If they had gotten the right certs; in this case we appear to have gotten
> lucky. Firefox 4 is safer: you'd have to obtain a spoofed intermediate cert in
> addition to the leaf cert. Not yet good enough.

That's better, I agree. Still, there's a window of time now that these certs are useful.

It seems like a seriously bad time for you guys to push out a new version of Firefox if you're not entirely sure of what they have, how they may use it, and what the impact will be. I'm guessing a lot of traffic will come your way and an attacker will be able to use those certs to their advantage in a big way.

At least in the short term, I really think enabling OCSP to required is the minimum safe thing to do before downloading new code. It's a work around that will fail closed and while it's a bad thing, it seems like the best out of all of a bunch of bad choices. The only other option I really think might make a difference is to alert the world what certs are being looked for - not just the serial number but the entire cert chain, etc
Comment 10 Jacob Appelbaum 2011-03-18 23:55:42 PDT
(In reply to comment #5)
> > Basically the bug is that Comodo is doing split disclosures
> 
> I don't think they are.  For example, every single entry in Chrome's list is in
> our list.  It does look like Chrome needs to update their list some more...
> 
> Past that, I'll defer to Dan.

I think I agree now. After talking with Google - it's clear that there was some lag. I think that there is something really seriously fishy going on with Comodo's CA though - they have many other revocations for the same day that are not in either list.

I want to give Comodo credit - they could have swept this under the rug but I also think that the rest of the world needs to understand what happened. There are lots of other browsers out in the wild who won't have a blocklist of serials and they're all in trouble.
Comment 11 Gervase Markham [:gerv] 2011-03-19 00:15:03 PDT
(BTW, congratulations on some good detective work :-)

You assume that the attacker has a continued ability to issue certs; Comodo assure us that this is not the case. The reason two patches were required was that they forgot (!) to tell us about two of the certs until we had spun up builds once already. [You can draw your own conclusions as to how pleased we are about that.]

I have been impressing upon Comodo the need to go public at the right moment, and explain what happened in detail - for all the reasons you give. Your work means I think you deserve to know more. One of their RAs was broken into (electronically, I understand) and their creds used to issue these certs. Because of the value of the targets, a post-issuance business intelligence ("we have a big new customer!") style check spotted them. That RAs account is now (of course) shut down, and all issuances have been scrutinized.

As to your larger points, based on this event I want to start a discussion within Mozilla about where we see the future of internet identity verification going. But the time for that is not right now.

(In reply to comment #9) 
> Oh? I thought they did get AMO?

Your confusion is, fortunately, theirs :-) AMO != AUS. addons.mozilla.org (addons service) != aus.mozilla.org (update service). And there are extra protections around aus that we added in Firefox 4 where it checks the issuer, which means they could not compromise AUS even if they had a cert for it.

> At least in the short term, I really think enabling OCSP to required is the
> minimum safe thing to do before downloading new code. 

I strongly suspect we will look at that again but, given that the channel of bad certs is shut, the patch we have achieves the same effect with lesser fallout.

Gerv
Comment 12 Daniel Veditz [:dveditz] 2011-03-19 01:31:26 PDT
addons.mozilla.org is the website where you go to browse and download new extensions.

versioncheck.addons.mozilla.org is where you check for addon updates, different cert. You can't even try a social engineering "if it's broken just add an exception" hack because the update checks do not allow you to override bad certs.

Client updates ping aus2.mozilla.org in older versions, and aus3.mozilla.org in Firefox 4. Same deal as the extension update checks: bad cert overrides are not allowed.

> there are extra protections around aus that we added in Firefox 4 where
> it checks the issuer, which means they could not compromise AUS even if
> they had a cert for it.

True, but if you obtained a signing cert you could create both a fake issuer and fake leaf cert. We need to sign the contents of the update response and not just hack around it.

Argh, we're not doing this for add-on update checks. That blows. You still need only one bogus cert to compromise anyone with an addon (luckily, a different cert than the ones we know were compromised). We estimate between 1/3 and 1/2 of our users have an addon installed so that's just about as good as getting the app-update cert.

To repeat my initial point though, neither the application update server nor the addon update server were among the certs Comodo told us about.
Comment 13 Jacob Appelbaum 2011-03-19 11:45:58 PDT
(In reply to comment #11)
> (BTW, congratulations on some good detective work :-)
> 
> You assume that the attacker has a continued ability to issue certs; Comodo
> assure us that this is not the case. The reason two patches were required was
> that they forgot (!) to tell us about two of the certs until we had spun up
> builds once already. [You can draw your own conclusions as to how pleased we
> are about that.]

Yes. I assumed that there was a reason that Comodo would send two batches was either that Comodo or UTN was further compromised, perhaps Comodo made a mistake, perhaps Comodo was incompetent, or worse. Your explanation seems to point to well meaning incompetence at the least. :-)

> I have been impressing upon Comodo the need to go public at the right moment,
> and explain what happened in detail - for all the reasons you give.

I've informed the EFF about this issue, they've also agreed to keep quiet. It appear that from the observatory data we need to know _exactly_ what signed these keys as UTN is reasonably large:

CA: C=US, ST=UT, L=Salt Lake City, O=The USERTRUST Network, OU=http://www.usertrust.com, CN=UTN-USERFirst-Hardware
  50 total sub-CAs, 43 immediate
  205794 total leaves, 85440 immediate

Were these certs signed by a sub-CA as I assume? If so, what the key for signing stored in hardware?

> Your work
> means I think you deserve to know more. One of their RAs was broken into
> (electronically, I understand) and their creds used to issue these certs.
> Because of the value of the targets, a post-issuance business intelligence ("we
> have a big new customer!") style check spotted them. That RAs account is now
> (of course) shut down, and all issuances have been scrutinized.
> 

All of them for all time? What prevents the other accounts on the same system from doing this same thing?

> As to your larger points, based on this event I want to start a discussion
> within Mozilla about where we see the future of internet identity verification
> going. But the time for that is not right now.

Yes, I agree. I think this will probably happen on the sslobservatory list run by the EFF very soon. I'm guessing Tuesday.

> 
> (In reply to comment #9) 
> > Oh? I thought they did get AMO?
> 
> Your confusion is, fortunately, theirs :-) AMO != AUS. addons.mozilla.org
> (addons service) != aus.mozilla.org (update service). And there are extra
> protections around aus that we added in Firefox 4 where it checks the issuer,
> which means they could not compromise AUS even if they had a cert for it.
> 

Theirs? The attacker? Oh the hilarity.

> > At least in the short term, I really think enabling OCSP to required is the
> > minimum safe thing to do before downloading new code. 
> 
> I strongly suspect we will look at that again but, given that the channel of
> bad certs is shut, the patch we have achieves the same effect with lesser
> fallout.

One channel of bad certs that we know about is purportedly shut. The entire CA model is a disaster for privacy, security, confidentiality and so on.

I'm glad that Comodo did the right thing but I don't think that it absolves their business model of being a losing direction for the internet in the long run.
Comment 14 Jacob Appelbaum 2011-03-19 11:50:51 PDT
(In reply to comment #12)
> addons.mozilla.org is the website where you go to browse and download new
> extensions.

Right. Having control of this allows for code execution.

> 
> versioncheck.addons.mozilla.org is where you check for addon updates, different
> cert. You can't even try a social engineering "if it's broken just add an
> exception" hack because the update checks do not allow you to override bad
> certs.
> 

OK. Unless there's a bug in that ssl code too. Here's hoping.

> Client updates ping aus2.mozilla.org in older versions, and aus3.mozilla.org in
> Firefox 4. Same deal as the extension update checks: bad cert overrides are not
> allowed.

OK.

> 
> > there are extra protections around aus that we added in Firefox 4 where
> > it checks the issuer, which means they could not compromise AUS even if
> > they had a cert for it.
> 
> True, but if you obtained a signing cert you could create both a fake issuer
> and fake leaf cert. We need to sign the contents of the update response and not
> just hack around it.
> 

I agree.

> Argh, we're not doing this for add-on update checks. That blows. You still need
> only one bogus cert to compromise anyone with an addon (luckily, a different
> cert than the ones we know were compromised). We estimate between 1/3 and 1/2
> of our users have an addon installed so that's just about as good as getting
> the app-update cert.
> 

Right.

> To repeat my initial point though, neither the application update server nor
> the addon update server were among the certs Comodo told us about.

Ok - to be clearer: what was the FQDN list that you know of?
Comment 15 Jacob Appelbaum 2011-03-19 11:55:13 PDT
Some more stats on the following CA from the EFF SSL Observatory dataset:

CA: C=US, ST=UT, L=Salt Lake City, O=The USERTRUST Network,
OU=http://www.usertrust.com, CN=UTN-USERFirst-Hardware
  50 total sub-CAs, 43 immediate
  205794 total leaves, 85440 immediate

85,440 leaf certs were signed by that CA's key directly
~115,000 leaf certs were signed by intermediate CAs below it 

I'm not a fan of capital punishment in meatspace but I'd like to advocate for an internet death sentence for the specific signing CA. Unless we find extremely compelling evidence that demonstrates they could not have had a key compromise, we cannot assume that things are hunky dory.
Comment 16 Jacob Appelbaum 2011-03-19 12:57:35 PDT
(In reply to comment #11)
> > At least in the short term, I really think enabling OCSP to required is the
> > minimum safe thing to do before downloading new code. 
> 
> I strongly suspect we will look at that again but, given that the channel of
> bad certs is shut, the patch we have achieves the same effect with lesser
> fallout.
> 

I wanted to really drive home a strong counter point. If we assume that Comodo is totally honest and has made no mistake (doubtful), I still think OCSP should be required (it's already enabled) because there are so many CAs that may also revoke silently. If revocation actually worked as it stands today, you guys wouldn't need to hard code serials into the binary.
Comment 17 Gervase Markham [:gerv] 2011-03-21 02:56:51 PDT
(In reply to comment #16)
> If revocation actually worked as it stands today, you guys
> wouldn't need to hard code serials into the binary.

I can only say that this is now the starting point for my own thinking on the matter.

You are also right that Comodo could have just revoked silently, and we would have never known. So them coming forward does them credit.

Gerv
Comment 18 Jacob Appelbaum 2011-03-22 01:40:36 PDT
I'm writing a guest blog post that will go live on the EFF website tomorrow - I'd like to get a few things together - one of which is the list of (CN) names for the certs - will you guys please disclose them?
Comment 19 Daniel Veditz [:dveditz] 2011-03-22 10:37:32 PDT
We will if Comodo doesn't, but they've asked us not to for the time being because other vendors are affected who may not be able to update as quickly. There were a total of six domain names plus one that makes no sense. The two serial numbers added later were reissues of one of the original set of certs.

I will confirm that 'addons.mozilla.org' was one of the certs. This is Not Good but could not be used to hijack add-on updates, only visits to the web site itself.

The odd one no one can make any sense of was "global trustee" with a space in it. Don't know where such a cert would be valid. We did find a globaltrustee.com in Florida, but the cert issued would not work to MITM that site.
Comment 20 Gervase Markham [:gerv] 2011-03-22 12:04:54 PDT
Jacob: we have just had a message from Comodo to say they have word from Microsoft that they intend to release a mitigation package tomorrow (Wednesday) at 10am PDT.

They further say:

"We are wondering if you [Mozilla] would consider delaying an announcement so we can all release together, tomorrow.  This way, we can provide a coordinated response with as much mitigation for this issue as possible.

They feel, and we agree, that this engenders the concept of community-based defence and best serves our respective customers.

Regards

Robin Alden  M.Sc.  MIET
CTO -- Comodo"

We are content to wait until then. Are you?

Gerv
Comment 21 Jacob Appelbaum 2011-03-22 12:17:29 PDT
(In reply to comment #19)
> We will if Comodo doesn't, but they've asked us not to for the time being
> because other vendors are affected who may not be able to update as quickly.
> There were a total of six domain names plus one that makes no sense. The two
> serial numbers added later were reissues of one of the original set of certs.
> 
> I will confirm that 'addons.mozilla.org' was one of the certs. This is Not Good
> but could not be used to hijack add-on updates, only visits to the web site
> itself.
> 

Great. I'll include that information in the blog post. Thank you very much Dan.

> The odd one no one can make any sense of was "global trustee" with a space in
> it. Don't know where such a cert would be valid. We did find a
> globaltrustee.com in Florida, but the cert issued would not work to MITM that
> site.

That's an odd one, I agree. Can the certs be used for code signing? I know that a friend disabled the CA and lost the ability to validate S/MIME - I suspect that the issues here run very deep.
Comment 22 Jacob Appelbaum 2011-03-22 12:37:49 PDT
(In reply to comment #20)
> Jacob: we have just had a message from Comodo to say they have word from
> Microsoft that they intend to release a mitigation package tomorrow (Wednesday)
> at 10am PDT.
> 
> They further say:
> 
> "We are wondering if you [Mozilla] would consider delaying an announcement so
> we can all release together, tomorrow.  This way, we can provide a coordinated
> response with as much mitigation for this issue as possible.

This is not a normal attack. Disclosure does not allow anyone else to perform this attack - only the attacker with the certificate is able to take advantage of this situation. Only the attacker will benefit from a delay.

> 
> They feel, and we agree, that this engenders the concept of community-based
> defence and best serves our respective customers.

End users are not customers. What is being done to protect them right now? Nothing unless they upgrade. This is really unacceptable as a solution.

This kind of attack will happen again and again. Downloading a new binary isn't going to cut it. What happens when the attacker gets certificates for the Mozilla update process? The users will then be absolutely screwed without question.

> 
> We are content to wait until then. Are you?
> 

Yes, I will wait for a fixed, specific amount of time. 10AM PST tomorrow is when the blog post will go live. I will not agree to any further delays.
Comment 23 Jacob Appelbaum 2011-03-22 12:44:22 PDT
 Sorry to be a pain in your collective asses but it looks like someone visiting https://addons.mozilla.org/en-US/firefox/addon/imtranslator/ will certainly install code (https://addons.mozilla.org/firefox/downloads/latest/2257/addon-2257-latest.xpi?src=addondetail) from the attacker who controls addons.mozilla.org.

Is Mozilla really putting all of their users at risk to benefit Comodo and Microsoft?

Why isn't Mozilla telling the entire world to enable OCSP and pushing for a security update? You guys can mitigate this without fully disclosing and the specific targeted users are totally in trouble because you guys have not done this.
Comment 24 christian 2011-03-22 13:31:42 PDT
Jacob, we have not determined if we are waiting for Microsoft and up until Gerv's comment above the potential to do so wasn't even mentioned...please don't jump to conclusions.
Comment 25 Jacob Appelbaum 2011-03-22 13:39:11 PDT
Have you guys released any information on the topic? Or pushed for mitigation solutions? If not, I'm correct until you guys do so. If so, you're right, I'm mistaken.
Comment 26 Johnathan Nightingale [:johnath] 2011-03-22 16:38:37 PDT
Hi Jacob,

We are pushing fixes for this as part of the 4.0 launch and, now that we've confirmed our servers won't be DoS'd on launch day (a risk we've encountered during prior launches) we're pushing it live to 3.6 and 3.5.

We will absolutely NOT put our users at risk to benefit anyone else's schedule.

The planning around out-of-band security releases like this is often kept to a small group, and through no fault of his own, Gerv was accidentally left off that list on this issue, which meant his request about holding off was made without knowledge of the release plan.

When we release an out of band fix like this that may have security impact for users, we typically accompany it, same day, with a post on blog.mozilla.org/security, and we have such a post queued for this release as well. It is currently slated to go live at 5pm PDT. This doesn't match Comodo/Microsoft's request, but much as that saddens me, our principle obligation is to our users, and that includes informing them about risks as soon as practical. As you anticipate, the post doesn't include specifics about the certificates involved since that needlessly jeopardizes users of unprotected software.

NOW:
 - The software updates are already being put on the wire, no discussion of blog posts gets in their way.
 - The blog post itself was slated to go at 5pm PDT, but given the confusion in this bug and your restraint to date, I'm okay to delay it a small amount, if only to give you and others on this bug a chance to react to the comment I'm posting here. It will go live tonight, though.
 - The blog post is quite matter of fact and describes the issue and the status of supported Firefox versions. It explicitly does not preclude further action on our part, a question we are still very seriously discussing, and are likely to start public discussions on soon. I hope that you will be a part of those discussions.

Sorry for the confusion here. Mozilla has to own the mixed messages we've sent and we'll do better in future. But I want to be absolutely clear that at no time was the topic of delaying a fix to our users for the sake of external stakeholders even a matter for discussion.

If you want to talk more about this, I've sent you an email privately with my cell phone number.
Comment 27 Jacob Appelbaum 2011-03-22 17:17:42 PDT
I'm happy to post my blog post in a few hours rather than waiting until tomorrow morning. I'll give you a ring shortly.
Comment 28 christian 2011-03-22 17:21:31 PDT
FWIW the updates hit a bit of a snag, we're looking at having them available and announcing a couple hours or so from now (obviously later than 5pm PDT).
Comment 29 Daniel Veditz [:dveditz] 2011-03-22 18:33:40 PDT
(In reply to comment #21)
> That's an odd one, I agree. Can the certs be used for code signing? I know that
> a friend disabled the CA and lost the ability to validate S/MIME - I suspect
> that the issues here run very deep.

The issued certs were strictly SSL Server certs. Not code signing, not S/MIME, not certificate signing certs.

The CA root itself is trusted to issue S/MIME and object signing certs in addition to SSL certs. Disabling the root disables any leaf certs that chain to it for whatever purpose.

(In reply to comment #22)
> This is not a normal attack. Disclosure does not allow anyone else to perform
> this attack - only the attacker with the certificate is able to take advantage
> of this situation. Only the attacker will benefit from a delay.

Not necessarily. If the attacker is content with the thought that blocking OCSP/CRL on the network is enough to keep the attack going they may not have thought to block browser updates. If they're smart they already thought about that. If they're breathing they'll catch on if there's a big todo about the issue and how people need to get updates. Fine for the people who got updated before the attacker caught on, sucks for the rest of them.

It's anyone's guess whether this is one of the smart attackers or not, or how much attention they're paying, or whether the attack has been deployed somewhere or is just in the planning stages.

> End users are not customers. What is being done to protect them right now?
> Nothing unless they upgrade. This is really unacceptable as a solution.

Right, and we're trying to get those updates out ASAP. Even Microsoft is pushing a _way_ out of cycle update tomorrow and you know how rare that is even when 0-day attacks are in the wild -- we're all taking this quite seriously.

Blogging/tweeting about it isn't going to get the updates out any faster. Even if picked up by the tech press isn't going to reach that big an audience compared to the number of users on the internet. It's far more likely, in fact, to be noticed by an attacker paying attention than be noticed by most users. If that attacker is not already blocking updates that would be bad. If they are already blocking updates then it doesn't matter because most people still won't see our message to turn on OCSP.

> This kind of attack will happen again and again. Downloading a new binary
> isn't going to cut it. What happens when the attacker gets certificates for
> the Mozilla update process? The users will then be absolutely screwed
> without question.

This is painfully obvious to us. We have several things we need to change in future releases. Not changes that can be rushed out in a security fix.

(In reply to comment #23)
> it looks like someone visiting [AMO] will certainly install code ...
> from the attacker who controls addons.mozilla.org.

Yes, said that in comment 12

> Is Mozilla really putting all of their users at risk to benefit Comodo and
> Microsoft?

No, a quick, quiet, update serves our users better than potentially alerting the attacker while only helping a tiny fraction that will hear the message. Or at least that's my belief, and it has no reference to other vendors. I don't believe a delay helps Comodo in any way: the thrashing will commence whether it starts now or tomorrow or next week.
Comment 30 Jacob Appelbaum 2011-03-22 20:27:03 PDT
(In reply to comment #29)
> (In reply to comment #21)
> > That's an odd one, I agree. Can the certs be used for code signing? I know that
> > a friend disabled the CA and lost the ability to validate S/MIME - I suspect
> > that the issues here run very deep.
> 
> The issued certs were strictly SSL Server certs. Not code signing, not S/MIME,
> not certificate signing certs.

That's good to know - do we know if the private key was compromised? Or how the certs were issued with such certainty of compromise?

> 
> The CA root itself is trusted to issue S/MIME and object signing certs in
> addition to SSL certs. Disabling the root disables any leaf certs that chain
> to it for whatever purpose.
> 

Of course.

> (In reply to comment #22)
> > This is not a normal attack. Disclosure does not allow anyone else to perform
> > this attack - only the attacker with the certificate is able to take advantage
> > of this situation. Only the attacker will benefit from a delay.
> 
> Not necessarily. If the attacker is content with the thought that blocking
> OCSP/CRL on the network is enough to keep the attack going they may not have
> thought to block browser updates. If they're smart they already thought about
> that. If they're breathing they'll catch on if there's a big todo about the
> issue and how people need to get updates. Fine for the people who got updated
> before the attacker caught on, sucks for the rest of them.
> 

I don't buy this argument at all. The attacker owned a real live CA and they're not smart enough to block updates? What do you know about this attacker that presents themselves as both a state level adversary and a total, absolutely untalented idiot? :-)

> It's anyone's guess whether this is one of the smart attackers or not, or how
> much attention they're paying, or whether the attack has been deployed
> somewhere or is just in the planning stages.

I think the evidence leans in the direction of a skilled and probably aware attacker.

> 
> > End users are not customers. What is being done to protect them right now?
> > Nothing unless they upgrade. This is really unacceptable as a solution.
> 
> Right, and we're trying to get those updates out ASAP. Even Microsoft is
> pushing a _way_ out of cycle update tomorrow and you know how rare that is even
> when 0-day attacks are in the wild -- we're all taking this quite seriously.
> 

I think that you are responding to this specific issue very reasonably. Chrome has already pushed their main update.

There's a meta issue here though and I think actually Firefox has majorly dropped the ball here.  After our rogue CA attack, Firefox should have taken steps towards fixing their CA policy. After Moxie's attack on OCSP, it should have been a priority for Firefox to create a way to ensure that users would be protected against his technique. After the entire CA model has shown time and time again to be security nightmare, Firefox should have led the way. That's what the internet expects from Mozilla and it's what I expect from Mozilla. You have some incredibly brilliant people and a metric ton of cash on hand.

So enough with the half measures. It's time for everyone to work on new directions for trust metrics and we can stop this crazy cycle. We have to stop playing because everyone loses when we stay on this exploit-o-round.

> Blogging/tweeting about it isn't going to get the updates out any faster. Even
> if picked up by the tech press isn't going to reach that big an audience
> compared to the number of users on the internet. It's far more likely, in fact,
> to be noticed by an attacker paying attention than be noticed by most users. If
> that attacker is not already blocking updates that would be bad. If they are
> already blocking updates then it doesn't matter because most people still won't
> see our message to turn on OCSP.
> 

This is a seriously difficult issue and I agree that the incentives are all kinds of strange. So I propose a counter question to you - what have you done to ensure that all re-branders, all people who ship a Firefox, all other browsers, are notified of this compromise?

The Tor Browser Bundle will be updated because I discovered it. Are other people who use entirely other web browsers so lucky? I think not.

This is why telling the world is an important step. This is also the only way that a specifically targeted has any chance at all of hearing that something could be amiss at all.

If you have the certificates, Mozilla should publish them. Send them to the EFF observatory mailing list - do something with them so that the internet can make the choices that you're currently not giving them.

> > This kind of attack will happen again and again. Downloading a new binary
> > isn't going to cut it. What happens when the attacker gets certificates for
> > the Mozilla update process? The users will then be absolutely screwed
> > without question.
> 
> This is painfully obvious to us. We have several things we need to change in
> future releases. Not changes that can be rushed out in a security fix.
> 

So I'm sorry to harp about this as you and I have literally had this discussion for *years* but... I think it's worth clarifying: The future is now.

It's time for Mozilla to make a public commitment to this seemingly difficult task. What is the plan for Mozilla? Tor's plan moving forward is with a peer reviewed update system we're calling Thandy: https://gitweb.torproject.org/thandy.git

I really endorse that you guys dedicate some serious security engineering time to this, produce a road map and hopefully we can solve it together. Many free software projects are in a similar position and while Thandy solves most of the updating problems, it doesn't solve your specific deployment issues. I also really endorse you read the work by Justin Cappos (and many others) from the University of Washington: https://www.updateframework.com/browser/specs/tuf-spec.txt

> (In reply to comment #23)
> > it looks like someone visiting [AMO] will certainly install code ...
> > from the attacker who controls addons.mozilla.org.
> 
> Yes, said that in comment 12

Ok. I wanted to confirm that the updates server wasn't the only way to get code execution. There is no question now that the attacker has this ability and there is no way for the user to know that this is happening.

> 
> > Is Mozilla really putting all of their users at risk to benefit Comodo and
> > Microsoft?
> 
> No, a quick, quiet, update serves our users better than potentially alerting
> the attacker while only helping a tiny fraction that will hear the message. Or
> at least that's my belief, and it has no reference to other vendors. I don't
> believe a delay helps Comodo in any way: the thrashing will commence whether it
> starts now or tomorrow or next week.

I don't know how we measure this except by the data that I've seen so far. So I'll just have to voice my dissent based on the extremely limited ability of anyone other than the attacker to exploit this and the ability of every person to protect themselves.
Comment 32 Jacob Appelbaum 2011-03-22 20:41:43 PDT
(In reply to comment #31)
> http://blog.mozilla.com/security/2011/03/22/firefox-blocking-fraudulent-certificates/

Thank you for keeping me in the loop. I'll publish something soon and update this bug with the blog url when I do.
Comment 33 Jacob Appelbaum 2011-03-22 21:58:51 PDT
I've posted my previously embargoed blog post here:
https://blog.torproject.org/blog/detecting-certificate-authority-compromises-and-web-browser-collusion
Comment 34 Jacob Appelbaum 2011-03-23 11:21:49 PDT
Can you guys please make this a public bug?
Comment 35 Daniel Veditz [:dveditz] 2011-03-23 11:57:23 PDT
The issue is now public and discussion should take place in more appropriate fora. We have product changes we'd like to make (OCSP improvements, for example) but those are tracked in separate bugs. I'm resolving this bug because it's hard to extract specific "next actions" that lead to a fix that aren't already covered in other bugs.

From comment 0:
> I think this is eligible for a bug bounty.

Unfortunately it is not: we knew about (and were working to fix) the issue before you reported it.
Comment 36 Gervase Markham [:gerv] 2011-03-25 08:45:51 PDT
Mozilla has made a more detailed statement about the Comodo misissuance incident:
http://blog.mozilla.com/security/2011/03/25/comodo-certificate-issue-follow-up/

Please continue discussion in the mozilla.dev.security.policy discussion forum:
https://www.mozilla.org/about/forums/#dev-security-policy

Gerv

Note You need to log in before you can comment on or make changes to this bug.