698753 - Entrust SubCA: 512-bit key issuance and other CPS violations; malware in the wild

Status: VERIFIED FIXED

(NSS :: CA Certificates Code, task)

Description

[Gervase Markham [:gerv]]

I just had a phone call from Tim Moses of Entrust. They have a subCA in Malaysia which is (confusingly) called DigiCert, even though it is entirely unrelated to the mainstream DigiCert CA who live at digicert.com. (Please make this very clear in any public communications about this issue. Their full name is DigiCert Sdn. Bhd. - I suggest that in any public-facing communication we use something like "the Malaysian company DigiCert Sdn. Bhd.") 

http://www.digicert.com.my/

According to Entrust, they are fairly well known in the region, having several government customers, including the central bank.

Entrust has discovered that this subCA has been operating in contravention of a) their contract with Entrust, b) their own CPS, and c) CA good practice, in at least 3 ways:

1) They have issued at least 12 certificates, which are currently valid, using 512-bit 
   RSA keys.
2) Even though Entrust's agreement covers only SSL, many, if not all, of their certs have 
   no EKU (so can be used for anything, including code signing)
3) There are no revocation pointers of any kind in many (probably all) of their 
   certificates.

This issue came to light because the private key for (at least) one of the 512bit keys has been obtained by an attacker (probably by reverse-engineering; 512bit RSA is no longer secure), and used to sign malware. This malware was then used in a (noticed) spear-phishing attack on the Asia Pacific office of another CA.

There is an issue here with how they managed to pass their audit (which said they were doing all the things they weren't) and who their auditors are, but that's a question for another day.

Entrust is planning to revoke their cross-cert, perhaps within an hour, but there is also a possibility they may give them 2 days grace to migrate their bigger and internal government customers very quickly to another solution.

Entrust tell us that Verizon also has a cross-cert with DigiCert Sdn. Bhd, but the exact relationships here are not yet clear. I hope Steve Medin will be able to enlighten us.

Our question for now is: what do we do in the way of security updates/revocations, if anything?

Gerv

Comment 1

[Gervase Markham [:gerv]]

As far as I know, Firefox still accepts 512-bit keys. Bug 134735 and bug 360126 are about stopping accepting them, but those bugs have got very little traction. So we can assume that any reverse-engineered 512-bit keys would be accepted by Firefox, Thunderbird et. al.

Given that, even if we do a special release, our turnaround time is > 2 days, it's not worth doing anything special with the 512-bit certs themselves. Our measures would focus on the intermediate(s) cross-signed by other CAs. I will attach the Entrust certificate when I have been sent a copy.

Gerv

Comment 2

[Steven Medin]

This afternoon, Verizon Cybertrust Security will revoke the following subordinate CA certificate under our GTE CyberTrust Global Root:

Serial number:  07 27 14 a9
Common Name: Digisign Server ID (Enrich)
Organization: Digicert Sdn. Bhd.
Organizational Unit:  457608-K
Country: MY
Issue date: 7/17/2007
Expiry date: 7/17/2012

This subordinate CA remained active pursuant to surviving termination clauses in our agreement to allow orderly migration to Entrust.  We are now informed that it violates criteria for continued operation.  I will post an update once we publish a CRL reflecting our action.

Comment 3

[Sid Stamm [:geekboy or :sstamm]]

I think it would be a good idea to fix bug 360126 on regular release schedule, but soon if possible.  Chrome is thinking of doing the same thing (https://bugzilla.mozilla.org/show_bug.cgi?id=360126#c1).

Comment 4

[Robert Relyea]

Can we get actual copies of the revoked certificates (both the CyberTrust and the Entrust certs) posted into this bug?

bob

Comment 5

[Adam Langley]

Here's the CyberTrust intermediate:

-----BEGIN CERTIFICATE-----
MIIDyzCCAzSgAwIBAgIEBycUqTANBgkqhkiG9w0BAQUFADB1MQswCQYDVQQGEwJV
UzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQLEx5HVEUgQ3liZXJU
cnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0IEds
b2JhbCBSb290MB4XDTA3MDcxNzE1MTc0OFoXDTEyMDcxNzE1MTY1NFowYzELMAkG
A1UEBhMCTVkxGzAZBgNVBAoTEkRpZ2ljZXJ0IFNkbi4gQmhkLjERMA8GA1UECxMI
NDU3NjA4LUsxJDAiBgNVBAMTG0RpZ2lzaWduIFNlcnZlciBJRCAoRW5yaWNoKTCB
nzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEArahkS02Hx4RZufuQRqCmicDx/tXa
VII3DZkrRSYK6Fawf8qo9I5HhAGCKeOzarWR8/uVhbxyqGToCkCcxfRxrnt7agfq
kBRPjYmvlKuyBtQCanuYH1m5Os1U+iDfsioK6bjdaZDAKdNO0JftZszFGUkGf/pe
LHx7hRsyQt97lSUCAwEAAaOCAXgwggF0MBIGA1UdEwEB/wQIMAYBAf8CAQAwXAYD
VR0gBFUwUzBIBgkrBgEEAbE+AQAwOzA5BggrBgEFBQcCARYtaHR0cDovL2N5YmVy
dHJ1c3Qub21uaXJvb3QuY29tL3JlcG9zaXRvcnkuY2ZtMAcGBWCDSgEBMA4GA1Ud
DwEB/wQEAwIB5jCBiQYDVR0jBIGBMH+heaR3MHUxCzAJBgNVBAYTAlVTMRgwFgYD
VQQKEw9HVEUgQ29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJlclRydXN0IFNv
bHV0aW9ucywgSW5jLjEjMCEGA1UEAxMaR1RFIEN5YmVyVHJ1c3QgR2xvYmFsIFJv
b3SCAgGlMEUGA1UdHwQ+MDwwOqA4oDaGNGh0dHA6Ly93d3cucHVibGljLXRydXN0
LmNvbS9jZ2ktYmluL0NSTC8yMDE4L2NkcC5jcmwwHQYDVR0OBBYEFMYWk04WF+wW
royUdvOGbcV0boR3MA0GCSqGSIb3DQEBBQUAA4GBAHYAe6Z4K2Ydjl42xqSOBfIj
knyTZ9P0wAp9iy3Z6tVvGvPhSilaIoRNUC9LDPL/hcJ7VdREgr5trGeOvLQfkpxR
gBoU9m6rYYgLrRx/90tQUdZlG6ZHcRVesHHzNRTyN71jyNXwk1o0X9g96F33xR7A
5c8fhiSpPAdmzcHSNmNZ
-----END CERTIFICATE-----

Comment 6

[Adam Langley]

Entrust intermediate (I pulled this one from SSL Observatory data):

-----BEGIN CERTIFICATE-----
MIIEzjCCA7agAwIBAgIETA5jajANBgkqhkiG9w0BAQUFADCBtDEUMBIGA1UEChML
RW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5lbnRydXN0Lm5ldC9DUFNfMjA0OCBp
bmNvcnAuIGJ5IHJlZi4gKGxpbWl0cyBsaWFiLikxJTAjBgNVBAsTHChjKSAxOTk5
IEVudHJ1c3QubmV0IExpbWl0ZWQxMzAxBgNVBAMTKkVudHJ1c3QubmV0IENlcnRp
ZmljYXRpb24gQXV0aG9yaXR5ICgyMDQ4KTAeFw0xMDA3MTYxNzIzMzdaFw0xNTA3
MTYxNzUzMzdaMGUxCzAJBgNVBAYTAk1ZMRswGQYDVQQKExJEaWdpY2VydCBTZG4u
IEJoZC4xETAPBgNVBAsTCDQ1NzYwOC1LMSYwJAYDVQQDEx1EaWdpc2lnbiBTZXJ2
ZXIgSUQgLSAoRW5yaWNoKTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AMWJ5PQNBkCSWccaszXRDkwqM/n4r8qef+65p21g9FTob9Wb8xtjMQRoctE0Foy0
FyyX3nPF2JAVoBor9cuzSIZE8B2ITM5BQhrv9Qze/kDaOSD3BlU6ap1GwdJvpbLI
Vz4po5zg6YV3ZuiYpyR+vsBZIOVEb7ZX2L7OwmV3WMZhQdF0BMh/SULFcqlyFu6M
3RJdtErU0a9Qt9iqdXZorT5dqjBtYairEFs+E78z4K9EnTgiW+9ML6ZxJhUmyiiM
2fqOjqmiFDXimySItPR/hZ2DTwehthSQNsQ0HI0mYW0Tb3i+6I8nx0uElqOGaAwj
vgvsjJQAqQSKE5D334VsDLECAwEAAaOCATQwggEwMA4GA1UdDwEB/wQEAwIBBjAS
BgNVHRMBAf8ECDAGAQH/AgEAMCcGA1UdJQQgMB4GCCsGAQUFBwMBBggrBgEFBQcD
AgYIKwYBBQUHAwQwMwYIKwYBBQUHAQEEJzAlMCMGCCsGAQUFBzABhhdodHRwOi8v
b2NzcC5lbnRydXN0Lm5ldDBEBgNVHSAEPTA7MDkGBWCDSgEBMDAwLgYIKwYBBQUH
AgEWImh0dHA6Ly93d3cuZGlnaWNlcnQuY29tLm15L2Nwcy5odG0wMgYDVR0fBCsw
KTAnoCWgI4YhaHR0cDovL2NybC5lbnRydXN0Lm5ldC8yMDQ4Y2EuY3JsMBEGA1Ud
DgQKBAhMTswlKAMpgTAfBgNVHSMEGDAWgBRV5IHREYC+2Im5CKMx+aEkCRa5cDAN
BgkqhkiG9w0BAQUFAAOCAQEAl0zvSjpJrHL8MCBrtClbp8WVBJD5MtXChWreA6E3
+YkAsFqsVX7bQzX/yQH4Ub7MJsrIaqTEVD4mHucMo82XZ5TdpkLrXM2POXlrM3kh
Bnn6gkQVmczBtznTRmJ8snDrb84gqj4Zt+l0gpy0pUtNYQA35IfS8hQ6ZHy4qXth
4JMi59WfPkfmNnagU9gAAzoPtTP+lsrT0oI6Lt3XSOHkp2nMHOmZSufKcEXXCwcO
mnUb0C+Sb/akB8O9HEumhLZ9qJqp0qcp8QtXaR6XVybsK0Os1EWDBQDp4/BGQAf6
6rFRc5Mcpd1TETfIKqcVJx20qsx/qjEw/LhFn0gJ7RDixQ==
-----END CERTIFICATE-----

Comment 7

[Steven Medin]

Verizon Cybertrust Security confirms accuracy of @5 above.

Comment 8

[Gervase Markham [:gerv]]

I've emailed release-drivers to ask about our options here. The risks of the 512-bit crack are that:

1) The attacker can impersonate the website(s) the cert(s) were originally issued to
2) The attacker can code-sign anything (due to lack of EKU)
3) The attacker can't do anything relevant related to email

1) is a problem, but it's not one on the scale of DigiNotar's "we have certs for google.com and other properties used by tens of millions of people each day out there in the wild".

2) Could be a problem. What exactly does Firefox do with code-signing? Java uses its own cert store IIRC, as does the Adobe PDF plugin. (I advised Entrust to contact both companies.) Addons can be signed - does that actually give them any more privs, or does it just put the person's name in the install dialog? AIUI, most addons are not signed.

Gerv

Comment 9

[Robert Relyea]

NSS currently has not minimum requirements for key size usage. There are a couple of ways we can kill 768 or smaller signatures Signatures:

1) We can start rejecting signatures in NSS:
   Checks in PK11_VerifyRecover will affect all RSA signatures, except SSL client auth/protocol usage (which uses PK11_Verify).
   Checks in secvfy.c:vfy_CreateContext can affect any signature (including DSA an ECC), except SSL clientauth/protocol usage. We can also change DecryptSigBlock() and only affect RSA signatures (like changes to PK11_VerifyRecover).
   Checks in certvfy.c:CERT_VerifySignedDataWithPublicKey() will affect signatures on certificates, crls, and I think OCSP. This is where we currently reject forbidden hash functions by policy. This would effect all signature types (EC, DSA, RSA), so we probably would want a normalized notion of key strength for these algorithms.
    Checks in the NSS verify function, we can check the key strength. This probably doesn't give us any more than the certvfy.c unless we want to distiguish between different policies, or different cert types (root certs versus intermediates).

2) Checks in Mozilla. We can check the chain returned from NSS and look at the key sizes of the various certs in the chain. This gives the finest grain checks, but is localized to mozilla. This is probably preferable to checks in the NSS verify function.

The question is how locally or broadly do we want to control this.

bob

Comment 10

[Robert Relyea]

Gerv, what kind of cert are we talking about?

An intermediate certificate means the attacker can issue any kind of certificate that does the root that signed it is privileged to issue.

A leaf cert depends on the leaf itself. IIRC, NSS will default SSL and S/MIME to certificates that don't have explicit EKU's and has the appropriate attributes (email addresses and websites). Object Signing requires a specific EKU. 

None of the checks I mentioned would affect an SSL leaf cert. RSA ssl certs don't sign (well except in the 'step-down' case, hybrid cases like DHE and ECDHE ciphers signed by RSA -- in which case it would use the PK11_Verify() function). The VerifyRecover/CreateContext proposals would affect email and object signing. I don't know if CERT_VerifySignedData() would affect object signing (It would't affect S/MIME).

bob

Comment 11

[Robert Relyea]

Attachment: Creates interference certs for these two

This patch will prevent the 'traditional' NSS cert verifier from verifying these intermediates. The certs in this patch are not the real certs, but ones that will replace the real ones in chain processing.

For NSS 3.13, we could just include the revoked certs, and use the distrust flag (which is now validated in 3.13).

I've included this patch because the released versions of mozilla are still using NSS 3.12.

bob

Comment 12

[Robert Relyea]

Attachment: Cybertrust interference cert.

Comment 13

[Robert Relyea]

Attachment: Entrust interference cert

Comment 14

[Robert Relyea]

I would feel most comfortable if Kai can look over the patches and maybe even build a ckbi release with them. I don't think he's awake right now, though.

Comment 15

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This bug shouldn't be hidden.

Comment 16

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Bob's patch does not do anything to resolve the issue with trusting 512-bit certs. That would need an additional patch.

AFAICT, the major benefit of Bob's approach is to reject these certs that are also valid for code signing of extensions. (Do we use code signing for anything else?)

We should add these CAs to the blacklist like Robert has done in his patch for Firefox 8 now instead of worrying about whether to do it later after Firefox 8.

I agree that Kai would be the best reviewer for these patches. I will try to do a supplementary review.

There are multiple ways to handle this. IMO, the safest alternative is to issue a new NSS 3.12.12 release that includes Bob's patch, then import NSS 3.12.12 into mozilla-central and change configure.in to require NSS 3.12.12.

Another alternative would be to do a NSS 3.13.2 release with Bob's patches, then import that release into mozilla-central and then update configure.in to require NSS 3.13.2. This would require us to make an additional one-line patch to disable the extra BEAST mitigation for Firefox 8, because it causes compatibility problems and we need to give times for websites to update. There is the most regression risk with this approach.

Another alternative would be to do a NSS 3.13.2 release with Bob's patches, take the patches into mozilla-central privately, and set configure.in to require NSS 3.13.2 for Linux systems configured to use system NSS.

Comment 17

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Please attach a list of all the certificates (including issuer DN, subject DN, and serial number) that Malaysian company DigiCert Sdn. Bhd has issued, obtained directly from them. Let's get that list confirmed by Entrust and Verizon.

Comment 18

[Adam Langley]

Since it appears that both root CAs which have issued intermediates to DigiCert Sdn. Bhd will be revoking them, I plan on blocking the CA rather than blocking individual certificates.

Comment 19

[Kai Engert (:KaiE:)]

I didn't see this until now.

Comment 20

[LegNeato]

Comment on attachment 571095 [details] [diff] [review]
Creates interference certs for these two

[triage comment]

For aurora we can wait until we have a review. Please renominate it when we have a r+. For beta, we are still discussing.

Is there any outstanding information that would alter the approach in this patch?

Comment 21

[Kai Engert (:KaiE:)]

Brian,

in my understanding, Bob's patch is supposed to block the intermediate CA certs, and thereby will block all old and future certificates issued by the non-conforming CA. In my understanding, this is the high priority issue that we would like to fix.

Yes, I agree that Bob's patch doesn't solve the 512-bit issue "in general" - but assuming that other CA's are conforming, blocking of smaller key sizes could be done in a secondary step in a separate bug?

I agree we should add these "knockout" certs to the blacklist, Bob, thanks for making them.

Brian we don't need a *full* NSS release, not 3.12.12, not 3.13.2

We are able to release the most recent versions of NSS in combination with just a newer roots+trust module (named nssckbi). We did that repeatedly in the past. This avoids all code changes for our desired trust update.

Such an approach is completely safe to land on all branches.

I could produce patches for both 3.12.x and 3.13.x branches that could land on top of the respective branches.

Comment 22

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Kai Engert (:kaie) from comment #21)
> in my understanding, Bob's patch is supposed to block the intermediate CA
> certs, and thereby will block all old and future certificates issued by the
> non-conforming CA. In my understanding, this is the high priority issue that
> we would like to fix.

Right.

> Yes, I agree that Bob's patch doesn't solve the 512-bit issue "in general" -
> but assuming that other CA's are conforming, blocking of smaller key sizes
> could be done in a secondary step in a separate bug?

Yes, that bug is bug 360126 and we can do it on a different schedule.

> Brian we don't need a *full* NSS release, not 3.12.12, not 3.13.2
> 
> We are able to release the most recent versions of NSS in combination with
> just a newer roots+trust module (named nssckbi). We did that repeatedly in
> the past. This avoids all code changes for our desired trust update.
> 
> Such an approach is completely safe to land on all branches.
> 
> I could produce patches for both 3.12.x and 3.13.x branches that could land
> on top of the respective branches.

This sounds great but I don't fully understand it. Do you think you and Bob could work together on the necessary changes? What can I do to help?

In what you propose here, what would happen for users in the --use-system-nss case? Don't we have to have a NSS version number bump to handle that?

Comment 23

[Kai Engert (:KaiE:)]

> This sounds great but I don't fully understand it. 


mozilla/security/nss/lib/ckbi

This is a pkcs#11 module.
It contains the roots and their trust.
It has it's own version number.

If you look at directory ftp://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/
you'll see that we re-released some NSS versions with a newer CKBI.


> Do you think you and Bob
> could work together on the necessary changes? What can I do to help?


I'm certain that Bob and I have the same understanding of what should be done.
In comment 14 he also talked building a newer CKBI.


> In what you propose here, what would happen for users in the
> --use-system-nss case? Don't we have to have a NSS version number bump to
> handle that?

That shouldn't be a problem for anyone.

Linux packages include more than just the library version - they also contain a suffix that indicates a package build/release version. They simply need to increase that.

Comment 24

[Kai Engert (:KaiE:)]

The knockout certs created by Bob have negative serial numbers.
I'm worried it might cause issues?
I just talked to Bob, he thinks it shouldn't cause issues, but he agrees if I make a newer patch.

Comment 25

[Steven Medin]

@2 revocation completed, new CRL posted.

Comment 26

[Kai Engert (:KaiE:)]

Comment on attachment 571095 [details] [diff] [review]
Creates interference certs for these two

These knockout certs aren't faked to be self-signed.

We did that for DigiNotar. If I remember correctly, we did that to enable users to manually configure site overrides, if necessary.

(Because if we don't fake them to be self signed, they will be reported with "invalid signature" and no override will be offered.)

Comment 27

[Robert Relyea]

Kai is quite correct. I purposefully chose the 'invalid signature' route because it gives the same semantics as revocation. I don't think we have the same fail over issues we had with DigiNotar, but Mozilla drivers should decide which there preference should be (more non-mozilla products it doesn't matter how we do it, since they don't have the same override controls as mozilla does).

Kai is also correct that these knock out the entire intermediate, not just the improperly issued certificates.

bob

Comment 28

[Robert Relyea]

Oh, I'm also OK with kai tweeking the serial numbers to something non-negative, just as long as we don't pick something that would colide with a legitimately issued certificate from either Entrust or Digicert.

bob

Comment 29

[Robert Relyea]

err change DigiCert to CyberTrust above.

bob

Comment 30

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Robert Relyea from comment #27)
> Kai is quite correct. I purposefully chose the 'invalid signature' route
> because it gives the same semantics as revocation. I don't think we have the
> same fail over issues we had with DigiNotar, but Mozilla drivers should
> decide which there preference should be.

I agree with Kai and Robert that we shouldn't make these errors user-overridable, since the CAs have revoked the intermediates, and we don't let users override revocation errors.

Comment 31

[Kai Engert (:KaiE:)]

(In reply to Robert Relyea from comment #28)
> Oh, I'm also OK with kai tweeking the serial numbers to something
> non-negative, just as long as we don't pick something that would colide with
> a legitimately issued certificate from either Entrust or Digicert.

That would be the advantage of making fake-self-signed certs - there is no risk for serial number collision.

Comment 32

[Kai Engert (:KaiE:)]

In my opinion, the benefit of avoiding potential conflicts with real certs from CyberTrust and Entrust is a strong argument for using the fake-self-signed certs.

I've created them.

Here are diffs of the dumps compared with the ones I've created:

Bob, does this look good?



--- org-cybertrust.txt  2011-11-01 23:11:18.836967275 +0100
+++ cyb-new.txt 2011-11-02 00:29:11.862769563 +0100
@@ -1,12 +1,12 @@
 Certificate:
     Data:
         Version: 3 (0x2)
-        Serial Number: 120001705 (0x72714a9)
+        Serial Number: 268435455 (0xfffffff)
         Signature Algorithm: sha1WithRSAEncryption
-        Issuer: C=US, O=GTE Corporation, OU=GTE CyberTrust Solutions, Inc., CN=GTE CyberTrust Global Root
+        Issuer: C=MY, O=Digicert Sdn. Bhd., OU=457608-K, CN=Digisign Server ID (Enrich)
         Validity
-            Not Before: Jul 17 15:17:48 2007 GMT
-            Not After : Jul 17 15:16:54 2012 GMT
+            Not Before: Jul 17 15:17:49 2007 GMT
+            Not After : Jul 17 15:16:55 2012 GMT
         Subject: C=MY, O=Digicert Sdn. Bhd., OU=457608-K, CN=Digisign Server ID (Enrich)
         Subject Public Key Info:
             Public Key Algorithm: rsaEncryption


--- org-entrust.txt     2011-11-01 23:11:27.294883741 +0100
+++ en-new.txt  2011-11-02 00:29:02.335863600 +0100
@@ -1,12 +1,12 @@
 Certificate:
     Data:
         Version: 3 (0x2)
-        Serial Number: 1276011370 (0x4c0e636a)
+        Serial Number: 2147483647 (0x7fffffff)
         Signature Algorithm: sha1WithRSAEncryption
-        Issuer: O=Entrust.net, OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.), OU=(c) 1999 Entrust.net Limited, CN=Entrust.net Certification Authority (2048)
+        Issuer: C=MY, O=Digicert Sdn. Bhd., OU=457608-K, CN=Digisign Server ID - (Enrich)
         Validity
-            Not Before: Jul 16 17:23:37 2010 GMT
-            Not After : Jul 16 17:53:37 2015 GMT
+            Not Before: Jul 16 17:23:38 2010 GMT
+            Not After : Jul 16 17:53:38 2015 GMT
         Subject: C=MY, O=Digicert Sdn. Bhd., OU=457608-K, CN=Digisign Server ID - (Enrich)
         Subject Public Key Info:
             Public Key Algorithm: rsaEncryption

Comment 33

[Kai Engert (:KaiE:)]

I proposed to add these two certs with the following nicknames in the roots module (analog to what we did for DigiNotar):

Explicitly Distrusted Malaysian Digicert Sdn. Bhd. (cyb)
Explicitly Distrusted Malaysian Digicert Sdn. Bhd. (en)

This addresses the request to make it clear which CA this refers to, and I've also included a hints about the intermediate issuers (cyb/en) without raising false assumptions about them.

Comment 34

[Kai Engert (:KaiE:)]

Attachment: Fake selfsigned knockout for Cybertrust issused intermediate

Comment 35

[Kai Engert (:KaiE:)]

Attachment: Fake selfsigned knockout for Entrust issused intermediate

Comment 36

[Kai Engert (:KaiE:)]

Do we have URLs for SSL sites that use related certs?
(If you don't want to post in public, please send to me by private email, for testing purposes.)

Thanks

Comment 37

[Kai Engert (:KaiE:)]

Attachment: patch v2 -- branches with nss.h saying 3.13.x -- run gmake generate after applying

Comment 38

[Kai Engert (:KaiE:)]

Attachment: patch v2 -- branches with nss.h saying 3.12.x -- run gmake generate after applying

Comment 39

[Kai Engert (:KaiE:)]

I would like to note:

The certs made by me are extending Bob's work.
The only differences are:
- changed issuer to the same value as the subject
- changed serial numbers to 0xfffffff or 0x7fffffff to make it easier to identify them
- adjusted the necessary encoding lengths

The certs use the same validity times that Bob had suggested.

Comment 40

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

We should get the full list from Entrust and Verizon.

Here is a list from a snapshot of the SSL observatory data. Note that many (all?) gov.my sites seem to be down.

https://securelogin.cimbbizchannel.com
https://payment.m3gps.com
https://wallgarden.izzi.com.my
https://kapas.malaysia.gov.my
https://www.bib.com.my
https://ttpm.kpdnkk.gov.my
https://www.psp.upm.edu.my
https://lfxsys.lfx.com.my
https://melati.malaysia.gov.my
https://ikp.inai.com.my
https://www.jaring.my
https://www.online.jaring.my
https://utmshare.utm.my
https://newss.statistics.gov.my
https://www.traffic.jaring.my
https://cnii.cybersecurity.my
https://pcm.cybersecurity.my
https://www.one.jaring.my
https://www.cybersecurity.my
https://est.poslaju.com.my
https://payments.bnm.gov.my
https://raportal.gitn.net.my
https://www.innosabahonline.com.my
https://intranet.digicert.com.my
https://www.theregency.com.my
https://jksm.esyariah.gov.my
https://webmail.tunemoney.com
https://mail.jpa.gov.my
https://www.hltmtakaful4all.com.my
https://www.speks.com.my
https://e-services.tnb.com.my
https://kenari.malaysia.gov.my
https://email.mohe.gov.my
https://www.mohe.gov.my
https://www.iguarantee.com.my
https://payloan.mohe.gov.my
https://remote.ambankgroup.com
https://202.184.252.14
https://www.bsnet.com.my
https://www.stpg.com.my
https://webmail.kettha.gov.my
https://e.hasil.gov.my
https://online.mohe.gov.my/emailAddress=[e-mail], ST=PUTRAJAYA
https://www.hla.com.my
https://mail.gmi.edu.my
https://www.ccdms-websvr.akpk.org.my
https://kelisa.malaysia.gov.my
https://elatihan.hasil.gov.my
https://www.jupiteronline.com.my
https://www.pfmb.com.my
https://mylocker.uthm.edu.my
https://dccrakyat.bankrakyat.com.my
https://[e-mail]
https://epayment.hasil.gov.my
https://infokop.skm.gov.my
https://r2.bnm.gov.my
https://smpc.treasury.gov.my
https://www.malaysia.gov.my
https://www.estats.gov.my
https://www.edmsservice.com.my
https://systest.bnm.gov.my
https://ai.audit.gov.my
https://202.184.48.26
https://alert.ambg.com.my
https://sys.bnm.gov.my
https://sftp.ambg.com.my
https://spsd.hasil.gov.my
https://ef.hasil.gov.my
https://scrs-mida.digicert.com.my
https://www.amauto.ambg.com.my
https://bless-ap.digicert.com.my
https://tmpointvirtual.com.my
https://stamps.hasil.gov.my
https://emel.pos.com.my
https://intranet.kln.gov.my, ST=PutraJaya
https://www.stars.ambg.com.my
https://sys2.bnm.gov.my
https://fast.bnm.gov.my
https://iscap.bnm.gov.my
https://anjungnet.mardi.gov.my, ST=SERDANG
https://ambg2k128
https://ekls.hasil.gov.my
https://honeynet.org.my
https://www.epmfb.com.my
https://ccris.bnm.gov.my, ST=KUALA LUMPUR
https://www.agribazaar.com.my
https://mail.hasil.gov.my
https://crs2.digicert.com.my
https://autodiscover.hasil.gov.my
https://egcrs.digicert.com.my
https://smp.ukm.my
https://excsrvr.stats.gov.my
https://mcrs2.digicert.com.my
https://mcrs.digicert.com.my
https://staf-digital.upsi.edu.my
https://bnmapp.bnm.gov.my
https://www.ti-ipos.com.my
https://cbis.ambg.com.my
https://webmail.doe.gov.my
https://epintas.bnm.gov.my
https://www.posonline.com.my
https://lpkp2u.lpkp.gov.my
https://owa.wildlife.gov.my
https://www.iteps.bnm.gov.my
https://www.kompetensi.jpa.gov.my
https://tms.jpapencen.gov.my
https://tpg.jpapencen.gov.my
https://mygovxiis.malaysia.gov.my
https://www.fomema-infocentre.com.my
https://helpdesk.tga.com.my
https://satuid.treasury.gov.my
https://isgmail02.isys.com.my
https://icems.mmu.edu.my
https://www.sunway.com.my
https://owa.averis.biz
https://www.progressive-ins-jpj.com.my
https://extranet.ing.com.my
https://mail.pantai.com.my
https://dcheqs.bnm.gov.my
https://itepsinfo.bnm.gov.my
https://bnmapp.finet.bnm.gov.my
https://etrade.kaf.com.my
https://lantik.jac.gov.my
https://mail.ing.com.my
https://kijangnet.bnm.gov.my
https://www.ing.com.my
https://ccris-transfer.bnm.gov.my
https://securedr.cimbbizchannel.com
https://bicc.jkr.gov.my
https://bnmapptest.bnm.gov.my
https://www.terengganu.esyariah.gov.my
https://laporan.terengganu.esyariah.gov.my
https://eperunding.treasury.gov.my
https://www.wp.esyariah.gov.my
https://agency1.ing.com.my
https://laporan.wp.esyariah.gov.my
https://mygovxpg.malaysia.gov.my
https://www.hladr.com.my
https://choices.bhinsurance.com.my
https://sys1.bnm.gov.my
https://www.perlis.esyariah.gov.my
https://www.kedah.esyariah.gov.my
https://www.bankislam.biz
https://webcash.com.my
https://www.penang.gov.my
https://[e-mail]
https://sukaweb.penang.gov.my
https://WWW.TMEPG.COM
https://epayment.johor.gov.my
https://my
https://logon.rhb.com.my
https://matriix.mida.gov.my
https://.my
https://ebanker.bankislam.biz
https://pangkor.malaysia.gov.my
https://www.allianz.com.my
https://notes.shearndelamore.com.my
https://worksite.shearndelamore.com.my
https://mail.prokhas.com.my
https://oga.customs.gov.my
https://www.hladr2.com.my
https://ehome.kpkt.gov.my
https://www.ecsm-online.com.my
https://www.bpfk.gov.my
https://sso.ppj.gov.my
https://bawal.malaysia.gov.my
https://www1.mylofsa.gov.my
https://www.mylofsa.gov.my
https://tmlinx2005test.tm.com.my
https://tmlinx2005.tm.com.my
https://www.msebroker.com.my
https://exchange.netmyne.com
https://www.ekasih.gov.my
https://blessapp.bless.gov.my
https://www.bless.gov.my
https://mydocs.usim.edu.my
https://oaserver.smecorp.gov.my
https://www.myscm.com.my
https://www.tokiomarine.com.my
https://putra.malaysia.gov.my
https://hrms.comm.jaring.my
https://webmail.jaring.my
https://67.192.230.254

Comment 41

[Gervase Markham [:gerv]]

I think Bob is right; if the intermediate certs themselves are revoked, Firefox should behave as closely as possible to how it would behave if it were told that the certs were revoked. Ideally it would throw the same error; but if that's not possible, it should at least have the same behaviour with respect to overrides. 

So I think that means, if I read comment 27 correctly, we should go down the "invalid signature" route. And I think Kai's current patches are the "self-signed" route. If going down the "invalid signature" route means there's a possible problem with serial numbers, we can fix that:

Entrust and Verizon representatives: would you each be able to issue us with a guaranteed-unused serial number to use for our knockout certs, so we don't accidentally affect a cert you have issued?

However, if there is a deadline here related to Firefox 8, I would much rather we took the patches we have now, than nothing! Can someone on release-drivers clarify about what deadlines there are, if any? We move to release on November 8th - does that mean final builds are still not made? Do we expect to make them on the 8th, or some time before?

In terms of release mechanics, let's pick the right approach, get patches, get them through any necessary review, have the NSS team release a new NSS version with the same code but an updated CKBI on all supported branches, and import that into the various Firefox and Thunderbird trees.

bsmith: given that we are knocking out the intermediates, why do we need a full list of certs issued? I doubt that either Verizon or Entrust keeps that list in the normal course of business. I guess they may have obtained it in the last day or two... But what do we need it for? Testing, or something else?

Gerv

Comment 42

[Kai Engert (:KaiE:)]

Unfortunately my patch/certs cause confusion in NSS.
While attemping to verify the certs, NSS will NOT detect that they are self-issued.

Instead, NSS will build a (limited) recursive loop, repeatedly finding self as its issuer.
The recursion can be seen in PSM's cert viewer, in the details tab's chain view.

certvfy.c contains
        /* find the certificate of the issuer */
        issuerCert = CERT_FindCertIssuer(subjectCert, t, certUsage);

After executing this statement, the subjectCert and issuerCert pointers are identical.

I assume the source of this confusion is the "Authority Key ID" contained in the cert,
which is prefered by NSS.

This means, our usual strategy for creating fake self-signed certs fails in this scenario.

Given that we cannot create fake self-signed certs, I agree it would be wise to ask both Entrust and CyberTrust for new, unique, non-conflicting serial numbers.

Comment 43

[Kai Engert (:KaiE:)]

"Entrust and Verizon representatives: 

Would you each be able to issue us with a guaranteed-unused serial number to use for our knockout certs, so we don't accidentally affect a cert you have issued?" (as Gerv said)


Cybertrust, we'd need a new, unique serial number for this issuer name:

Issuer: C=US, O=GTE Corporation, OU=GTE CyberTrust Solutions, Inc., CN=GTE CyberTrust Global Root


Entrust, we'd need a new, unique serial number for this issuer name:

Issuer: O=Entrust.net, OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.), OU=(c) 1999 Entrust.net Limited, CN=Entrust.net Certification Authority (2048)

Comment 44

[Gervase Markham [:gerv]]

While when we open this bug is an issue for us, I have heard from Tim at Entrust, who says:

"I believe we will disclose our plan today."

He also says:

"We are working to get DigiCert Malaysia's highest profile customers equipped with an alternative before the cross-cert is revoked."

So they appear to be delaying revocation for a short time to minimise the impact on legitimate certificate users.

Gerv

Comment 45

[Steven Medin]

@43, Responding for Verizon Cybertrust Security, any serial number sized larger than 4 octets will be guaranteed unique for the remaining functional life of the GTE CyberTrust Global Root.

Comment 46

[Robert Relyea]

Kai, go ahead and just make your change to the serial number, and not use the self-signed certs.

We could get self-signed to work by removing the Auth KID extention (or adding a corresponding key id extension, but I think we agree we want the same semantics as bad signature/revocation.

bob

Comment 47

[Bruce Morton]

I am investigating and getting a guaranteed unique unused serial number. Can someone please explain in layman terms how it will be used?

Thanks, Bruce.

Comment 48

[Kai Engert (:KaiE:)]

(In reply to Bruce Morton from comment #47)
> I am investigating and getting a guaranteed unique unused serial number. Can
> someone please explain in layman terms how it will be used?

In short, we will make an untrusted fake cert, that will look similar to your cert (because it has the same subject), but will contain newer validity dates. When NSS builds a trust chain, if NSS finds both your original cert and our fake cert, it will select/prefer our untrusted fake cert, thereby causing an invalid trust chain.

We use the fake cert as a workaround, because the crypto engine in Firefox (NSS) does not yet have the technical feature to explicitly distrust your cert.

We require that original cert and fake cert have different serial numbers, because if NSS ever encounters two certs with identical issuer/serial, it will "freak out".

Comment 49

[Kai Engert (:KaiE:)]

(In reply to Robert Relyea from comment #46)
> Kai, go ahead and just make your change to the serial number, and not use
> the self-signed certs.
> 
> We could get self-signed to work by removing the Auth KID extention (or
> adding a corresponding key id extension, but I think we agree we want the
> same semantics as bad signature/revocation.


I'm OK with this plan, but I haven't succeeded yet.

Steven's request requires that I increase the portion of the asn.1 encoding that stores the serial number (from 4 to at least 5). I haven't yet figured out which other portions of the encoding need to be adjusted (in the hex editor) to keep the encoding valid.

Comment 50

[Gervase Markham [:gerv]]

Steve M: if it's reasonably easy for you to pick and reserve for us an explicit 4-byte value, is there any chance of you doing that? That reduces the risk and complexity of the change on our side.

Gerv

Comment 51

[Steven Medin]

@48, Kai, when you say that NSS does not yet have the technical feature to explicitly distrust the certificate, do you mean that our efforts to revoke and publish a new root CRL yesterday offer no benefit to user agents derived from NSS?

Comment 52

[Kai Engert (:KaiE:)]

(In reply to Kai Engert (:kaie) from comment #49)
> 
> I'm OK with this plan, but I haven't succeeded yet.

Never mind. I learned how to increase it.

Comment 53

[Kai Engert (:KaiE:)]

(In reply to Steven Medin from comment #51)
> @48, Kai, when you say that NSS does not yet have the technical feature to
> explicitly distrust the certificate, do you mean that our efforts to revoke
> and publish a new root CRL yesterday offer no benefit to user agents derived
> from NSS?

NSS is able to use CRLs, but we don't automatically download them.

So yes, because most users of NSS based software probably don't automatically download CRLs, your CRL doesn't have an effect.

That's why we are considering a software update to disable the intermediate.

Comment 54

[Gervase Markham [:gerv]]

(In reply to Steven Medin from comment #51)
> @48, Kai, when you say that NSS does not yet have the technical feature to
> explicitly distrust the certificate, do you mean that our efforts to revoke
> and publish a new root CRL yesterday offer no benefit to user agents derived
> from NSS?

What Kai means is that recently an NSS feature was implemented which allows us to import a certificate into the root store and mark it as "never trust this certificate". This is a simpler way of doing active dis-trust than our current method, which involves creating "overriding" or "knock-out" certificates with similar characteristics to the dis-trusted ones, and which NSS will prefer when attempting to construct chains. (You can see that process going on in this bug.)

This is not related to NSS's and/or Firefox's support or otherwise for revocation by various methods.

Gerv

Comment 55

[Kai Engert (:KaiE:)]

Attachment: Fake selfsigned knockout for Cybertrust issused intermediate (v3)

This cert is similar to the one from Bob.
The only difference is:
- it increases the size of the serial number from 4 bytes to 6 bytes
- it sets the serial number to 07:ff:ff:ff:ff:ff

Comment 56

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Gervase Markham [:gerv] from comment #41)
> However, if there is a deadline here related to Firefox 8, I would much
> rather we took the patches we have now, than nothing! Can someone on
> release-drivers clarify about what deadlines there are, if any? We move to
> release on November 8th - does that mean final builds are still not made? Do
> we expect to make them on the 8th, or some time before?

We were hoping to have the RC built two days ago. So, the sooner the better, at least. We should get this finished today, or tomorrow at the very latest.

IMO, the user-overridable vs. non-user-overridable issue does not matter much. Making it non-overridable is nice to have, not a mandatory requirement. An attacker could spoof a certificate that looks just like it was issued directly by Verisign and even very sophisticated users would not be able to tell the difference. Consequently, it really doesn't matter.

> In terms of release mechanics, let's pick the right approach, get patches,
> get them through any necessary review, have the NSS team release a new NSS
> version with the same code but an updated CKBI on all supported branches,
> and import that into the various Firefox and Thunderbird trees.
> 
> bsmith: given that we are knocking out the intermediates, why do we need a
> full list of certs issued? I doubt that either Verizon or Entrust keeps that
> list in the normal course of business. I guess they may have obtained it in
> the last day or two... But what do we need it for? Testing, or something
> else?

Testing and determining user impact. (Also, why wouldn't a CA know exactly what its sub-CAs are issuing? That would be irresponsible, at best.)

As for whether this bug should be closed: what criteria for being closed does it meet? We had this discussion last time during Comodogate and I thought we had agreed not to do things this way anymore. security-group isn't for protecting partners' reputations.

Comment 57

[Kai Engert (:KaiE:)]

Attachment: Fake selfsigned knockout for Entrust issused intermediate (v3)

This cert is similar to the one from Bob.
The only difference is:
- it increases the size of the serial number from 4 bytes to 6 bytes
- it sets the serial number to 07:ff:ff:ff:ff:ff

Comment 58

[Kai Engert (:KaiE:)]

Bruce, Entrust currently uses 4 byte serial numbers for certs issued by

  O=Entrust.net, OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.), 
  OU=(c) 1999 Entrust.net Limited, CN=Entrust.net Certification Authority (2048)

If 4 or 5 bytes are sufficient for the lifetime of this particular intermediate, then we could use the same (suggested) serial number for our fake/knockout/override cert:

I'd propose 07:ff:ff:ff:ff:ff
decimal: 8796093022207

That's what I've used for now, but please let us know if you want a different one, e.g. an even longer one.

Comment 59

[Kai Engert (:KaiE:)]

Attachment: patch v3 -- branches with nss.h saying 3.13.x -- run gmake generate after applying

Comment 60

[Kai Engert (:KaiE:)]

Attachment: patch v3 -- branches with nss.h saying 3.12.x -- run gmake generate after applying

Comment 61

[Kai Engert (:KaiE:)]

Technical landing instructions (for engineers adding the patches to Mozilla software).

- look at file security/nss/lib/nss/nss.h in your branch
- NSS_VERSION will either be 3.12.x or 3.13.x
- pick the correct patch
- please don't merge the patch. It must apply cleanly. If not, ping me.
- after you applied the patch:
  cd security/nss/lib/ckfw/builtins/
  gmake generate
  (this runs a perl script that will produce file certdata.c )
- done

Comment 62

[Gervase Markham [:gerv]]

bsmith: I'll try and get you that list. Although I suspect that, for testing purposes, the Observatory list is probably a bit more useful, as it's more current.

I also think it's time to open this bug. 

Gerv (about to knock off for the night)

Comment 63

[Kai Engert (:KaiE:)]

Test instructions:

step 1: verify the test case is still valid

- use unpatched software
- go to https://smp.ukm.my/
- you should see a valid page
- click the identity button to the left of the URL bar,
  click "more information",
  click "view cert"
- verify that "Issued By, Organization (O)" still says 
    "Digicert Sdn. Bhd."
- if yes, proceed to step 2. 
- if not, try to find another URL in comment 40 that still gives you the above result


step 2: verify the patch works

- use patched software
- go to the same URL from step 1
- you should see an error message with
    (Error code: sec_error_bad_signature)

Comment 64

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

We may end up not blocking the intermediate, but something less. Now, we need Entrust to give us the full set of certs that should NOT be blocked, so that we can build a whitelist. I am not sure if we will do this or not.

Comment 65

[Kai Engert (:KaiE:)]

(In reply to Brian Smith (:bsmith) from comment #64)
> We may end up not blocking the intermediate, but something less. Now, we
> need Entrust to give us the full set of certs that should NOT be blocked, so
> that we can build a whitelist. I am not sure if we will do this or not.


I hope this is not true.

This is a full opposite of what was requested in the initial comment of this bug.

If it's true, I've wasted a lot of time.

Comment 66

[Robert Relyea]

RE: NSS checking of CRLs and other revocations stuff...

NSS has 2 different cert validation routines, our old routines, and the new libpkix routines. Currently both are available in the system. Firefox generally uses the old routines, only switching to the libpkix routines when doing the secondary EV check.

Libpkix can do OCSP and dynamic CRL processing on all certs on the cert chain (depending on the policy the application requests. It can cache these requests on a session basis, but does not have any persistent cache, so the CRLs are refetch on restart.

The old cert processing does OCSP on the EE cert only, but does do CRLs on all certs, if the CRL has already been downloaded and is present. (It does not fetch CRLs on the fly).


In addition to the above, the ability to explicitly distrust certs was added in NSS 3_13. Before that we couldn't explicitly distrust intermediate certificates. One way to work around this issue is to create a 'bad' cert which looks like it was issued more recently than the original intermediate. NSS would pick up this bad cert first, then fail because it is not properly signed. That is the trick that is being done in this patch. Once 3_13 is out and in general use, we can simply just distrust the given intermediate.

bob

Comment 67

[Robert Relyea]

Kai, can you post the diff of 

pp -t certificate < {certname} 

Of the original cert and your changed cert for review. (or whatever tool you used in comment 32)

Thanks.

Comment 68

[Robert Relyea]

> We may end up not blocking the intermediate, but something less.

If the intermediate is revoked, why should there be a whitelist. If the intermediate is revoked, at least in chrome, the whole chain is already gone. I think IE is the same (though I haven't tested it).

bob

Comment 69

[Gervase Markham [:gerv]]

Attachment: Full list of certs issued by DigiCert Malaysia (source: Entrust)

Comment 70

[Gervase Markham [:gerv]]

Attachment: ZIP file of remaining valid 512-bit certs issued by Digicert Malaysia (source: Entrust)

Comment 71

[Kai Engert (:KaiE:)]

(In reply to Robert Relyea from comment #67)

Bob, here are the diffs of the plaintext dumps, between your original certs, and my latest certs:


--- bob-cybertrust.txt	2011-11-01 23:13:34.392628451 +0100
+++ cyb-big.txt	2011-11-02 19:09:44.035528231 +0100
@@ -1,7 +1,8 @@
 Certificate:
     Data:
         Version: 3 (0x2)
-        Serial Number: -2 (-0x2)
+        Serial Number:
+            07:ff:ff:ff:ff:ff
         Signature Algorithm: sha1WithRSAEncryption
         Issuer: C=US, O=GTE Corporation, OU=GTE CyberTrust Solutions, Inc., CN=GTE CyberTrust Global Root
         Validity



--- bob-entrust.txt	2011-11-01 23:13:41.446558781 +0100
+++ en-big.txt	2011-11-02 19:17:20.935999834 +0100
@@ -1,7 +1,8 @@
 Certificate:
     Data:
         Version: 3 (0x2)
-        Serial Number: -2 (-0x2)
+        Serial Number:
+            07:ff:ff:ff:ff:ff
         Signature Algorithm: sha1WithRSAEncryption
         Issuer: O=Entrust.net, OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.), OU=(c) 1999 Entrust.net Limited, CN=Entrust.net Certification Authority (2048)
         Validity

Comment 72

[Gervase Markham [:gerv]]

The just-attached spreadsheet contains details of 985 certificates issued by Digicert Malaysia. Of those, 669 have not expired as of today. Of those, 21 or 22 are marked as 512-bit.

Gerv

Comment 73

[Kai Engert (:KaiE:)]

(In reply to Robert Relyea from comment #68)
> > We may end up not blocking the intermediate, but something less.
> 
> If the intermediate is revoked, why should there be a whitelist. If the
> intermediate is revoked, at least in chrome, the whole chain is already
> gone. I think IE is the same (though I haven't tested it).


If I understand correctly, only the cross-signed cert (made by Verizon Cybertrust) has been revoked.

If I understand correctly, the Entrust issued intermediate has not yet been revoked - despite the original announcement in the initial comment in this bug.

Comment 74

[Kai Engert (:KaiE:)]

(In reply to Gervase Markham [:gerv] from comment #72)
> The just-attached spreadsheet contains details of 985 certificates issued by
> Digicert Malaysia. Of those, 669 have not expired as of today. Of those, 21
> or 22 are marked as 512-bit.

Why do we need them?
Why has the original plan to revoke the intermediate been changed?

Comment 75

[Adam Langley]

I've not heard any updates and Chrome is still planning on killing both intermediate certificates at the present time.

Comment 76

[Johnathan Nightingale [:johnath]]

(In reply to Kai Engert (:kaie) from comment #74)
> (In reply to Gervase Markham [:gerv] from comment #72)
> > The just-attached spreadsheet contains details of 985 certificates issued by
> > Digicert Malaysia. Of those, 669 have not expired as of today. Of those, 21
> > or 22 are marked as 512-bit.
> 
> Why do we need them?
> Why has the original plan to revoke the intermediate been changed?

Plans haven't changed. Contingencies are being discussed to make sure our bases are covered, and Gerv's adding information to the bug in case we need it.

Comment 77

[Bruce Morton]

(In reply to Kai Engert (:kaie) from comment #58)
> Bruce, Entrust currently uses 4 byte serial numbers for certs issued by
> 
>   O=Entrust.net, OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.), 
>   OU=(c) 1999 Entrust.net Limited, CN=Entrust.net Certification Authority
> (2048)
> 
> If 4 or 5 bytes are sufficient for the lifetime of this particular
> intermediate, then we could use the same (suggested) serial number for our
> fake/knockout/override cert:
> 
> I'd propose 07:ff:ff:ff:ff:ff
> decimal: 8796093022207
> 
> That's what I've used for now, but please let us know if you want a
> different one, e.g. an even longer one.

That works for us. We will eventually be issuing certificates with long serial numbers, but they will be much longer than 6 bytes. We will never issue a certificate with a 6 byte serial number from this CA.

Bruce.

Comment 78

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

We are going to kill the intermediate for Firefox 8. The plan:

1. I will do a local build and do some testing myself while we wait for Bob to r+ the patch for NSS 3.12.
2. I will push the r+d NSS 3.12 patch to mozilla-beta
3. After that, and after Bob r+s the patch for NSS 3.13, I will do the same for mozilla-central and mozilla-aurora.
4. We may push some other ridealong patches to mozilla-beta and then do a RC build and close mozilla-beta.

Work needed for Firefox 3.6.x is TBD.

Comment 79

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 571393 [details] [diff] [review]
patch v3 -- branches with nss.h saying 3.13.x -- run gmake generate after applying

This patch is ready for Bob to review too, right Kai?

Comment 80

[Kai Engert (:KaiE:)]

If you want formal r+ for both, sure.

But I'd say, one r+ for one patch is sufficient.
The other branch patch simply differs in syntactic sugar.

Comment 81

[Gervase Markham [:gerv]]

One additional, possibly-relevant bit of information that Entrust tell me (which I presume agl can confirm) is that Google's current plan is to block only the 512-bit certs for now.

Gerv

Comment 82

[Adam Langley]

(Was in a meeting for a while)

Entrust have emailed me 22 DER encoded certificates with 512-bit keys and have asked that only they be blocked for now, not the whole CA.

We hadn't decided to kill this CA, we were only reflecting the actions of the roots in killing it and, since there's no hint of a CA compromise, I'm not of a mind to demand that it die immediately.

So my inclination is to respect this request. If Mozilla is still planning on killing the CA, that would be interesting and we would have to reconsider our response.

Comment 83

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Adam, our thinking is that Entrust asked for one month for a transition period. Firefox 8 would ship in one week. We think that is enough time for the affected sites to get another cert. Also, I have been told that the 512-bit certs are not the only issue with this CA. (I don't know how thoroughly the summary of this bug--comment 0--completely describes all the problems, or if there are other (known by others at Mozilla, or unknown to us) issues.

Comment 84

[Kathleen Wilson]

(In reply to Brian Smith (:bsmith) from comment #83)
> Also, I have been told that the
> 512-bit certs are not the only issue with this CA. 

For example, many of the end-entity certs don't have CRL Distribution Points (or OCSP). Not just the 512-bit certs.  

See Description item #3: There are no revocation pointers of any kind in many (probably all) of their  certificates.

Comment 85

[Bruce Morton]

In addition the key size issue, our observations are that the certificates do not contain the EKU extension limiting the certificate use to SSL and do not contain the CRL distribution point extension making revocation ineffective. We do not have any evidence that the CA was compromised or that the validation was done improperly. 

As the CA has been operating outside of policy our plan is to revoke the intermediate certificate. The only question has been of timing as we understand that there will be impact to web-sites including some operated by the Malaysian government.

Comment 86

[Adam Langley]

Since we are no longer simply reflecting Entrust's decision to revoke, we have to make a decision ourselves about whether to revoke this CA. That will take some time. 

Certainly we can drop in CyberTrust's revocation as soon as this is public but I'll have to chat to people here about what to do about the Entrust intermediate.

Comment 87

[Steven Medin]

@86, Adam, is there an obstacle preventing presentation of our revocation?  We work quickly to cause these to happen, it hurts to see the Rube Goldberg we begin when we always thought we were the end of the process.

Comment 88

[LegNeato]

As Brian mentioned in comment 83, Mozilla is planning on shipping revocation to users on November 8th as we have available release vehicles. There doesn't look to be much debate internally based on the reasons Brian pointed out in comment 83 and Kathleen in comment 84 as well.

Though this is basically a rehash of comment 83, I wanted to make sure the current Mozilla plan is explicitly clear for all involved.

Comment 89

[Adam Langley]

(In reply to Steven Medin from comment #87)
> @86, Adam, is there an obstacle preventing presentation of our revocation? 

No, the CyberTrust revocation will be included as it is already effective. We're just waiting on this event going public before submitting it to the Chromium code repository.

Having discussed it, we don't see a pressing need to revoke the Entrust intermediate ahead of Entrust themselves. We'll block the 22 serial numbers provided in the mean time.

Comment 90

[Robert Relyea]

Comment on attachment 571393 [details] [diff] [review]
patch v3 -- branches with nss.h saying 3.13.x -- run gmake generate after applying

r+ rrelyea.

Comment 91

[Robert Relyea]

Comment on attachment 571396 [details] [diff] [review]
patch v3 -- branches with nss.h saying 3.12.x -- run gmake generate after applying

r+ This is the same patch I just r+'ed,

Comment 92

[Robert Relyea]

Hmm, now that I think about it, only the Entrust interference cert is needed, as it's newer than the CyberTrust. It doesn't hurt to include both, however.

bob

Comment 93

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Verified on mozilla-beta on Verizon/GTE CyberTrust-rooted site : https://utmshare.utm.my/

Verified on mozilla-beta on Entrust-root site: https://lfxsys.lfx.com.my/index.asp

Without the fix, both sites are accessible. With the fix, both sites fail to load with an "invalid signature" error.

I decided to add an extra step to this process before I push this to mozilla-beta and other repositories. Please hold off on pushing this.

Comment 94

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Shannon, as soon as I push this to mozilla-beta and the other repositories, you are going to get press inquiries about it.

Comment 95

[Al Billings [:abillings - ex-MoCo]]

Does this patch apply to 1.9.2?

Comment 96

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Al, I am trying it now.

Comment 97

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

https://hg.mozilla.org/releases/mozilla-beta/rev/935c58d51e00

Will now land the 3.6.24 patch on 1.9.2 and then the 3.13 patch on mozilla-central and mozilla-aurora.

Comment 98

[Kai Engert (:KaiE:)]

Just some clarifications for Bob:


(In reply to Robert Relyea from comment #92)
> Hmm, now that I think about it, only the Entrust interference cert is
> needed, as it's newer than the CyberTrust. It doesn't hurt to include both,
> however.

If you said this, because you assume both certs subject names are identical - that's not true. One of the certs contains an additional, easy to overlook "- " string.


(In reply to Robert Relyea from comment #91)
> Comment on attachment 571396 [details] [diff] [review] [diff] [details] [review]
> patch v3 -- branches with nss.h saying 3.12.x -- run gmake generate after
> applying
> 
> r+ This is the same patch I just r+'ed,

No, the patches are different. While they are functional equivalent, the difference is that they are using the respective tokens for certdata.txt, which are different on 3.12 and 3.13 branches.

Comment 99

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

1.9.2 default branch:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/3facffd01b4f (wrong version)
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/df871fe0c2b7 (backs out the above)
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/0c8a1be25963 (correct version)

1.9.2 GECKO19224_2011103116_RELBRANCH:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/7239e49f6d37

Working on -central and -aurora now.

Comment 100

[Ludovic Hirlimann [:Usul]]

(In reply to Kai Engert (:kaie) from comment #61)
> Technical landing instructions (for engineers adding the patches to Mozilla
> software).

Shall we cc the linux distribution maintainers ?

Comment 101

[Bruce Morton]

All 512-bit certificates have been revoked and are listed on the CRL at http://crl.digicert.com.my/Enrich.crl.

Comment 102

[Kai Engert (:KaiE:)]

Patch v3 checked in to NSS trunk (3.13.x):

Checking in certdata.c;
/cvsroot/mozilla/security/nss/lib/ckfw/builtins/certdata.c,v  <--  certdata.c
new revision: 1.83; previous revision: 1.82
done
Checking in certdata.txt;
/cvsroot/mozilla/security/nss/lib/ckfw/builtins/certdata.txt,v  <--  certdata.txt
new revision: 1.80; previous revision: 1.79
done
Checking in nssckbi.h;
/cvsroot/mozilla/security/nss/lib/ckfw/builtins/nssckbi.h,v  <--  nssckbi.h
new revision: 1.33; previous revision: 1.32
done


Patch v3 checked in to NSS 3.12 branch:

Checking in certdata.c;
/cvsroot/mozilla/security/nss/lib/ckfw/builtins/certdata.c,v  <--  certdata.c
new revision: 1.67.2.14; previous revision: 1.67.2.13
done
Checking in certdata.txt;
/cvsroot/mozilla/security/nss/lib/ckfw/builtins/certdata.txt,v  <--  certdata.txt
new revision: 1.64.2.14; previous revision: 1.64.2.13
done
Checking in nssckbi.h;
/cvsroot/mozilla/security/nss/lib/ckfw/builtins/nssckbi.h,v  <--  nssckbi.h
new revision: 1.24.2.9; previous revision: 1.24.2.8
done

Comment 103

[Bruce Morton]

The Entrust issued intermediate certificate will be revoked on or before 8 November 2011. Please see http://www.entrust.net/advisories/malaysia.htm.

Comment 104

[Vlad [QA]]

Attachment: screenshot

I have test this considering comment63.

I've followed the step1 from the comment and everything was as described on:
Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0 beta 1 
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0 beta 1
Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0 beta 1
Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20100101 Firefox/8.0 beta 1

After this, I've tryed the link on Firefox 8 beta 6 build 2 across platforms and I got the same error as described in comment63. (see screenshot)

IMO this bug is VERIFIED FIXED.

Comment 105

[Kai Engert (:KaiE:)]

In my understanding this has not yet landed into mozilla-central, 
and not yet into mozilla-aurora.

We should do that before resolving this bug.

Comment 106

[Kai Engert (:KaiE:)]

The updated NSS root module has been tagged in the NSS CVS tree.

The roots module compatible with the NSS 3.12 branch can be retrieved using tag
  NSS_3_12_CKBI_1_88_RTM

The roots module compatible with NSS 3.13 can be retrieved using tag
  NSS_3_13_CKBI_1_88_RTM

The FTP directories contain tarballs that contain combine the most recent NSS releases (3.12.11 and 3.13.1) with the new roots module version 1.88

Comment 107

[Al Billings [:abillings - ex-MoCo]]

Verified fix for 1.9.2 using https://utmshare.utm.my/.

Pre-fix, it loads fine. Post-fix, the user gets a "Secure Connection Failed" error page with:

An error occurred during a connection to utmshare.utm.my.

Peer's certificate has an invalid signature.

(Error code: sec_error_bad_signature)

Oh, and people, quit clearing the 1.9.2 flags. Thanks.

Comment 108

[Tony Chung [:tchung]]

(In reply to Al Billings [:abillings] from comment #107)
> Verified fix for 1.9.2 using https://utmshare.utm.my/.
> 
> Pre-fix, it loads fine. Post-fix, the user gets a "Secure Connection Failed"
> error page with:
> 
> An error occurred during a connection to utmshare.utm.my.
> 
> Peer's certificate has an invalid signature.
> 
> (Error code: sec_error_bad_signature)
> 
Same repro steps, verified fix against Android, Galaxy S 2:  Mozilla/5.0 (Android; Linux armv7l; rv:8.0 Gecko/201101102 Firefox/8.0 Fennec/8.0

Comment 109

[Kai Engert (:KaiE:)]

Comment on attachment 571393 [details] [diff] [review]
patch v3 -- branches with nss.h saying 3.13.x -- run gmake generate after applying

Please approve this for aurora, so we can complete the work on this task.
Thanks.

Comment 110

[Kai Engert (:KaiE:)]

Pushed to mozilla-inbound.

TODO: Aurora

Comment 111

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

The patches have already been approved to land on all branches by johnath and legneato in person. The reason this didn't land on aurora yet is because we should land NSS 3.13.1 first or at the same time so that this patch doesn't get overwritten by the NSS 3.13.1 landing. So, we are waiting for the approval of bug 669061.

Comment 112

[Kai Engert (:KaiE:)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/505216fa1c65

Comment 113

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

https://hg.mozilla.org/mozilla-central/rev/2432457fa32b

Comment 114

[Vlad [QA]]

Verified on:

Mozilla/5.0 (Windows NT 6.1; rv:9.0a2) Gecko/20111106 Firefox/9.0a2
Mozilla/5.0 (Windows NT 6.1; rv:10.0a1) Gecko/20111106 Firefox/10.0a1

Mozilla/5.0 (Windows NT 5.1; rv:9.0a2) Gecko/20111106 Firefox/9.0a2
Mozilla/5.0 (Windows NT 5.1; rv:10.0a1) Gecko/20111106 Firefox/10.0a1

Mozilla/5.0 (X11; Linux x86_64; rv:9.0a2) Gecko/20111106 Firefox/9.0a2
Mozilla/5.0 (X11; Linux x86_64; rv:10.0a1) Gecko/20111106 Firefox/10.0a1

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0a2) Gecko/20111106 Firefox/9.0a2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a1) Gecko/20111106 Firefox/10.0a1

The error is showing every time. I've tested this based on STR from comment63.
Considering comment104 and this one, setting resolution to Verified Fixed.

This is the error:
"Secure Connection Failed      
        
An error occurred during a connection to smp.ukm.my.

Peer's certificate has an invalid signature.

(Error code: sec_error_bad_signature)        

  The page you are trying to view can not be shown because the authenticity of the received data could not be verified.
  Please contact the website owners to inform them of this problem. Alternatively, use the command found in the help menu to report this broken site."

Comment 115

[Daniel Veditz [:dveditz]]

According to comment 97, comment 112 and 113 this should be fixed in Firefox 8-10

Comment 116

[LegNeato]

Comment on attachment 571393 [details] [diff] [review]
patch v3 -- branches with nss.h saying 3.13.x -- run gmake generate after applying

[triage comment]

Approved for aurora. Please land today if at all possible.

Comment 117

[Kai Engert (:KaiE:)]

(In reply to Christian Legnitto [:LegNeato] from comment #116)
> Approved for aurora. Please land today if at all possible.

already landed, see comment 112

Comment 118

[Bruce Morton]

The Digicert Malaysia cross-certificate issued by Entrust was revoked this morning. The CRL can be found at crl.entrust.net/2048ca.crl. The certificate serial number is 4C0E636A.