1414039 - Let's Encrypt: Attacker-controlled google.tg certificate being used in the wild.

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

Received from Adam Langley:
~~
Dear all,

Google has become aware of a certificate (https://crt.sh/?id=245397170) for google.tg and some subdomains of google.tg. This certificate was issued by Let's Encrypt after a compromise of the .tg registry, Google was not involved in this issuance and the private key is not controlled by Google. Additionally, we have reason to believe that this certificate is being used in the wild.

Chrome would never have accepted this certificate due to public-key pinning and we have also blocked the SPKI by emergency push in case other attacker certificates share that public key.

We are sending a similar alert to other browsers.

Comment 1

[Gervase Markham [:gerv]]

Seems like an obvious job for OneCRL?

Gerv

Comment 2

[Dana Keeler (she/her) [:keeler]]

Agreed. However, I believe our pinning implementation would also block this:

google.tg uses the Google roots pinset:
https://dxr.mozilla.org/mozilla-central/rev/cd7217cf05a2332a8fd7b498767a07b2c31ea657/security/manager/ssl/StaticHPKPins.h#980
which I don't belive includes the Let's Encrypt hierarchy:
https://dxr.mozilla.org/mozilla-central/rev/cd7217cf05a2332a8fd7b498767a07b2c31ea657/security/manager/ssl/StaticHPKPins.h#378
(unless there's a cross-sign?)

Comment 3

[Kathleen Wilson]

CCADB indicates two "Let's Encrypt Authority X3" certs:

-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFjTCCA3WgAwIBAgIRANOxciY0IzLc9AUoUSrsnGowDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTYxMDA2MTU0MzU1
WhcNMjExMDA2MTU0MzU1WjBKMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDEjMCEGA1UEAxMaTGV0J3MgRW5jcnlwdCBBdXRob3JpdHkgWDMwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCc0wzwWuUuR7dyXTeDs2hjMOrX
NSYZJeG9vjXxcJIvt7hLQQWrqZ41CFjssSrEaIcLo+N15Obzp2JxunmBYB/XkZqf
89B4Z3HIaQ6Vkc/+5pnpYDxIzH7KTXcSJJ1HG1rrueweNwAcnKx7pwXqzkrrvUHl
Npi5y/1tPJZo3yMqQpAMhnRnyH+lmrhSYRQTP2XpgofL2/oOVvaGifOFP5eGr7Dc
Gu9rDZUWfcQroGWymQQ2dYBrrErzG5BJeC+ilk8qICUpBMZ0wNAxzY8xOJUWuqgz
uEPxsR/DMH+ieTETPS02+OP88jNquTkxxa/EjQ0dZBYzqvqEKbbUC8DYfcOTAgMB
AAGjggFnMIIBYzAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADBU
BgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEBATAwMC4GCCsGAQUFBwIB
FiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQub3JnMB0GA1UdDgQWBBSo
SmpjBH3duubRObemRWXv86jsoTAzBgNVHR8ELDAqMCigJqAkhiJodHRwOi8vY3Js
LnJvb3QteDEubGV0c2VuY3J5cHQub3JnMHIGCCsGAQUFBwEBBGYwZDAwBggrBgEF
BQcwAYYkaHR0cDovL29jc3Aucm9vdC14MS5sZXRzZW5jcnlwdC5vcmcvMDAGCCsG
AQUFBzAChiRodHRwOi8vY2VydC5yb290LXgxLmxldHNlbmNyeXB0Lm9yZy8wHwYD
VR0jBBgwFoAUebRZ5nu25eQBc4AIiMgaWPbpm24wDQYJKoZIhvcNAQELBQADggIB
ABnPdSA0LTqmRf/Q1eaM2jLonG4bQdEnqOJQ8nCqxOeTRrToEKtwT++36gTSlBGx
A/5dut82jJQ2jxN8RI8L9QFXrWi4xXnA2EqA10yjHiR6H9cj6MFiOnb5In1eWsRM
UM2v3e9tNsCAgBukPHAg1lQh07rvFKm/Bz9BCjaxorALINUfZ9DD64j2igLIxle2
DPxW8dI/F2loHMjXZjqG8RkqZUdoxtID5+90FgsGIfkMpqgRS05f4zPbCEHqCXl1
eO5HyELTgcVlLXXQDgAWnRzut1hFJeczY1tjQQno6f6s+nMydLN26WuU4s3UYvOu
OsUxRlJu7TSRHqDC3lSE5XggVkzdaPkuKGQbGpny+01/47hfXXNB7HntWNZ6N2Vw
p7G6OfY+YQrZwIaQmhrIqJZuigsrbe3W+gdn5ykE9+Ky0VgVUsfxo52mwFYs1JKY
2PGDuWx8M6DlS6qQkvHaRUo0FMd8TsSlbF0/v965qGFKhSDeQoMpYnwcmQilRh/0
ayLThlHLN81gSkJjVrPI0Y8xCVPB4twb1PFUd2fPM3sA1tJ83sZ5v8vgFv2yofKR
PB0t6JzUA81mSqM3kxl5e+IZwhYAyO0OTg3/fs8HqGTNKd9BqoUwSRBzp06JMg5b
rUCGwbCUDI0mxadJ3Bz4WxR6fyNpBK2yAinWEsikxqEt
-----END CERTIFICATE-----

Comment 4

[Dana Keeler (she/her) [:keeler]]

Thanks - I can't find any paths from those intermediates to anything in the pinset for google.tg (on crt.sh, anyway). (Although feel free to double-check, and again we probably should still put this in OneCRL.)

Comment 5

[Kathleen Wilson]

Thanks David!

Downgrading to Normal, since this bad cert should not work in FF due to pinning.

Mark and JC, please add to OneCRL at your earliest convenience.

Comment 6

[Kathleen Wilson]

Update from AGL:
~~
Please note that the set of certificates is wider than we first thought. See recent issuance in .tg.

A number of Google trademarks appear to have been targeted, as well as other companies. However, in several cases we do not own the domain in question and so this has similarities to a phishing attack. Unlike the initial certificate, we do not have evidence that these certificates have been used in the wild (but we have no evidence that they have not).
~~

Comment 7

[Kathleen Wilson]

So, when a registry such as this gets compromised, does it make sense to block all of those top-level domains? 
i.e. block *.tg

Comment 8

[J.C. Jones [:jcj] (he/they)]

(In reply to Kathleen Wilson from comment #7)
> So, when a registry such as this gets compromised, does it make sense to
> block all of those top-level domains? 
> i.e. block *.tg

Blocking *.tg would be pretty drastic. I don't think we have the technical capability to do it, either.

I'm working on the OneCRL change right now.

Comment 9

[Kathleen Wilson]

There might be more that we need to add:  https://crt.sh/?q=%25.tg

JC, would you please make sure that the Let's Encrypt folks have done something to block cert issuance to *.tg for now?

Comment 10

[J.C. Jones [:jcj] (he/they)]

(In reply to Kathleen Wilson from comment #9)
> There might be more that we need to add:  https://crt.sh/?q=%25.tg

OK... How do we choose which ones to block?

> JC, would you please make sure that the Let's Encrypt folks have done
> something to block cert issuance to *.tg for now?

LE Operations says that *.tg was blocked from production as of 13:43 PDT.

Comment 11

[Kathleen Wilson]

(In reply to J.C. Jones [:jcj] from comment #10)
> (In reply to Kathleen Wilson from comment #9)
> > There might be more that we need to add:  https://crt.sh/?q=%25.tg
> 
> OK... How do we choose which ones to block?

Top ??? sites?

> 
> > JC, would you please make sure that the Let's Encrypt folks have done
> > something to block cert issuance to *.tg for now?
> 
> LE Operations says that *.tg was blocked from production as of 13:43 PDT.

Seems like I should be sending email to all CAs to tell them to stop cert issuance to *.tg. But not sure if that would be OK to do -- i.e. is this public knowledge? Is there any reason why I should not send the email to the CAs?

Comment 12

[J.C. Jones [:jcj] (he/they)]

Attachment: bug1414039-exceptions.json.diff

Here's the first certificate. Please confirm the tooling did it right, and I'll do the same.

Comment 13

[Kathleen Wilson]

I think the "name" and "why" get permanently published. And "name" gets displayed here: https://crt.sh/mozilla-onecrl

So, maybe set both to "cert misissuance" ?

Comment 14

[J.C. Jones [:jcj] (he/they)]

Attachment: bug1414039-exceptions.json.diff

Ah, good to know. Try #2!

Comment 15

[Kathleen Wilson]

Comment on attachment 8924742 [details] [diff] [review]
bug1414039-exceptions.json.diff

Review of attachment 8924742 [details] [diff] [review]:
-----------------------------------------------------------------

Correct entry to add. Thanks!

Comment 16

[J.C. Jones [:jcj] (he/they)]

The primary cert in question was added to OneCRL in Bug 1414089 and is now live.

Comment 17

[Kathleen Wilson]

Thanks, JC! I'll verify in the OneCRL on my system tomorrow.

Update from AGL: The issuance appears to have stopped and the NS records for google.tg are correct again. I think .tg have stopped the bleeding.

So, I don't think we need to contact the CAs about this.

Comment 18

[Kathleen Wilson]

I confirm that the entry for https://crt.sh/?id=245397170 has been added to OneCRL.

Now we need to figure out what to do about other certs that were issued while the *.tg registry was compromised.

JC, would you please get the URL to the Let's Encrypt Authority X3 CRL that has the revocations for the certs that were issued while the *.tg registry was compromised?

In my opinion, there is a lot of damage that can be done during the xmas shopping season, and these certs will all have validity until the end of January. So, I suppose we need to add them to OneCRL until after January.

Also, Some of the *.tg certs were issued by other CAs, so I think I should contact those CAs to have them re-validate and let us know which ones need to be revoked/blocked.

What do you all think?

Comment 19

[J.C. Jones [:jcj] (he/they)]

(In reply to Kathleen Wilson from comment #18)
> JC, would you please get the URL to the Let's Encrypt Authority X3 CRL that
> has the revocations for the certs that were issued while the *.tg registry
> was compromised?

Let's Encrypt doesn't publish X.500 CRLs for end entities, but I will request one from the CA, or scan their OCSP responder for the data.

> In my opinion, there is a lot of damage that can be done during the xmas
> shopping season, and these certs will all have validity until the end of
> January. So, I suppose we need to add them to OneCRL until after January.

OK. I'll start that process this afternoon.

For all of these, and yesterday's as well, LE has requested we classify them as a registry compromise, not a CA misissuance. Is that OK by you?

> Also, Some of the *.tg certs were issued by other CAs, so I think I should
> contact those CAs to have them re-validate and let us know which ones need
> to be revoked/blocked.

Sounds like the safest thing to do.

Comment 20

[Kathleen Wilson]

(In reply to J.C. Jones [:jcj] from comment #19)
> For all of these, and yesterday's as well, LE has requested we classify them
> as a registry compromise, not a CA misissuance. Is that OK by you?


We'd have to be ready to explain what that means.

I will contact the owners of the *.tg registry to confirm the compromise and get the exact time frame for the compromised domains.

Comment 21

[J.C. Jones [:jcj] (he/they)]

The certificates' serials that are now revoked by Let's Encrypt for this incident all chain to the same intermediate, and they are:

030046a34666a7a291af2020bae32f5c04c8
0313e532ac640720a7dd736e60c85834c729
0315b4fae0ec7f20927e110276c186a550fc
031e899bb6a3578f6da8781ff67627cd1088
03260c82e4a8d66cb8e3d399ab9d42decdd9
0328cd438767183dc50fa58be6062b62bbbb
0332f8b4bba495e909f254a1e959d1312ae4
03404e69ff546c9c73a81b9d4b27acfdc10c
035578757d2d4dbd6b75367175672e67b611
0374d6039032971c34c7c6d5beb99448d25d
03750d4c13875246eafa4f7df6725e489745
037595cb657ed951645ad32f03a1c5c2786a
03765089b3c6499f273d56ee71c682bd47da
039a044fa5815b1ef60a894a7f4b71a16c91
03b197faccdd2bcffb29fd3152e6ab7f220d
03b442c4c7b54bd1dbec43734719749b118f
03b4f457aa38eeb8022a5de851bee31763e1
03bbb2fab9936afead0c7e1d46bb27bd7187
03c2d5e33724c5cc1db6d6d04a4d043e7a00
03c683835fc80383bc82330f64756a3c8fb0
03cc199e17ee63a5485754b01ac4c6351ecb
03d051c0ec1b5d146109efa4726825816df3
03e442418c21a1f99733efe1c5dca8533908
03e972df2d6b54fe7d93f30a7dc1370e812a
03fec31c27330673f9a94561d23176a6fc01
03ff7d6d90b34a97b160be72e9d4abc839f7
04357cf5059913d3096250a915052fe58d96
0474fa08ae81e7af66fdd77974496e04e11d
0490c79ed863a0346eb7115124f1678b16d4
04aa1bce3acec5affa9024742262a8a9a416
04aaf18b6ff5885c47105cf266f7a0c6ae42
04b9501c9eb5d5e399b9e745ac581501f02c
04e2088a9cacc40cf9c4720c98546f61c858
04e347a8b231fe26d0134f08408ca819a5e0
04e3f08f29f97aa7dea31b3b674cb7bea34d
04e735d64780f5627d476d17418f213bbca2
04e9dc5e1ec8669d52372761b547728763b6
04f56ac7851b29501b2521532b0adc16bc94

Each can be downloaded via ACME via a URL like:

https://acme-v01.api.letsencrypt.org/acme/cert/03cc199e17ee63a5485754b01ac4c6351ecb

I'm producing the OneCRL change now.

Comment 22

[Kathleen Wilson]

For Summary (name/why), how about if we use something like: "registry problems - remove in Feb 2018"?
Because I would like to remove these from OneCRL after they have expired.

Comment 23

[J.C. Jones [:jcj] (he/they)]

Attachment: git-87aaac58338a769b1ce91578880dff7c114fe7c0.diff

It looks like the other entries set "why" to "." and "name" to a more descriptive field, so here's one that does that for these.

Comment 24

[Kathleen Wilson]

Comment on attachment 8925145 [details] [diff] [review]
git-87aaac58338a769b1ce91578880dff7c114fe7c0.diff

Review of attachment 8925145 [details] [diff] [review]:
-----------------------------------------------------------------

These are the correct entries to add to OneCRL. Thanks!

Comment 25

[Kathleen Wilson]

I'm going to close this bug, because I believe we have taken appropriate action, and follow-up does not need to be tracked here.

Summary of the problem, to the best of my knowledge:

From October 25 to November 2 the *.tg registry was compromised such that a perpetrator set the NS records for domains to name servers controlled by them, and then successfully got SSL certificates for those domains. 

= Actions taken =

An entry has been added to OneCRL for the cert in the subject of this bug.

We also added entries to OneCRL for other *.tg certs that the Let's Encrypt CA revoked.

Both the Comodo and Let's Encrypt CAs have stopped DV cert issuance to *.tg domains. 

Comodo is re-validating certs that they issued to the *.tg domains.

I have sent email to GlobalSign requesting them to re-validate such certs as well.

= Future Action =

I plan to add an item to the November CA Communication to request that CAs check for SSL cert issuance to *.tg domains, and re-validate any such certs issued from October 25 to November 2 2017.

Upon request from CAs, we will consider adding additional revoked *.tg SSL certs to OneCRL.

Comment 26

[Kathleen Wilson]

Jeremy Rowley exchanged email with folks at the .tg Registry:
~~
CAS (FR): On a besoin de savoir exactement la nature de la probleme que vous avez eu.  Aussi on a besoin de savoir quand le probleme a commence et quand a ete finalisee.
CAS (EN): What was the exact nature of the problem?  Also, we need to know when the problem started and when it was resolved?

TG Registry (FR) : Nous avons eu une altération des informations des noms de domaines. Le problème a commencé le 01/11/2017. Il a été réglé et confirmé comme tel le 10/11/2017.
TG Registry (EN): Alteration of domain name information were made. The problem started on 1 Nov 2017. We confirmed the problem was resolved 10 Nov 2017.
~~

However, we have email from another CA saying:
~~
"We see a history of applications for <something>.gouv.tg certificates which we had been rejecting and only in the last week or so have we been issuing them - which would support the notion of the .tg registry being compromised."
"Looks to me like 25th October was the transition to chaos.  That is when we issued the first bad <something>.gouv.tg certificates, anyway."

https://crt.sh/?q=%25.gouv.tg only shows certificates that have been issued and logged to CT.  It doesn't show you certificates that haven't been logged, and it certainly doesn't show you "applications...which we had been rejecting" (because in these cases no certificates were issued!) 
~~

Date Problem at Registry Started: October 25, 2017
Based on the evidence from CAs, it looks like the problem started on October 25. I'm guessing that November 1 date from the Registry was when they received notification of the problem.

Date Problem at Registry Ended; November 10, 2017
Based on information from Google, I had thought the problem was resolved on November 1, but based on the information from the Registry, it sounds like they believe they resolved the problem on November 10.

Comment 27

[Kathleen Wilson]

Received today from the Let's Encrypt CA:
~~
Based on the email below, which confirms both compromise and
remediation of .tg registry systems, we will resume issuing to .tg
domains. We will monitor .tg issuance daily for a week or two in case
their fixes prove insufficient.

Aside from getting much of this information during a phone call this
morning between cafe.tg and Let's Encrypt staff, this is the first
direct confirmation of a compromise and a resolution that we have
received since the incident(s) occurred around November 1. I think our
response was appropriate but it would be good to continue the
discussion (on a public forum) about how the PKI community should
handle these situations because it's going to happen again. I don't
necessarily think we need new rules (e.g. in the BRs or from root
programs), but we should at least aim to get people to think about
this kind of possibility and what a good response looks like.

Thanks,
Josh

---------- Forwarded message ----------
From: Kido NOAGBODJI <[e-mail]>
Date: Wed, Nov 15, 2017 at 12:17 PM
Subject: Re: Restoring issuance for .tg
<snip>
On the 1st of November 2017, we have noticed that the whois
information of some domains were altered.
We have then started investigation to find out that there was a flaw
in the web platform we are using for the www.nic.tg website.
This flaw has allowed the attacker to gain access to the registry database,
We have therefore patched the system by updating the web platform to
the latest version. We have reset all passwords, and review the
database architecture to enhance security.
After that, we have also contracted with SUCURI (www.sucuri.net) to
scan the server for eventual backdoor or other malware, which proved
to be clean.
Sucuri scan occurs every 12 hours. and since then the website has been
removed from all blacklist forum.
We have also restored all the domain information from previsous backup
to the right information (whois and ns).

I understand that letsencrypt has suspended all certificate issuance
to .tg domains.
We sincerely believe that the adopted measure have secured the
information system.
We hope that you will be able to start issuing .tg certificate again.
For additional security check, if needed, the registry information can
be verified with us through phone or email.
We are open to other suggestion that can reassure you about the
security of the platform and on the reliability of the domain owner
information

Thank you,

Kind Regards

Kido NOAGBODJI
COO/CTO CAFE Informatique & Télécom
~~