1270657 - Monitor the Certificate Transparency log for unexpected issuance of certificates on Mozilla domains

Status: RESOLVED FIXED

(Security Assurance :: General, task)

Description

[Julien Vehent]

Certificate transparency [1] is an immutable log containing certificates issued by certificate authorities. CAs willingly participate in CT and publish a log entry for each certificate they issue. Google is actively pushing CAs to publish their logs into CT, and provides public logs for everyone to audit [2].

While CT does not yet cover 100% of CAs, it would be beneficial to monitor those logs for mis-issuance of certificates that contain Mozilla domains. This could be done using :jcj's ct-sql tool [3], or a custom integration that feeds logs into Mozdef. Some firepower is needed as logs can get very large in size.

Links:
[1] https://www.certificate-transparency.org/
[2] https://www.certificate-transparency.org/known-logs
[3] https://github.com/jcjones/ct-sql

Comment 1

[J.C. Jones [:jcj] (he/they)]

https://github.com/tobermorytech/certificate-transparency-monitor or https://github.com/kyprizel/ct_mon may be good starting points. Better than my ct-sql anyway, which stores complete state of the CT ecosystem. :)

Comment 2

[Julien Vehent]

Loading the entire `pilot` log certificates from Google would only consume 21GB of space in database. It's small enough that we should integrate CT log monitoring in the TLS Observatory, and use the existing tlsobs-runner to send alerts.
I created an issue for it, but no one assigned to it yet: https://github.com/mozilla/tls-observatory/issues/118

Comment 3

[April King [:April]]

SSL Mate has a CT log monitor with an API:

https://sslmate.com/certspotter/

Once it is a bit more mature and they have their pricing figured out, I am inclined to use that.

Comment 4

[Gene Wood [:gene]]

I've deployed certspotter ( https://github.com/gene1wood/certspotter-cloudformation ) and am monitoring CT logs for certs issued in the domains : .mozilla.com, .mozilla.org, .mozilla.net, .firefox.com, .getfirefox.com

It checks each hour for new certs.

At the moment the notifications of new certs in these domains goes to me but I'll switch it to the infosec team in a few days after I've validated that over time there's not noise.

Since the tool just checks for new certificates since the last check it never needs to store the full CT log.

April, ulfr,
   Assuming this is switched to email the infosec team, what are your thoughts on this satisfying our need to notice rogue letsencrypt certs being issued? I don't propose that this is the final solution, only something that keeps us safe and enables us to get unblacklisted from letsencrypt while we wait for a final solution (maybe involving the observatory, using mozdef, etc)?

Hows the domain list look? Those cover those domains and all subdomains.

Comment 5

[April King [:April]]

Domain list looks pretty good to me.  I was never particularly worried about Let's Encrypt in the first place, but this is nice to have in place.

Comment 6

[Julien Vehent]

The list look good, my only concern is:

> At the moment the notifications of new certs in these domains goes to me but I'll switch it to the infosec team in a few days after I've validated that over time there's not noise.

There is going to be noise. Probably a few notifications every week as people issue certs for our hundreds of sites. What do we do with that data? How do we distinguish between a legitimate issuance and a fraudulent one?

Comment 7

[April King [:April]]

That's a good point with the noise.  Perhaps what we really want is to only monitor the really high profile names?

Assuming we get to the point where most everything is automated with Let's Encrypt.  That's a few thousand certs every 90 days, and will be very difficult for somebody to stay on top of.  If we only monitor things like AMO, BMO, Bedrock, AUS, etc., then then it's likely we'll catch something that's actually critical.

Comment 8

[Julien Vehent]

Attachment: 7 valid certificates not issued by Digicert

Do we need to alert for certificates issued by Digicert? There are 214 trusted certs for '\.(firefox|mozilla)\.(com|org|net)$' in the TLS Observatory, out of which only 7 (attached) were not issued by Digicert.

I personally don't see a point in monitoring Digicert.

Comment 9

[April King [:April]]

That seems reasonable.  Monitor the critical sites and alert whenever a cert is issued that isn't in the DigiCert CT log?

Comment 10

[Gene Wood [:gene]]

> There is going to be noise. Probably a few notifications every week as people issue certs for our hundreds of sites. What do we do with that data?

I've updated the tool to emit the discovered certs into MozDef instead of emailing them.

https://mana.mozilla.org/wiki/display/SECURITY/Certificate+Transparency+Monitoring

> How do we distinguish between a legitimate issuance and a fraudulent one?

Very good question. Here are two ideas :
* Establish a webapp/db that mozillians can go to, auth with MFA, and register their intent to create a letsencrypt cert for a given domain name. When a MozDef event comes in showing a letsencrypt cert some agent checks the db. If it finds a registered intent to create that cert then everything is good. If it doesn't the agent could do something like point the DNS at a landing page explaining how to register with this DB
* Require that people go through the CAB before requesting a letsencrypt cert and check against CAB records

> Perhaps what we really want is to only monitor the really high profile names?

This would be doable. If we could produce a list of domain names that we consider high profile we could alert in MozDef anytime certificates are created for those specific domains. Can we produce that list of domains which should never be produced by any entity other than digicert?

April,
 Is the current state that we're in, where the data is being collected in MozDef but we don't yet have an alerting rule that would alert us if someone somehow created a www.mozilla.org letsencrypt cert, an acceptable state to request unblacklisting us with letsencrypt?

Comment 11

[Julien Vehent]

> Establish a webapp/db that mozillians can go to, auth with MFA, and register their intent to create a letsencrypt cert for a given domain name.
> Require that people go through the CAB before requesting a letsencrypt cert and check against CAB records

Isn't the point of LE/ACME to *not* have to jump through hoops to get a certificate? 

Maybe requesting the /contribute.json endpoint of the subject of an LE cert would provide the information needed to locate the owner and issue a notification, if standards are followed.

> Perhaps what we really want is to only monitor the really high profile names?

I just want to remind here that high profile names should use pinning (preload & hpkp) such that mis-issuance from any CA would have no security impact. That doesn't mean we shouldn't monitor for it, just that monitoring is more of an awareness item than a defense mechanism.

Comment 12

[Gene Wood [:gene]]

I talked with April over IRC and got a "yes" to my question of "Is the current state that we're in, where the data is being collected in MozDef but we don't yet have an alerting rule that would alert us if someone somehow created a www.mozilla.org letsencrypt cert, an acceptable state to request unblacklisting us with letsencrypt?"

> Isn't the point of LE/ACME to *not* have to jump through hoops to get a certificate?

Totally. I'm personally against the CAB idea for exactly that reason. I don't think that there should be a human gate that slows a user down from getting a cert from LE.

> high profile names should use pinning...monitoring is more of an awareness item than a defense mechanism

I would say that until high profile names *actually do* use pinning (instead of should use pinning), this is a defense mechanism. Indeed some of our high profile names use pinning but only some.

:ulfr or :April

If we could produce a list of domain names that we consider high profile we could alert in MozDef anytime certificates are created for those specific domains. Can we produce that list of domains which should never be produced by any entity other than digicert?

Comment 13

[Julien Vehent]

All sensitive domains used by Cloud Services are listed in the preloaded pins (which I'm requesting to update in bug 1301956). You could use that list as a starting point.

Comment 14

[April King [:April]]

I've muddled though a list like that at some point in bug 1251768.  Here is that list:

Including subdomains:
aus.mozilla.org
aus2.mozilla.org
aus3.mozilla.org
aus4.mozilla.org
aus5.mozilla.org
aus6.mozilla.org
aus7.mozilla.org
aus8.mozilla.org
aus9.mozilla.org
addons.mozilla.org
addons.mozilla.net
bugzilla.mozilla.org
bmoattachments.org
cdn.mozilla.org
cdn.mozilla.net
download.mozilla.org
firefox.com
getfirefox.com
services.mozilla.com

Domain only:
mozilla.org
www.mozilla.org
mozilla.com
www.mozilla.com
wiki.mozilla.org
support.mozilla.org
developer.mozilla.org
irc.mozilla.org
hg.mozilla.org

Comment 15

[Gene Wood [:gene]]

Here's the link to the current pin list : https://hg.mozilla.org/mozilla-central/file/tip/security/manager/tools/PreloadedHPKPins.json

Comment 16

[Gene Wood [:gene]]

With CT data in MozDef now for some time (via https://github.com/mozilla/certspotter-cloudformation ) this is resolved

Comment 17

[Gene Wood [:gene]]

I've written up a summary page that gives the background to all of this : Certificate Authority Authorization CAA including Let's Encrypt