1331257 - Set CAA DNS record for services.mozilla.com to Digicert

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Julien Vehent]

CAA records [1] are used to specify which certificate authorities are allowed to issue certificates for a given domain. CAs are expected to check the CAA record of a domain prior to issuing a certificate for it, to prevent an attacker who obtained control of a record from requesting a valid certificate from any CA.

We can use CAA records in coordination with the certificate pins hardcoded in Firefox (bug 1301956): the pins allow both Digicert and Let's Encrypt (LE) to issue certs for s.m.c. Under normal operations, we don't want LE to issue certificates for this domain, so setting a CAA record to only Digicert will tell Let's Encrypt to refuse issuance.

Should we need to lift this restriction and issue certs from LE, we would simply remove the DNS record and wait for TTL expiration. We need to keep the TTL low, say 1 hour, which shouldn't be a problem since this record will only be queried by CAs.

See also bug 882128 for CAA discussion on mozilla.org

[1] https://tools.ietf.org/html/rfc6844

Comment 1

[Julien Vehent]

Attachment: tlsobs query for non-digicert mozilla.com domains

It looks like AWS Route53 doesn't support CAA records at the moment: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ResourceRecordTypes.html

We have two options:

1) Wait for AWS support. Maybe ask them to see if there's a workaround.

2) Set it at the root of mozilla.com for the entire domain, not just smc. This may be problematic for that one domain, in attachment, that uses a non-digicert certificate.

Comment 2

[Gene Wood [:gene]]

2)

I'm not sure we can issue a mozilla.com CAA record barring use of Let's Encrypt certs for it and all subdomains (as CAA records are inherited by children) as *enabling* lets encrypt on mozilla.com subdomains is something it seems that Mozilla wants very much (with of course the exceptions that were baked into the blacklist which includes services.mozilla.com and *.services.mozilla.com : https://mana.mozilla.org/wiki/display/WebOps/Using+Let%27s+Encrypt+at+Mozilla#UsingLet%27sEncryptatMozilla-BlacklistDetails )

I would very much like to be conveying this blacklist information via CAA records so that we have control over it if we need to make changes like you mention but I don't think we should block all of mozilla.com to achieve it.

What do you expect the likelihood would be that we would want to issue services.mozilla.com certificates with Let's Encrypt in the future, specifically before AWS supports CAA?
How would you categorize the risk of changing the Let's Encrypt blacklist to allow *today* the issuance of services.mozilla.com certificates from Let's Encrypt despite not needing that functionality until some point in the future?

Comment 3

[Julien Vehent]

> block all of mozilla.com to achieve it

At the moment, all of our domains on mozilla.com use Digicert, except for one. Adding a CAA record wouldn't impact anyone until that cert expires on 2017-08-20, and maybe not even then if GlobalSign doesn't implement CAA checks.

> What do you expect the likelihood would be that we would want to issue services.mozilla.com certificates with Let's Encrypt in the future, specifically before AWS supports CAA?

Very close to zero. The Firefox org (services & releng) will stick to Digicert and only use LE as a backup. We need to fix our DNS cleanup processes before being able to use LE, and even then we don't have the automation in place to support ACME and perform daily/month cert renewals on all domains.

> How would you categorize the risk of changing the Let's Encrypt blacklist to allow *today* the issuance of services.mozilla.com certificates from Let's Encrypt despite not needing that functionality until some point in the future?

Fairly high without CAA. We have had reports on dangling domains that are left abandoned and taken over by bug bounty reporters. With LE, those takeovers could obtain a certs that Firefox considers valid given our current pinning policy (bug 1301956). I very much want to have CAA in place *before* LE lifts the blacklist.

Comment 4

[Julien Vehent]

Route53 is planning to add CAA support: https://forums.aws.amazon.com/ann.jspa?annID=4799

Comment 5

[Julien Vehent]

This is now supported by route53. Can we setup a record for services.mozilla.com?

Comment 6

[Julien Vehent]

I tested the CAA record on stage.fx-trans.net and it works as expected.

> $ dig +short CAA stage.fx-trans.net
> 0 issue "letsencrypt.org"
> 0 issuewild ";"

For digicert, we want the value to allow regular and wildcard certs, which is done with a single record:

> 0 issue "digicert.com"

TLS Observatory confirmed that all services.mozilla.com domains are served by Digicert certificates:

>     id    |    issuer    |                   subject                   |               subjectaltname                
> ----------+--------------+---------------------------------------------+---------------------------------------------
>      6748 | DigiCert Inc | push.services.mozilla.com                   | updates.push.services.mozilla.com
>      6764 | DigiCert Inc | *.sync.services.mozilla.com                 | sync.services.mozilla.com
>      6765 | DigiCert Inc | shavar.services.mozilla.com                 | tracking-protection.services.mozilla.com, tracking.services.mozilla.com
>    983278 | DigiCert Inc | search.services.mozilla.com                 | search.services.mozilla.com
>   1194740 | DigiCert Inc | push.services.mozilla.com                   | ua.push.tefdigital.com, updates.push.services.mozilla.com
>   1622705 | DigiCert Inc | location.services.mozilla.com               | location.services.mozilla.com
>   1708005 | DigiCert Inc | api.everything.me                           | appsearch.services.mozilla.com
>   1708041 | DigiCert Inc | embedly-proxy.services.mozilla.com          | embedly-proxy.services.mozilla.com
>   1708063 | DigiCert Inc | location-leaderboard.services.mozilla.com   | location-leaderboard.services.mozilla.com, loc-leaders.prod.mozaws.net, loc-leaders.services.mozilla.com
>   1708859 | DigiCert Inc | firefox.settings.services.mozilla.com       | firefox.settings.services.mozilla.com
>   1708970 | DigiCert Inc | shavar.services.mozilla.com                 | shavar.prod.mozaws.net, shavar.services.mozilla.com, tracking-protection.services.mozilla.com, tracking.services.mozilla.com
>   1771493 | DigiCert Inc | *.services.mozilla.com                      | services.mozilla.com
>   1799104 | DigiCert Inc | blocklists.settings.services.mozilla.com    | blocklists.settings.services.mozilla.com
>   1804086 | DigiCert Inc | push-dashboard.services.mozilla.com         | push-dashboard.services.mozilla.com
>   1825032 | DigiCert Inc | webextensions.settings.services.mozilla.com | webextensions.settings.services.mozilla.com
>   1827172 | DigiCert Inc | tls-observatory.services.mozilla.com        | tls-observatory.services.mozilla.com
>  23088155 | DigiCert Inc | stubdownloader.services.mozilla.com         | stubdownloader.services.mozilla.com
>  26858235 | DigiCert Inc | location.services.mozilla.com               | location.services.mozilla.com
>  91668188 | DigiCert Inc | mozphab.services.mozilla.com                | mozphab-phabhost.devsvcprod.mozaws.net, mozphab.services.mozilla.com, phabricator.services.mozilla.com


Given that setting up the CAA record does not impact production traffic, but only certificate issuance, I would say this is good to go. needinfo our DNS masters for sanity check.

Comment 7

[Jeremy Orem [:oremj]]

As long as this only affects certificate issuance, as you mentioned, I don't have a problem going forward with this.

Comment 8

[Julien Vehent]

This has been implemented.