1462735 - Let's Encrypt: Case-sensitive CAA tag processing

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

Josh Aas from Let's Encrypt posted the following incident report to mozilla.dev.security.policy:

At 12:45 UTC we received a report to our [e-mail] contact address that Let’s Encrypt was improperly handling CAA records with mixed case tags, resulting in mis-issuance under the baseline requirements. Thanks to Corey Bonnell of TrustWave for the report.

RFC 6844 Section 5.1[0] says: “Matching of tag values is case insensitive.” Let’s Encrypt’s implementation of CAA record processing processed CAA tags case sensitively[1]. This led to CAA tags that were not lowercase being ignored during CAA validation.

The problem was quickly confirmed, and a fix was developed and reviewed[2], with tests, by 13:28 UTC - under an hour from the initial report. We deployed the fix to our staging environment at 14:37 UTC. We disabled issuance of new certificates in our production environment at 14:45 UTC, to prevent additional misissuance from occurring. We deployed the fix to our production environment at 15:20 UTC.

Our logging of the CAA records processed does not provide the case information we need to determine whether other issuances were affected by this bug. We plan to perform these two post-incident remediation items to start with:

1. Improving the CAA validation logging in Boulder[3] to log CAA records prior to our processing.

2. Performing a scan of current CAA records for the domain names we have issued for in the past 90 days, specifically looking for tags in CAA records with non-lowercase characters. We’ll examine such instances on a case-by-case basis to determine the appropriate action.

The original reporter identified one certificate (https://crt.sh/?id=469407542) that was issued in violation of the CAA RFC as part of their testing. We have revoked this certificate as of 15:41 UTC.

[0] - https://tools.ietf.org/html/rfc6844#section-5.1

[1] - https://github.com/letsencrypt/boulder/blob/9990d14654661736a6ee6dc1520f605d0896c72d/va/caa.go#L82-L100

[2] - https://github.com/letsencrypt/boulder/pull/3722

[3] - https://github.com/letsencrypt/boulder/issues/3724

Comment 1

[Wayne Thayer]

Josh: please update this bug with status on the remediation actions you've proposed.

Comment 2

[Josh Aas]

1. CAA logging improvements are in this commit to our CA software:

https://github.com/letsencrypt/boulder/pull/3726

2. We scanned domains and identified 11 certificates that we would not have issued with their current CAA records. We sent emails to ACME account contacts giving them four days to respond. We received no responses. During that time 3 of the 11 certificates identified expired. The remaining 8 were revoked.

We now consider the incident to be closed.