1752670 - Let’s Encrypt: TLS Using ALPN Allows Additional Identifiers in Challenge Certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jillian]

This is a preliminary incident report.

At 19:41 UTC on Friday, January 28, 2022, Andrew Ayer reported to Let’s Encrypt / ISRG that we issued a certificate validated with the TLS-ALPN-01 method in such a way that it was non-compliant with RFC 8737. In particular, RFC 8737 states that “The ACME server verifies that… the certificate returned contains… a subjectAltName extension containing the dNSName being validated and no other entries”

The subjectAltName extension of the non-compliant challenge certificate was:

X509v3 Subject Alternative
DNS:a2.coolsite510.com, IP Address:127.0.0.1

Because the challenge certificate’s subjectAltName extension contained entries (namely, IP Address:127.0.0.1) other than the dNSName being validated the resulting final certificate issued by Let’s Encrypt was mississued. The Boulder validation code enforced that only one dNSName was presented in the challenge certificate, but did not enforce that the Subject Alternative Names extension did not contain identifiers of any other kind.

We confirmed this report and immediately began working on a fix. At 21:22 UTC, we revoked the certificate that was known to be misissued. We developed and merged a fix for the issue, and at 01:56 UTC on 29 January 2022 we deployed the fix to our production environment. We are also examining our issuance logs to determine if any other certificates have been issued based on a similarly malformed challenge certificate, and will revoke any such certificates by 2022-02-02 19:41, five days after we became aware of the incident.

We will provide our next update on Monday, 31 Jan 2022.

Comment 1

[Jillian]

We are still auditing our validation logs to identify whether additional affected authorizations, and thereby affected certificates, exist. We will still revoke any additional affected certificates we discover by 2022-02-02 19:41 UTC, and will provide a full incident report on or before Thursday, 03 Feb 2022.

Comment 2

[Jillian]

Summary

At 2022-01-28 19:41 UTC Let’s Encrypt was made aware of an instance of non-compliance in our implementation of the TLS-ALPN-01 challenge type (RFC 8737), which is the basis of the TLS Using ALPN validation method (BRs Section 3.2.2.4.20).

RFC 8737, Section 3, states that “The ACME server verifies that… the certificate returned contains… a subjectAltName extension containing the dNSName being validated and no other entries”. Our TLS-ALPN-01 validation code was ensuring that the certificate returned contained exactly one dNSName, but did not ensure that the certificate did not contain any other identifiers (such as an iPAddress or an rfc822Name) in the SAN extension.

This issue has been fixed, and we have identified and revoked all certificates containing one or more identifiers whose validations were affected by this bug. The affected certificates were revoked within five days from the discovery of the issue.

Incident Report

How we first became aware of the problem.

Let’s Encrypt received a bug report from Andrew Ayer to our security@ mailbox at 19:41 UTC on 2022-01-28. We immediately began incident response, and had confirmed the presence of the bug within an hour.

We note that this bug would have been detected by our comprehensive review of the TLS Using ALPN standards and implementation that we have committed to in Bug 1751984. We are grateful to Andrew for conducting his own review of this critical code.

Timeline of incident and actions taken in response.

All times are UTC; some events listed only at date granularity as timestamps are not available for RFC and BR publication events.

2018-02-22

• First draft of RFC 8737 posted, which already includes “and no other entries” language

2018-06-06

• 18:04 TLS-ALPN-01 challenge type added to Boulder, which checks that there is exactly 1 dNSName but does not check other SAN types

2020-02-29

• RFC 8737 published

2020-09-22

• Ballot SC33, which references RFC 8737, becomes effective in the Baseline Requirements. (Incident Begins)

2022-01-28

• 19:41 Bug report sent to Let’s Encrypt/ISRG engineering
• 19:44 Initial response to reporter that we are reviewing the report
• 20:32 Root cause identified and fix started
• 21:22 The reported certificate with serial 0382fa79c0754b5a40cfa33376c169d5f635 was revoked
• 23:18 Search for additional affected validations begins
• 23:38 Fix merged into Boulder

2022-01-29

• 00:07 New version of Boulder deployed to Staging
• 01:37 New version of Boulder deployed to Production

2022-02-01

• 00:11 Identified candidate set of 625,028 challenge response certificates which were presented as part of validations for names that eventually had successful TLS-ALPN-01 validation
• 01:35 Confirmed that all but 812 of the above challenge response certificates contained exactly one SAN entry, and as such were not affected
• 23:20 Confirmed that all but 3 of the above challenge response certificates were presented as part of failed validations, and as such were not affected
• 23:20 Identified the 3 authorizations, including the one in the initial bug report, which were affected

2022-02-02

• 00:48 All 3 affected authorizations revoked (Incident Ends)
• 00:49 Confirmed that only 3 certificates, including the one in the initial bug report which was previously revoked, were issued based on the affected authorizations
• 00:51 Both remaining affected certificates revoked

Whether we have stopped the process giving rise to the problem or incident.

We have merged and deployed a fix, so that no new authorizations will be affected. We have identified all affected authorizations, and have revoked them so that they cannot be used for new issuances. We have identified all affected issuances, and revoked those certificates.

Summary of the affected certificates.

Three certificates issued between 2022-01-26 00:13 UTC and 2022-01-29 01:56 UTC were affected. Although there may have been additional unexpired affected certs issued prior to 2022-01-26 00:13 UTC, any such certificates were revoked as part of the remediation for Bug 1751984, and so were not considered in our analysis.

Complete certificate data for the affected certificates.

Full details of the three affected certificates can be seen at the following URIs:
https://crt.sh/?sha256=7963DFEE8ABBC855A18B2FB3C5B9417F2901B88A047424821DC907A1FF2B77C1
https://crt.sh/?sha256=5481D8C3FC8201EDE88E0EA645C3F662D485A98E877A2CD4FACED1879DD1DE6A
https://crt.sh/?sha256=5FF1A4B972BFE7DC93A095EF9244094FFACFEE449F4E114430C6B68A4C551B95

Explanation of how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In many ways this bug is similar to those described in Bug 1751984. We should have re-reviewed the code when RFC 8737 was standardized, and again when it was incorporated into the Baseline Requirements. We now have processes that ensure we do so, but did not at the time.

It appears to differ in one way, however: the discrepancy between the specification and the implementation code was present from the very first draft of the spec and the very first commit of the implementation. This bug did not emerge as the specification changed while the code remained static; it was present from the very beginning.

Looking further back, the TLS-ALPN-01 challenge type implementation in Boulder was mostly copied from an initial implementation inside Pebble, as a replacement of the TLS-SNI-02 challenge type. In that change, the check requiring there be exactly one dNSName was replacing a similar check from the TLS-SNI-01 implementation that required there be exactly two dNSNames. Interestingly, that check was also out of compliance with the early drafts of RFC 8555, which required that the challenge certificate “MUST have exactly two subjectAlternativeNames” and “[b]oth MUST be dNSNames”.

In turn, that early implementation of TLS-SNI-02 in Pebble was itself mostly based on the early implementation of TLS-SNI-01 in Boulder, which also only inspected the dNSNames in the SAN extension. However, the TLS-SNI-01 draft specification did not have any requirement that the SAN extension of the presented challenge certificate contain exactly one entry, so that very first implementation was in fact correct.

So in the end, the chain of events here is not so different from that in the Bug 1751984: the specification and the code evolved in parallel, and there was not a final re-audit of the code when the specification was finalized.

One additional contributing factor here is the Go standard library. The built-in crypto/x509 package provided by the Go stdlib, which parses DER-encoded certificates into easy-to-use in-memory formats, does not expose the full contents of the SAN extension. It provides easy access to any domain names (dNSName), email addresses (rfc822Name), IP addresses (iPAddress), and URIs (uniformResourceIdentifier) contained within the SAN, but it completely ignores any entries using other options allowed by RFC 5280. Thus properly implementing the check requires parsing the bytes of the SAN extension directly, a rare inconvenience given the general high utility of the crypto/x509 library.

List of steps we are taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The appropriate remediation items here, specifically performing comprehensive reviews of our validation method implementations to ensure that they have not diverged from the final specifications, are already covered by Bug 1751984. We recommit to those items and those timelines, and refer readers to that ticket for updates on them.

Comment 3

[Aaron Gable]

Weekly update: Please see Bug 1751984 for updates on the remediation items here.

If the root program managers have no further questions, we ask that this ticket be closed, as all updates will be on that other ticket. Otherwise, please set the Next Update field to 2022-03-11, when we have committed to completing remediations on the other ticket.

Comment 4

[Aaron Gable]

We have completed our remediation items on Bug 1751984. If there are no further questions, we ask that this issue by closed.

Comment 5

[Ben Wilson]

I will close this tomorrow 2022-03-04 unless there are any questions or concerns.