1735247 - Let's Encrypt: Mis-issued certificates related to SC48v2

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jillian]

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0

Steps to reproduce:

Let’s Encrypt has received and confirmed a mis-issuance report related to the SC48v2 ballot effective 2021-10-01. We immediately stopped issuance while reviewing the report. We are in the process of writing a patch, restoring issuance, and reviewing certificates that will need to be revoked. This is a preliminary notification and we will follow-up with a full incident report and list of mis-issued certificates.

Comment 1

[Jillian]

At 2021-10-12 00:54 UTC Let's Encrypt released a fix and restored issuance services. Within 24 hours of this update, we will post a full incident report including mis-issued certificate information and the revocation status.

Comment 2

[Jillian]

Summary

On 2021-10-01 a new Baseline Requirements revision (Ballot SC48v2) went into effect stating that “the Fully-Qualified Domain Name or the FQDN portion of the Wildcard Domain Name MUST consist solely of Domain Labels that are P-Labels or Non-Reserved LDH Labels”.

Let’s Encrypt had reviewed the requirement before the effective date, but missed a case to forbid a Reserved LDH Label when a hyphen is its second character. The code incorrectly allowed domains like a---foo.example.com but correctly forbade names like ab--foo.example.com.

On 2021-10-11 it was reported that Let’s Encrypt had issued some certificates that were not compliant with this new requirement. Let’s Encrypt verified the claim, and stopped CA issuance while a fix was deployed. An audit of certificates issued since 2021-10-01 revealed 7 affected certificates. The certificates were revoked within 24 hours of the report.

Incident Report

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

• Let’s Encrypt was notified on 2021-10-11 at 21:11 UTC by Michael Lettona (DigiCert) via our cert-prob-reports e-mail that our software was potentially not respecting SC48v2 and Let’s Encrypt had mis-issued certificates. The e-mail contained 4 possible mis-issued certificates.

A timeline of the actions your CA took in response.

Before Incident:

• 2021-07-15 15:00 UTC: SC48v2 voting period begins
• 2021-07-21 16:15 UTC: A Let’s Encrypt engineer files an internal ticket to review the Boulder CA software relevant to the proposed changes in SC48v2 and begins reviewing the ballot language and source code.
• 2021-07-21 21:05 UTC: The review is completed and the investigator concludes that the ballot language and source code match. Two additional engineers review the conclusion and agree.
• 2021-07-22 15:00 UTC: SC48v2 passes and the effective date will be 2021-10-01
• 2021-07-28 15:12 UTC: A Let’s Encrypt engineer closes the internal ticket with the conclusion that the Boulder CA software does not need updates

Incident:

• 2021-10-11 21:11 UTC: The report is sent to cert-prob-reports.
• 2021-10-11 21:47 UTC: A Let’s Encrypt engineer reviews the report and the team begins to evaluate it.
• 2021-10-11 22:10 UTC: Let’s Encrypt halts issuance to prevent possible, additional mis-issuance while the report is verified and confirmed.
• 2021-10-11 22:40 UTC: Let’s Encrypt confirms that the Boulder CA software does not comply with the Baseline Requirements once SC48v2 is in effect. The team begins to write a fix and review issued certificates for non-compliance.
• 2021-10-11 23:31 UTC: The fix is merged.
• 2021-10-12 00:06 UTC: The fix is deployed to the Let’s Encrypt staging environment and tested.
• 2021-10-12 00:54 UTC: The fix is deployed to the Let’s Encrypt production environment and issuance is restored.
• 2021-10-12 19:15 UTC: An audit of certificates issued since 2021-10-01 reveals 7 affected certificates. Affected subscribers were notified and the 7 certificates were revoked less than 24 hours after the problem report.

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

• Let’s Encrypt stopped issuance at 2021-10-11 22:10 UTC. Issuance services were restored at 2021-10-12 00:54 UTC after a fix was released.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued.

• Problem: certificates are issued with domain labels like a---foo which are Reserved Labels but they are not P-Labels and are non compliant with the Baseline Requirements after SC48v2 is in effect on 2021-10-01 00:00 UTC
• Total Certificates Affected: 7 certificates
• First Issued Certificate: 2021-10-01 21:48 UTC
• Last Issued Certificate: 2021-10-11 16:13 UTC

In a case involving TLS server certificates, the complete certificate data for the problematic certificates.

• https://crt.sh/?id=5344759725
• https://crt.sh/?id=5342618375
• https://crt.sh/?id=5351516513
• https://crt.sh/?id=5365452699
• https://crt.sh/?id=5395188891
• https://crt.sh/?id=5375328135
• https://crt.sh/?id=5336716810

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

• In July 2021, prior to the effective date of the SC48v2 requirement, Let’s Encrypt engineers reviewed the ballot language and the corresponding Boulder CA code that checks domain names for compliance with the baseline requirements. They mistakenly determined that the code was already compliant. The code allowed P-Label (xn--) domains, and forbade any FQDN where the first and second characters were a-z0-9 and the third and fourth characters were hyphens. The CA also forbids any FQDN beginning and ending with a hyphen. The case that was missed was disallowing a hyphen as the second character if the third and fourth characters are hyphens.
  The problem was not detected earlier because the pre-issuance linting checks did not contain a check for this problem.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• Patch the Boulder CA software to comply with the Baseline Requirements -- DONE 2021-10-12 00:54 UTC
• Expand our pre-issuance linting to catch this as a secondary check -- DUE 2021-11-01

Comment 3

[Ben Wilson]

I will close this next Wed. 20-Oct-2021, unless there are any follow-up questions or objections.