1789521 - Let’s Encrypt: Certificates issued to Elliptic Curve Debian Weak Keys

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Gabbitas]

Incident Report:

How your CA first became aware of the problem.

On 2022-07-08, Security Researcher, Hanno Böck started a thread in MozDev Security Policy with further information regarding the Debian Weak Key vulnerability (CVE-2008-0166) in relation to Elliptic Curve keys. The common belief was that the vulnerability was not applicable to Elliptic Curve keys, but Hanno pointed out that the affected version of OpenSSL did include support for EC keys.

Hanno also created a github repository of private keys that were generated using the vulnerable version of OpenSSL. The repository contained 598,824 keys on the ECP256 and ECP384 curves.

Let’s Encrypt began work to block the keys and revoke any affected certificates. During this process, we discovered that two certificates were issued matching the SPKI of two of the EC Debian Weak Keys.

A timeline of the actions your CA took in response.

All times are in UTC.
2022-07-08 10:28: Hanno Böck started a thread in MozDev Security Policy with further information regarding the Debian Weak Key vulnerability (CVE-2008-0166) in relation to Elliptic Curve keys. Let's Encrypt staff noted the thread and continued to monitor the discussion.
2022-07-09 16:48: Let's Encrypt staff created an internal work ticket to block the affected keys. Based on discussion in the thread, the ticket was not prioritized as urgent. As noted in the MozDev thread, during the time period of the vulnerability, EC keys were not widely supported and unlikely to be used in the wild.
2022-08-30 19:50: Dry run of key blocking tool started to find any affected certificates before initiating revocation.
2022-09-01 17:00: Other iterations of weak keys (RSA) had been blocked using a flat file format that didn’t support EC keys. We decided to use an existing database table to block EC keys instead of adding support to the flat file reader. A patch was deployed to production to support addition of weak keys to the database with a comment that differentiates a Debian weak key from normal keyCompromise revocations.
2022-09-02 03:24: Two affected certificates found during investigation. The private keys were manually blocked for these certificates triggering automatic revocation of the certificates.
2022-09-02 06:02: Dry run to check 598,824 keys completed. Aside from the two affected certificates already revoked, no additional certificates were found.
2022-09-03 19:43: Completed blocking 598,824 keys on the ECP256 and ECP384 curves. No additional certificates were found or revoked.

Whether your CA has stopped or has not yet stopped, issuing certificates with the problem.

Let's Encrypt has stopped issuing certificates with Debian Weak ECDSA Keys. All affected keys on the ECP256 and ECP384 curves have been blocked. These are the only EC curves Let's Encrypt supports issuance for.

A summary of the problematic certificates.

Two certificates were issued matching the SPKI of the EC Debian Weak Keys. Both were issued on 2022-07-07, one day prior to Hanno’s message to MozDev, and appear to be research certificates. The domains resolve to Apache loading pages with no https support.

The complete certificate data for the problematic certificates.

https://crt.sh/?sha256=DEAF4C6EC41EB8078A8A29DAF55044B7CED190D299102A23742731C4C3A2114D
https://crt.sh/?sha256=87F16334428EF9059AB056D7041FAAD9C490B7038D31A8801DC087CE17681A24

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We had previously determined, based on our own analysis, CAB/F discussion [1], and other reputable sources [2][3], that elliptic curve support was not included in the affected builds of OpenSSL on Debian. Based on the new evidence, we have taken the approach of including the ECP256 and ECP384 keys in the badkeys/debianopenssl repository as part of our Debian weak keys block list.

When Let's Encrypt staff started to investigate the need to block these new keys, it was discovered that the existing tooling to block Debian weak keys was not able to handle EC keys. Some engineering work was needed to support blocking EC weak keys.

[1]https://archive.cabforum.org/pipermail/servercert-wg/2022-July/003260.html
[2]https://community.letsencrypt.org/t/is-it-possible-to-make-ecdsa-keys-with-insecure-debian-openssl/133847/6
[3]https://hovav.net/ucsd/dist/debiankey.pdf

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The engineering teams met to decide if we should modify the CA code (boulder) to handle EC keys in addition to RSA and add tooling to generate the necessary flat file of hashes read by that library. We decided against this approach.

Instead, the team decided that we would block the keys via a database entry to our blockedKeys table. During webtrust audits, one of our compliance requirements is to show that we block weak keys, including those that are part of the Debian weak keys vulnerability. Our existing tooling to administratively block keys did not support writing to the comment field in the database. Engineering work was completed to support the comment field so that when the keys were added to the blockedKeys table they would be able to be distinguished as Debian weak keys from other blocked keys for audit purposes.

The tool was run against all keys in the ECP256 and ECP384 directories in the github repository first as a dry run to determine how many certificates would be revoked, and then run to block the keys which would trigger automatic revocation of affected certificates.

After the dry run, the keys with affected certificates were the first to be blocked and certificates revoked, then the rest of the keys blocked after that.

This work is complete and all keys are blocked and affected certificates revoked.

Comment 1

[Andrew Gabbitas]

Update: We have no additional remediation items and continue to monitor this ticket. We don't plan any future updates to this ticket and consider the issue resolved unless there are questions.

Comment 2

[Aaron Gable]

Since it appears there are no further questions, we would like to request that this ticket be closed.

Comment 3

[Ben Wilson]

I'll close this on or about Wed. 26-Oct-2022.