Closed Bug 1645276 Opened 2 years ago Closed 1 year ago

Let's Encrypt: Expired ISRG Root OCSP X1 Certificate

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: agabbitas, Assigned: agabbitas)

Details

(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0

Summary:

Let’s Encrypt uses a delegated OCSP signing certificate issued by ISRG Root X1 for the purpose of signing OCSP responses for ISRG intermediate certificates. This delegated signing certificate expired on 2020-06-04 and was reissued on 2020-06-09, 24 hours after the expiration was reported. During the period that it was expired but not replaced, TLS clients building chains to ISRG Root X1 would experience OCSP validation errors if checking OCSP and validating the signing certificate.

The OCSP responses served by our OCSP responder were signed as valid until December 2020, but because the signing certificate had expired, validation of the response would fail.

This affected the OCSP responses for the 2 non-expired intermediate certificates signed by ISRG Root X1: Let's Encrypt Authority X3 and Let's Encrypt Authority X4. All subscriber certificate OCSP responses are signed by the unexpired Let's Encrypt Authority X3 and were unaffected.

Between 2020-06-04 00:00 UTC and 2020-06-09 00:00 UTC, there were 32,400,723 requests for the OCSP status of the affected intermediates, and 1,090,441,047 requests for the OCSP status of the cross-signed (unaffected) intermediates, leading us to estimate that approximately 2.9% of clients verifying Let’s Encrypt intermediates received an response with the expired OCSP signer during the outage.

Incident Report:

1. How your CA first became aware of the problem.

  • 2020-06-08 17:53 UTC - Security officers received an encrypted email from Miłosz Kaniewski with Fudo Security, stating that the delegated OCSP signing certificate had expired causing errors when verifying the OCSP response for our intermediate certificates.

2. A timeline of the actions your CA took in response.

  • 2019-12-11: OCSP responses generated and signed for Let’s Encrypt Intermediate Authority X[1-4] certificates with next update of 2020-12-11.
  • 2020-06-08 18:29 UTC - A security officer decrypts the email report and begins an investigation.
  • 2020-06-08 18:44 UTC - The reported problem confirmed valid for TLS clients building chains to ISRG Root X1. TLS clients building chains to the cross-signed IdenTrust DST Root CA X3 were not affected by this.
    • $ openssl ocsp -no_nonce -issuer <( curl -fs https://letsencrypt.org/certs/isrgrootx1.pem.txt ) -serial "0x$(curl -s https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt | openssl x509 -noout -serial | sed 's/serial=//')" -url "http://ocsp.root-x1.letsencrypt.org/" -header "HOST=ocsp.root-x1.letsencrypt.org"
      Response Verify Failure
      131927289763648:error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:crypto/ocsp/ocsp_vfy.c:92:Verify error:certificate has expired
      131927289763648:error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:crypto/ocsp/ocsp_vfy.c:92:Verify error:certificate has expired
      0xD3B17226342332DCF40528512AEC9C6A: good
      This Update: Dec 11 00:00:00 2019 GMT
      Next Update: Dec 11 00:00:00 2020 GMT
  • 2020-06-08 18:58 UTC - Let’s Encrypt Staff coordinated datacenter access and began key ceremony preparations to issue a new delegated OCSP signing certificate.
  • 2020-06-09 19:04 UTC - Completed key ceremony to sign the new certificate.
  • 2020-06-09 22:28 UTC - New OCSP responses generated and served by the OCSP responder. Incident Resolved.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

  • We did not stop issuance.

4. A summary of the problematic certificates.

5. The complete certificate data for the problematic certificates.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

  • The internal tool we use to generate OCSP responses for our intermediate signing certificates had a design flaw. The tool was able to generate a response with an expiration date later than the expiration of the certificate doing the signing. Additionally, the delegated OCSP signing certificate was not being monitored for expiration which allowed the expiration to pass without being noticed.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

  • The internal tool has been modified (https://github.com/letsencrypt/boulder/pull/4855) to error if trying to sign a response greater than the expiration of the signing certificate. ETA to deployment [2020-06-25]
  • The tool we use to monitor OCSP responses in general did not verify that OCSP response lifetimes were within the lifetime of a delegated OCSP responder. Fixed (https://github.com/letsencrypt/boulder/pull/4852) and deployed 2020-06-11.
  • We will add expiration monitoring for the new delegated OCSP certificate. ETA to deployment [2020-06-25]
  • Review and improve all certificate expiration monitoring. ETA to deployment [2020-06-25]
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Assignee: bwilson → agabbitas
Whiteboard: [ca-compliance]

Have all of the fixes listed in Comment #1 been deployed?

Flags: needinfo?(agabbitas)

As of today, 2020-08-06, the remaining remediation items from https://bugzilla.mozilla.org/show_bug.cgi?id=1645276#c1 have been deployed.

I am inclined to close this bug as fixed on or about 13-Aug-2020 unless there are any additional questions or concerns raised.

Flags: needinfo?(agabbitas) → needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.