Open Bug 1891039 Opened 19 days ago Updated 13 days ago

Sectigo: Premature disabling of CRL generation for an inactive CA

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: martijn.katerbarg, Assigned: martijn.katerbarg)

Details

(Whiteboard: [ca-compliance] [crl-failure])

Incident Report

Summary

Last month we disabled CRL generation for a number of expired CAs. Yesterday evening, our team noticed that one of those disabled CRLs belongs to an expired S/MIME Root CA that is also the Subject of two unexpired cross-certificates issued by a currently trusted root, which means that disabling CRL generation for this CA caused a compliance incident.

We intend to resolve this matter by revoking those cross-certificates on Wednesday 17th April. We expect to post a full incident report by Friday 19th April.

Assignee: nobody → martijn.katerbarg
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [crl-failure]

Incident Report

Summary

In March of 2024, we disabled CRL generation for several expired CAs.

On April 10, 2024, we noticed that one of those disabled CRLs belongs to an expired S/MIME root that shares the same Subject Public Key as two unexpired Cross-Certified Subordinate CA Certificates issued by a currently trusted root, which means that disabling CRL generation for this CA caused a compliance incident.

Impact

1 CRL, issued by the UTN-USERFirst-Client Authentication and Email CA.

Although the Root CA Certificate expired in 2019, the following two Cross-Certified Subordinate CA Certificates with later expiration dates also exist:

Timeline

All times are UTC.

2007-07-06:

2008-07-28:

2023-07-13:

  • 16:00 CA/B Forum Ballot SC-63 passes, introducing changes to (amongst other things) the CRL Profile section. The effective date within this section is set to March 15, 2024.

2023-12-22:

  • 12:26 We deploy a pre-release of a planned CRL Monitor within our Internal Audit tooling, which is intended to complement our existing internal CRL monitoring. This CRL monitor has a more external perspective, taking its list of monitoring targets from the CRL URLs that we have disclosed to the CCADB. The intent of this is to detect any discrepancies between what we have reported to CCADB and what our CRL generation and publication systems are actually doing. This pre-release has logging enabled, but is not (yet) integrated with alert notifications.

2024-02-14:

  • 10:53 We are still issuing CRLs where the signature hash algorithm is SHA-1, mainly for some Offline Root CAs. We determine that continued issuance of these CRLs would be considered a violation of the new CRL Profile requirements that will go into effect on March 15, 2024.
  • 10:58 We make a configuration change so that new CRL signatures will use the SHA-256 algorithm instead of SHA-1. As most of these are for Offline Root CAs, these changes do not go into effect immediately but require an offline signing ceremony. We schedule an offline signing ceremony for February 20, 2024.
  • 18:32 We send an email to our internal WebPKI Incident Response mailing list, in which we propose disabling CRL generation for most of our expired CAs. No objection is filed.

2024-02-20:

  • 12:00 We start our planned offline signing ceremony, during which we produce signatures for several pre-generated CRLs.
  • 12:29 We submit the signatures to the person responsible for importing them into our online platform.
  • 12:33 We load the pre-generated CRLs into our online platform, and also remove all of the not-yet published SHA-1 based pre-generated CRLs.

2024-03-18:

  • 17:20 We disable CRL generation for most of our expired CAs.

2024-03-22:

2024-04-10:

  • 17:10 We manually review the output of the CRL Monitor available in the Internal Audit tooling. We notice that it has been showing failures since March 22, 2024 at 16:03:43.
  • 17:16 We escalate the finding internally.
  • 18:27 We come to the conclusion that generation of this CRL was disabled deliberately but that we had not taken into account the not-yet expired Cross-Certified Subordinate CA Certificates sharing the same Subject Public Key.

2024-04-11:

  • 10:36 Instead of re-enabling the affected CRL distribution point, we propose a plan internally to revoke the Cross-Certificate Subordinate CA Certificates instead, since this hierarchy has not had any active leaf certificates for several years.
  • 11:04 We create a ticket for our Policy Authority (PA), requesting authorization to revoke the affected CA Certificates.
  • 11:15 Anticipating the approval of the PA ticket, we book a timeslot for the Offline Revocation Ceremony. This is scheduled for April 17, 2024 at 12:00, to be within 7 days of discovering this incident.

2024-04-12:

  • 15:37 We receive the necessary PA votes, authorizing revocation of the two affected CA Certificates.

2024-04-16:

  • 15:21 We mark the two affected CA Certificates as revoked in our database and prepare the scripts for the Offline Revocation Ceremony.

2024-04-17:

  • 12:29 We complete the Offline Revocation Ceremony.
  • 13:36 We finish loading the signed artifacts into our CRL/OCSP distribution system. OCSP responses with a Revoked status are published straight away.
  • 14:10 An updated AAACertificateServices.crl that revokes the two affected CA certificates is published.
  • 14:13 We update the corresponding CCADB records to disclose the Revoked statuses.
  • 14:53 We confirm that our CDN is no longer caching the previous CRL.

Root Cause Analysis

Our policy for issuing Cross-Certified Subordinate CA Certificates to Root CAs that we own has always been to align the notAfter date with either the notAfter date of the self-signed Issuer CA certificate or the notAfter date of the self-signed Subject CA certificate, whichever is earlier.

The affected Cross-Certified Subordinate CA Certificates were issued by Comodo in 2007 and 2008, with the notAfter dates mistakenly aligned with the much later notAfter date of the Issuer CA. To help explain this, we have created the following overview of the relevant CA Certificates:

commonName Type notAfter
AAA Certificate Services Root 2028-12-31
UTN-USERFirst-Client Authentication and Email Root 2019-07-09
UTN-USERFirst-Client Authentication and Email Cross 2028-12-31
UTN-USERFirst-Client Authentication and Email Cross 2028-12-31

When making the decision last month to disable CRL generation for various expired CAs, we based the implementation on our long-standing policy, neglecting to factor in this mistake from the Comodo era.

The majority of the CRL distribution points that were disabled on March 18th belong to expired CAs that were not Cross-Certified, and we believe this contributed to a lack of focus on the small minority that were Cross-Certified. We did not have a well-established process for retiring CRLs or any associated tooling to ensure that no CRL was disabled prematurely. Instead, we relied on team members’ experience and knowledge prior to executing the disablement.

Lessons Learned

What went well

  • We were able to revoke the Cross-Certified Subordinate CA certificates within 7 days of discovering this incident.

What didn't go well

  • Prior to the disablement of CRL generation, we did not consider the possible existence of relevant and unexpired Cross-Certified Subordinate CA Certificates.
  • While we discovered the incident ourselves through monitoring, the monitoring system had not been fully completed. As such, email alerting was not yet set up.
  • At some point between 2024-03-18 and 2024-04-10 Rob noticed that https://crt.sh/mozilla-disclosures was flagging a problem with this CRL distribution point but he incorrectly assumed that it was a minor and low priority logic bug in the crt.sh code that did not need to be addressed urgently.
  • Whilst we do review https://sslmate.com/labs/crl_watch/ from time to time on an ad hoc basis, we did not happen to review that page between 2024-03-18 and 2024-04-10.

Where we got lucky

  • No other CA certificates were affected by this incident.
  • There are no unexpired leaf certificates that would have been in scope for the prematurely disabled CRL, and so compliance monitors are the only relying parties that would have noticed or been affected.

Action Items

Action Item Kind Due Date
Update our PA Policy to incorporate disablement of OCSP and CRL services when shutting down any given CA, including specifying a set of actions that need to be confirmed. One of these actions will be to confirm that no unexpired Cross-Certified Subordinate CA Certificates exist Prevent 2024-04-30
Complete our CRL Monitor and enable email alerts Detect Completed

Appendix

Details of affected certificates

Certificate issuance was not affected.

You need to log in before you can comment on or make changes to this bug.