Closed Bug 1991196 Opened 1 year ago Closed 11 months ago

Sectigo: OCSP, caIssuers, and CRL endpoints unavailable for a single Subordinate CA

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: martijn.katerbarg, Assigned: incident-response)

Details

(Whiteboard: [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure])

Attachments

(1 file)

Preliminary Incident Report

Summary

  • Incident description:

On September 25th, 2025 at 11:28 UTC, we noticed two leaf certificates attributed to Sectigo on OCSPWatch.

Preliminary investigation has shown that the domain used for the OCSP, caIssuers, and CRL endpoints has not remained under Sectigo’s control and has recently expired. While we work on a solution to reinstate the domain, we are also planning the revocation of affected leaf certificates as well as the corresponding Subordinate CA certificate.

  • Relevant policies: Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates version 2.1.7 Section 4.9.7.: “CRLs MUST be available via a publicly-accessible HTTP URL (i.e., "published").”

  • Source of incident disclosure: Self Reported

Assignee: nobody → incident-response
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ocsp-failure] [crl-failure]
Whiteboard: [ca-compliance] [ocsp-failure] [crl-failure] → [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure]

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000016
  • Incident description:

On September 25th, 2025 at 11:28 UTC, we noticed two leaf certificates attributed to Sectigo on OCSPWatch.

Preliminary investigation has shown that the domain used for the OCSP, caIssuers, and CRL endpoints has not remained under Sectigo’s control and has recently expired. While we work on a solution to reinstate the domain, we decided revoke the affected leaf certificates as well as the corresponding Subordinate CA certificate.

  • Timeline summary:

    • Non-compliance start date: 2025-09-24 15:38:36 UTC
    • Non-compliance identified date: 2025-09-25 11:28 UTC
    • Non-compliance end date: 2025-09-30 20:36:54 UTC
  • Relevant policies: Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates version 2.1.7 Section 4.9.7.: “CRLs MUST be available via a publicly-accessible HTTP URL (i.e., "published").”

  • Source of incident disclosure: Self Reported

Impact

  • Total number of certificates: 17
  • Total number of "remaining valid" certificates: 13. As part of this incident, we revoked the remaining 13 certificates, meaning there now are 0 remaining valid.
  • Affected certificate types: DV TLS
  • Incident heuristic: All unexpired certificates issued by the COMODO RSA Domain Validation Secure Server CA 2 CA.
  • Was issuance stopped in response to this incident, and why or why not?: Yes. Issuance for the affected Subordinate CA was halted to prevent increased impact due to this incident.
  • Analysis: N/A.
  • Additional considerations: N/A.

Timeline

All times are in UTC.

  • 2014-09-25
    o We issue the COMODO RSA Domain Validation Secure Server CA 2 CA Certificate. At this time, Comodo CA (still part of Comodo Group and not yet rebranded to Sectigo) is using a third party CDN to proxy OCSP and CRL traffic. The Subordinate CA in question has been established to issue leaf certificates for one particular customer, which is a different, major CDN provider. The domain “comodoca4.com” is setup specifically for this SubCA and is routed through the customer’s own CDN, rather than Sectigo’s default CDN provider.

  • 2017-10-01
    o Comodo CA is carved out of Comodo Group. As we have mentioned in the past, this means disentangling twenty years' worth of utterly interwoven software, data, and employee responsibilities. This task proves massive and elusive and presents challenges we didn't foresee when we set out to separate the two businesses. For reasons unclear at the time of this incident, comodoca4.com is not transferred to Comodo CA’s new domain registrar.

  • 2018-11-01
    o Comodo CA officially rebrands to Sectigo.

  • 2024-11-20
    o 12:45:52 We issue the last certificate on this Subordinate CA for the customer in question.

  • 2025-03-25
    o 08:13 In response to our communication to customers regarding changes to the Chrome Root Program Policy around single purpose hierarchies and the prohibition of the clientAuth EKU, we internally discuss which options we have available to issue serverAuth-only certificates ahead of our general deprecation of the clientAuth EKU.
    o 08:52 We reconfigure the now-unused “COMODO RSA Domain Validation Secure Server CA 2” CA to support serverAuth-only certificates at short notice. At this time, we neither review nor update the intended AIA and CRL endpoints.

  • 2025-05-14
    o We issue our new R40 set of Subordinate CAs. Issued by our single purpose Root CAs, this set is intended to be used for subsequent issuance of serverAuth-only certificates.

  • 2025-09-24
    o 15:38:36 The domain comodoca4.com expires.

  • 2025-09-25
    o 11:28 We notice two certificates issued by Sectigo listed on OCSPWatch showing DNS errors.
    o 11:37 We determine queries for neither crl.comodoca4.com, crt.comodoca4.com nor ocsp.comodoca4.com are returning a valid response.
    o 11:39 Our initial expectation is that DNS records have been removed incorrectly. The issue is escalated to our DevOps team to investigate.
    o 12:04 We confirm DNS records have not been removed nor modified.
    o 12:16 Further investigation shows the domain name is not registered through our default domain registry.
    o 12:22 We notice that while the Registry Expiry Date shows expiration in 2026, the Registrar Registration Expiration Date shows 2025-09-24T15:38:36Z, confirming the domain has expired and thus no longer resolving.
    o 12:45 With no team member having access to the registrar account, we suspect the domain name is still registered within the Comodo Group registrar account and was not transferred in the carve-out of Comodo CA.
    o 13:09 We send an email to contacts at the original company, requesting the domain name be transferred. As we suspect this process will take time, we discuss alternative means of resolving the incident.
    o 13:20 A decision is made to (a) revoke the leaf certificates pointing to comodoca4.com for AIA and CRL services, and (b) revoke the Subordinate CA itself. As the URLs within the leaf certificates are no longer resolving, revocation of the Subordinate CA is deemed the only option to resolve this incident in a timely fashion.
    o 13:25 We decide to halt issuance of all certificates by the affected Subordinate CA.
    o 13:47 Issuance access to the affected Subordinate CA is removed, effectively halting further issuance.
    o 14:34 Relevant internal stakeholders are made aware of the incident, the affected certificates, and affected subscribers.
    o 15:02 A revocation event is created internally to schedule the revocation of affected certificates and perform customer outreach.

  • 2025-09-26
    o 08:04 Policy Authority approval is requested for the revocation of the affected Subordinate CA certificate along with 6 other Subordinate CA certificates which are no longer in use and could potentially suffer the same consequence.
    o 14:24 Policy Authority approval is granted for the revocation of the Subordinate CA certificates.

  • 2025-09-30
    o 09:09 We revoke all unexpired and unrevoked certificates issued by the affected Subordinate CA.
    o 11:55 We revoke the Subordinate CA certificates.
    o 20:36:54 We publish new CRLs reflecting revoked Subordinate CA certificates.

Related Incidents

Bug Date Description
1930759 2024-11-12 Both incidents share a common cause of a domain name utilized for OCSP, caIssuers and CRLs having expired. Ultimately the root cause of the expiration differs, however.

Root Cause Analysis

Contributing Factor 1: No deep review of utilized OCSP, caIssuers and CRL settings

  • Description:

In response to our communication to subscribers about upcoming changes regarding the removal of the clientAuth EKU from TLS certificate issuance, we received a request from a subscriber to be able to issue serverAuth-only certificates on short notice.

As our next offline signing ceremony was not due for several more weeks, we discussed what alternative options were available on shorter notice, to allow and encourage customers to test well in advance.

The alternative option we decide to go forward with is to utilize one of our now-unused Subordinate CAs, specifically “COMODO RSA Domain Validation Secure Server CA 2”. This Subordinate CA has in the past been issued and utilized for 1 specific customer, however, was no longer being utilized.

As the Subordinate CA at this time had remaining time-valid certificates which did not show any signs of issues, we did not consider needing to review the certificate profile of the Subordinate CA.

To note: Even if such a review had been complete, at that time comodoca4.com was pointed to our nameservers, and our nameservers were correctly setup. As such, a review might not have led to us discovering we were not the registrant of the domain itself.

  • Timeline:

Communications about the clientAuth EKU deprecation and the activation of the Subordinate CA in question was done in March of 2025.

  • Detection:

Prior to the incident, there had not been any direct reason to perform a review of the profile used by this Subordinate CA, as it was working at the time of re-activation.

  • Interaction with other factors: N/A.

Contributing Factor 2: Domain name not transferred during Comodo CA carve-out

  • Description:

Sectigo (then Comodo CA) is carved out of Comodo Group in 2017. As we have noted in the past, that meant disentangling twenty years' worth of utterly interwoven software, data, and employee responsibilities. This task proved massive and elusive and presented challenges we didn't foresee when we set out to separate the two businesses.

The separation meant a substantial number of domain names were transferred between different registrars. For reasons unclear at time of this incident, comodoca4.com is not transferred to Sectigo’s new domain registrar.

Throughout the years, comodoca4.com remained pointed at our own nameservers, preventing this incident from occurring earlier.

  • Timeline:

Sectigo / Comodo CA were carved-out of Comodo Group in the fall of 2017.

  • Detection:

We detected the fact that we were no longer the registrant of the domain name during the investigation of this incident.

  • Interaction with other factors:

Without this contributing factor, the domain name would have been renewed per our default policies, meaning the incident would not have occurred.

Lessons Learned

  • What went well:

    • We were able to revoke the affected Subordinate CA certificate without causing harm for Subscribers.
    • We were able to halt issuance for the affected Subordinate CA on short notice.
  • What didn’t go well:

    • During the original carve-out of Comodo CA, we did not migrate the comodoca4.com domain.
    • When we re-activated the affected Subordinate CA, we did not review if the OCSP, caIssuers and CRL profile settings were still fully under our control.
  • Where we got lucky:

    • The Subordinate CA had recently only been used to issue a small number of leaf certificates.
    • The domain did not expire in previous years, at which time a much larger number of certificates were time-valid.
  • Additional: N/A.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Revoke Subordinate CA Prevent Contributing Factor 2 By revoking the Subordinate CA in question we have mitigated the active incident 2025-09-30 Completed
Review all active OCSP, caIssuers and CRL endpoints Prevent Contributing Factor 1 We want to confirm if any other Subordinate CAs are in the danger zone of having a similar incident occur. 2025-10-30 Ongoing

Appendix

See attachment #9518669 [details] for the affected leaf certificates.

Based on our action items, we'd like to request a next-update for 2025-10-30.

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure] → [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure] Next update 2025-10-30

Yesterday, we concluded our review of OCSP, caIssuers and CRL endpoints, completing our action items.

Report Closure Summary

  • Incident description:

The domain name used for the OCSP, caIssuers, and CRL endpoints of a single Subordinate CA has not remained under Sectigo’s control and has recently expired, resulting in these services becoming unavailable for a total of 17 leaf certificates.

  • Incident Root Cause(s):

The two companies failed to transfer this domain name during the carve-out of Sectigo (then Comodo CA) from Comodo Group in 2017. We did not observe this gap when we repurposed this Subordinate CA, leading to this incident.

  • Remediation description:

We have revoked the leaf certificates and Subordinate CA in question.

  • Commitment summary:

Historically, Sectigo allowed for more flexibility around the use of Subordinate CA endpoints, a practice that we have tightened over the last few years. We commit to keeping the endpoints as uniform across our issuance base as deemed appropriate.

We have completed all Action Items disclosed in this report as described and request its closure.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-11-06.

Whiteboard: [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure] Next update 2025-10-30 → [close on 2025-11-06] [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure]
Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Whiteboard: [close on 2025-11-06] [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure] → [ca-compliance] [ocsp-failure] [crl-failure] [policy-failure]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: