1985307 - Sectigo: OCSP and CRL traffic not being proxied for 3 Subordinate CAs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Preliminary Incident Report

Summary

• Incident description:

On August 26th, 2025 at 10:28 UTC we noticed on https://crt.sh/mozilla-disclosures#disclosureincomplete that CRLs issued by 3 recently established Subordinate CAs were unavailable and returning a 404 response code.

Upon investigation, we discovered that OCSP responses for certificates issued by the same CAs are returning an “unauthorized” response. We made the decision to halt issuance from these SubCAs, which was done by 10:44 UTC.

We confirmed that CRLs and OCSP responses have been signed and are available at our origin server but are not being served by the CDN proxy.

By 13:34 UTC, the incident was resolved.

• Relevant policies: Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates version 2.1.5 Section 4.9.7.: “CRLs MUST be available via a publicly-accessible HTTP URL (i.e., "published").”

• Source of incident disclosure: Self Reported

Comment 1

[Martijn Katerbarg]

Preliminary Incident Report

Summary

• CA Owner CCADB unique ID: A000016
• Incident description:

On August 26th, 2025 at 10:28 UTC we noticed on https://crt.sh/mozilla-disclosures#disclosureincomplete that CRLs issued by 3 recently established Subordinate CAs were unavailable and returning a 404 response code.

Upon investigation, we discovered that OCSP responses for certificates issued by the same CAs were returning an “unauthorized” response. We made the decision to halt issuance from these SubCAs, which was done by 10:44 UTC.

We confirmed that CRLs and OCSP responses had been signed and were available at our origin server but were not being served by the CDN proxy.

By 13:34 UTC, the incident was resolved.

• Timeline summary:

  • Non-compliance start date: 2025-08-20 14:52 UTC
  • Non-compliance identified date: 2025-08-26 10:28 UTC
  • Non-compliance end date: 2025-08-26 13:34 UTC

• Relevant policies:

  • Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates version 2.1.5 Section 4.9.7.: “CRLs MUST be available via a publicly-accessible HTTP URL (i.e., "published").”
  • CCADB Policy section 6.2: Under normal operating conditions, the CRL URLs provided by CAs in accordance with this section MUST be available such that relying parties are able to successfully retrieve the current CRL every 4 hours.

• Source of incident disclosure: Self Reported

Impact

• Total number of certificates: In total, 3 CRLs the OCSP endpoint for 3 Subordinate CAs were affected. Combined, these 3 Subordinate CAs have signed 2 certificates which were indirectly impacted.
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: N/A
• Incident heuristic: The CRLs and OCSP responses served for these three CAs were temporarily impacted.
• Was issuance stopped in response to this incident, and why or why not?: Yes. Certificate issuance by the three affected Subordinate CAs was halted 16 minutes after the incident was discovered.
• Analysis: N/A
• Additional considerations: N/A

Timeline

All times are in UTC.

• 2025-05-14:
  o 10:46 We create an internal ticket for the creation of 3 new, internally operated, Subordinate CAs for a specific customer, as part of our migration to our new single purpose hierarchies.
• 2025-08-12:
  o 09:00 Requested customer approval for the certificate profile of the to-be-issued Subordinate CAs
• 2025-08-18:
  o 08:00 Received customer approval for the certificate profile.
  o 08:12 All required internal and external reviews and approvals for the issuance of have been completed. The Subordinate CAs are deemed ready for our next offline signing ceremony.
• 2025-08-20:
  o 12:01 The offline signing ceremony is performed.
  o 14:52 The newly issued CA certificates are disclosed in CCADB and our internal ticket.
• 2025-08-25:
  o 11:37 Details to utilize the Subordinate CAs are sent to the customer.
• 2025-08-26:
  o 05:53 A first certificate is issued by one of the newly issued Subordinate CAs.
  o 06:20 A second certificate is issued by another one of the newly issued Subordinate CAs.
  o 10:28 On https://crt.sh/mozilla-disclosures#disclosureincomplete we notice that CRLs issued by the 3 newly issued Subordinate CAs are unavailable and returning a 404 response code.
  o 10:39 We confirm the OCSP responder URL for the Subordinate CAs responds with an “unauthorized” response for valid certificates.
  o 10:42 A decision is made to disable certificate issuance by the affected Subordinate CAs to limit the scope of the incident.
  o 10:43 We reach out to the customer, who operate a proxy for the CRL and OCSP endpoints disclosed and included within issued certificates, notifying them of the problem and requesting they investigate.
  o 10:44 Issuance by the affected Subordinate CAs is halted.
  o 10:56 Customer confirms they are working on the issue.
  o 12:30 The CRL endpoints start working correctly.
  o 13:34 Correct OCSP responses are being returned, resolving the ongoing incident.
  o 15:18 We open this bug.

Related Incidents

Bug	Date	Description
1855997	2023-09-29	Both bugs have the commonality of showing CRL URLs responding with a 40X response code, rather than a 200 response code. It appears the root causes do not relate to each other.
1895312	2024-05-06	Both bugs have the commonality of CRL URLs and OCSP responses being unavailable from a service delivery level. In the related bug, the root cause was due to hardware failure, which differs from this bug.
1908690	2024-07-18	The related bug caused the unavailability of a subset of CRLs due to the decommissioning of a deprecated system. While the symptoms are alike, the root cause differs compared to this bug.
1909203	2024-07-22	The related bug caused the unavailability of CRLs and OCSP due to a network outage. While the symptoms are alike, the root cause differs compared to this bug.
1963778	2025-05-01	The related bug caused the unavailability of CRLs and OCSP due to a nation-wide power outage. Lack of a fallback site in a different nation as the root cause, shows the root cause of the related bug is different from that of this bug.

Root Cause Analysis

Contributing Factor 1: Incorrect assumption and review of CDN setup

• Description:

During the approval process for the Subordinate CAs, it was requested to utilize crl.crlocsp.cn and ocsp.crlocsp.cn as domain names for OCSP and CRL services provided by the applicable CAs. Sectigo provides this option for a select number of SubCAs which mainly target the Asia market. These endpoints allow for OCSP and CRLs to be served through a China based CDN.

To note: Sectigo only provides this as an option to be included with certificates issued by the affected Subordinate CAs. Any CA certificate itself will include Sectigo’s main domain endpoints.

As part of the approval of a Subordinate CA, we verify that FQDNs for these services are in working condition. Since both crl.crlocsp.cn and ocsp.crlocsp.cn were already actively service CRL and OCSP traffic for a minor set of SubCAs, this was deemed valid.

During the incident however, we discovered that the CDN proxy does not automatically proxy data for other Subordinate CAs; rather, these are configured to only allow specific CRL URLs and OCSP responses for specific Subordinate CAs to pass through. We incorrectly assumed and did not verify that the proxy would forward all CRL and OCSP traffic.

• Timeline: N/A. See main incident timeline.
• Detection: This contributing factor was detected during contact with the customer, after we had become aware of the incident. As several years had passed since setting up a new Subordinate CA utilizing these endpoints and the documentation around these specific endpoints was lacking, we did not foresee this as a possible issue.
• Interaction with other factors:

Contributing Factor 2: Insufficient documentation for China-based CDN endpoints

• Description:
  As explained in Contributing Factor 1, we made incorrect assumptions during the approval process for these Subordinate CAs. This in part we have determined is due to a lack of documentation for the affected endpoints and their internal mechanism.
• Timeline: N/A. See main incident timeline.
• Detection: This contributing factor was observed as leading up to our incorrect assumptions, as outlined in contributing factor 1.
• Interaction with other factors: The lack of documentation led to incorrect assumptions being made during the approval process.

Contributing Factor 3: Insufficient communication and instructions prior to CA enablement and disclosure

• Description:
  Prior to enabling and disclosing the newly issued Subordinate CAs, based on the previous assumptions, we did not send clear instructions to the customer to enable proxying of OCSP responses and specific CRLs on the CDN.

Had we had this included within our procedure, the incident might not have taken place.

• Timeline: N/A. See main incident timeline.
• Detection: Based on our earlier assumptions, we did not believe there was a need for specific instructions to enable specific URLs within the CDN.
• Interaction with other factors: Due to the previously mentioned incorrect assumptions, were did not notify the CDN service to enable specific new URLs, causing the incident.

Lessons Learned

• What went well:
  • A total of only 2 certificates were issued prior to us being able to halt issuance, limiting the number of certificates without valid CRLs and OCSP responses
  • The incident was resolved within several hours of detection.
• What didn’t go well:

  • We did not have proper documentation available about the workings and configuration of this purpose-specific CDN.

  • Incorrect assumptions were made prior to approving the issuance of new SubCAs.

  • Insufficient instructions were sent to a customer during the approval process.

• Where we got lucky: N/A
• Additional: N/A

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Update Policy Authority Policy to add guidance for the approval of Subordinate CAs with non-Sectigo default CRL and OCSP domains.	Prevent	Contributing Factor 1	The issuance of Subordinate CAs with non-Sectigo default CRL and OCSP domains is not a regular occurrence. Adding proper guidance within the policy on which steps to follow prior to any such approval is expected to reduce the likelihood of this incident re-occurring.	2025-09-30	Ongoing
Document CDN behavior	Prevent	Contributing Factor 2	More precisely documenting the behavior and technical escalation contacts for the applicable CDN used by these Subordinate CAs should provide more insight into its inner workings and further prevent the possibility or a future incident.	2025-09-30	Ongoing
Update customer guidance	Prevent	Contributing Factor 3	Existing customer guidance is targeted towards general use of Subordinate CAs and not sufficient tailored toward the use-case outlined within this bug.	2025-09-30	Ongoing

Appendix

The CRLs and OCSP responses generated by the following CAs were temporarily impacted:

• https://crt.sh/?caid=427494
• https://crt.sh/?caid=427496
• https://crt.sh/?caid=427495

Comment 2

[Martijn Katerbarg]

Note: Comment #1 shows "Preliminary Incident Report" as title. This is however our Full Incident Report.

Comment 3

[Martijn Katerbarg]

Based on our action items, we'd like to request a Next-Update for 2025-09-30

Comment 4

[Martijn Katerbarg]

All our pending action items have been completed on schedule.

Report Closure Summary

• Incident description:

For three recently issued Subordinate CA certificates, the OCSP and CRL endpoints were not properly setup prior to disclosure of CRLs to CCADB and the issuance of leaf certificates.

• Incident Root Cause(s):

As part of the approval of a Subordinate CA, we verify that FQDNs for these services are in working condition. Since both crl.crlocsp.cn and ocsp.crlocsp.cn were already actively serving CRL and OCSP traffic for several other SubCAs, these FQDNs were deemed to be working. During the incident however, we discovered that the CDN proxy does not automatically proxy data for other Subordinate CAs; rather, the proxy is configured to only allow specific CRL URLs and OCSP responses for specific Subordinate CAs to pass through. We incorrectly assumed and did not verify that the proxy would forward all CRL and OCSP traffic.

• Remediation description:

We have updated both internal approval policies and customer guidance for this case, making sure every detail is verified before SubCA issuance.

• Commitment summary:

Sectigo is committed to keeping these changes in place. Additionally we are looking into reducing the number of cases where we use alternative endpoints.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 5

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-10-07.