2050383 - ANF AC: Incident Report - OCSP "unknown" response for CT precertificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Yulier Nuñez]

Preliminary Incident Report

Summary

• Incident description:

  ANF AC identified an OCSP inconsistency affecting a single Certificate Transparency (CT) precertificate. The affected precertificate returned an OCSP status of "does not know this certificate". Investigation determined that the corresponding subscriber certificate had subsequently been issued using a different certificate serial number, leaving the original precertificate published in Certificate Transparency without a corresponding OCSP record.

  On 2026-06-25, ANF AC registered the affected precertificate in the OCSP responder, resolving the issue. A review of historical issuance records identified no additional orphaned precertificates.

  ANF AC is currently performing a full root cause analysis and preparing permanent corrective actions.

• Relevant policies:

  The investigation is ongoing. ANF AC is assessing whether this incident constitutes non-compliance with the CA/Browser Forum Baseline Requirements and will provide a complete assessment in the Full Incident Report.

• Source of incident disclosure:

  The incident was brought to ANF AC's attention through a third-party notification referencing OCSPWatch. The initial notification, received on 2026-06-10, was incorrectly classified as spam by our mail system and was not seen by the operations team. The issue was identified on 2026-06-24 after the reporter added a second ANF AC contact to the email thread, allowing the incident to be immediately escalated and investigated.

Comment 1

[Yulier Nuñez]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000269
• Incident description: ANF AC identified an OCSP inconsistency affecting a single CT precertificate. The precertificate was successfully submitted to CT, but due to a communication timeout during the SCT retrieval workflow, the corresponding issuance attempt was abandoned. A subsequent retry generated a new certificate with a different serial number, leaving the original precertificate without a corresponding OCSP record.
  The affected precertificate was registered in the OCSP responder on 2026-06-25, resolving the issue. A review of historical issuance records confirmed that no additional orphaned precertificates exist.
• Timeline summary:
  • Non-compliance start date: 2025-09-25
  • Non-compliance identified date: 2026-06-24
  • Non-compliance end date: 2026-06-25
• Relevant policies:
  This incident is being disclosed as an operational issue affecting the certificate issuance workflow and OCSP availability for a CT precertificate.*
  Source of incident disclosure:
  Third-party notification referencing OCSPWatch.

---

Impact

• Total number of certificates: 1 precertificate
• Total number of "remaining valid" certificates: 0
• Affected certificate types: TLS server certificate precertificate
• Incident heuristic: OCSPWatch reported "OCSP responder does not know this certificate".
• Was issuance stopped in response to this incident, and why or why not?: No. The investigation confirmed that the issue was limited to a single historical precertificate. No evidence of an ongoing issuance problem or additional affected certificates was found.
• Analysis: The subscriber received a correctly issued certificate. The incident only affected the original precertificate, which remained published in CT without an OCSP record.
• Additional considerations: Historical records were reviewed and no additional orphaned precertificates were identified.

---

Timeline

Date	Event
2025-09-25	Precertificate generated and submitted to CT.
2025-09-25	SCT retrieval workflow timed out before completion.
2025-09-25	Certificate request was retried and issued with a different serial number.
2026-06-10	Third-party notification received but incorrectly classified as spam.
2026-06-24	Incident identified after a second ANF AC contact was added to the notification thread. Investigation started.
2026-06-25 09:06 CEST	Root cause identified.
2026-06-25 09:42 CEST	Affected precertificate registered in OCSP. Issue resolved.

---

Related Incidents

Bug	Date	Description
Bug 1844514	2023-08-07	Similar incident involving an orphaned CT precertificate without a corresponding OCSP record following an interrupted issuance workflow.

---

Root Cause Analysis

Root Cause #1: The issuance workflow was not resilient to communication failures after CT submission

• Description:
  The precertificate was successfully submitted to Certificate Transparency. During SCT retrieval, the issuance workflow timed out. The workflow interpreted this timeout as a failed issuance attempt, although the precertificate had already been published to CT.

When the certificate request was later retried, the workflow started a new issuance instead of recovering the existing one. As a result, a second certificate was generated using a different serial number. The original precertificate remained published in CT without a corresponding OCSP record, producing the OCSPWatch alert.

• Timeline:
  • 2025-09-25 10:15 UTC – The precertificate was successfully generated and submitted to one or more Certificate Transparency logs.
  • 2025-09-25 10:16 UTC – The issuance workflow timed out while waiting for SCT retrieval to complete and treated the issuance attempt as failed.
  • 2025-09-25 10:35 UTC – The certificate request was retried and a new certificate was issued using a different serial number.
  • 2026-06-24 – The orphaned precertificate was identified following a third-party notification referencing OCSPWatch.
  • 2026-06-25 09:42 CEST – The original precertificate was registered in the OCSP responder, resolving the inconsistency.
• Detection:
  The issue was detected through a third-party notification referencing OCSPWatch.
• Interaction with other factors:
  The retry logic assumed that the previous issuance had failed completely, rather than determining whether the precertificate had already been published.
• Root Cause Analysis methodology used:
  Log analysis of the certificate issuance workflow, CT submission process and OCSP registration records.

Root Cause #2: Absence of automated monitoring of OCSPWatch alerts

• Description:
  ANF AC did not have an automated process to continuously monitor OCSPWatch.

Although a third-party notification was received on 2026-06-10, it was incorrectly classified as spam by the mail system and was not seen by the operations team. The incident was identified only after a second ANF AC contact was added to the notification thread.

• Timeline:

• 2026-06-10 – A third-party notification referencing OCSPWatch was received but classified as spam.
• 2026-06-24 – The incident was identified after the reporter added a second ANF AC contact.
• 2026-06-25 – ANF AC decided to implement continuous automated monitoring of OCSPWatch as a preventive control.

• Detection:

The incident was detected through external notification rather than through ANF AC's own monitoring processes.

• Interaction with other factors:
  The workflow defect described in Root Cause #1 created the OCSP inconsistency. The absence of continuous OCSPWatch monitoring delayed its detection until it was reported externally.

---

Lessons Learned

• What went well:
  The issue was quickly identified, analysed and corrected once reported. Historical records confirmed that the incident was isolated.
• What didn’t go well:
  Existing monitoring did not detect orphaned precertificates, and the initial third-party notification was not acted upon because it was classified as spam.
• Where we got lucky:
  The incident affected only a single precertificate. The subscriber certificate was correctly issued and no additional orphaned precertificates were found.
• Additional:
  Future monitoring will no longer depend solely on email notifications from third parties.

---

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Implement automated monitoring of OCSPWatch.	Detect	Root Cause #2	Automatic alert generated for any ANF AC entry.	2026-06-26	Planned
Modify the issuance workflow to prevent retries from generating a different certificate serial number after precertificate creation.	Prevent	Root Cause #1	Verified through functional testing.	2026-07-03	Planned
Implement reconciliation for interrupted issuance workflows.	Prevent	Root Cause #1	Interrupted workflows resume existing issuance instead of creating a new one.	2026-07-03	Planned
Review SCT retrieval timeout and retry handling.	Prevent	Root Cause #1	Timeout scenarios tested without creating orphaned precertificates.	2026-07-03	Planned

Comment 2

[Yulier Nuñez]

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Implement automated monitoring of OCSPWatch.	Detect	Root Cause #2	Automatic alert generated for any ANF AC entry.	2026-06-26	Complete

This action item has been completed.
We have integrated automated OCSPWatch monitoring into our OCSP service.
The service periodically queries the OCSPWatch API (https://web.api.sslmate.com/ocspwatch/problems) and checks for any reported issues affecting ANF AC. If an issue affecting ANF AC is detected, an alert is automatically sent to the operations team for immediate investigation.
This ensures that future OCSPWatch findings affecting ANF AC are detected promptly.

Comment 3

[chrome-root-program]

(In response to Comment 2)

Thank you for the update.

While we appreciate ANF AC taking steps to monitor community-reported issues, relying on a third-party ecosystem monitor (OCSP Watch) as the primary alerting mechanism for operational infrastructure should not be considered a complete or systemic remediation.

Integrating with the OCSP Watch API seems to outsource ANF AC's primary detective controls to a third party. While community tools like OCSP Watch are without question beneficial for Web PKI ecosystem accountability, integrity, and transparency - they are not a substitute for a CA implementing its own independent, comprehensive infrastructure monitoring.

We agree with the principle often repeated by some ecosystem members - "don't blame the linter." We understand the phrase to mean a CA cannot deflect responsibility for misissuance by pointing out that a third-party linting tool failed to catch an actionable finding. The same logic applies here to OCSP (and CRL) availability and correctness. Relying solely on a third-party tool introduces a precarious external dependency. If the tooling experiences downtime, changes its API, or simply fails to detect a specific edge-case failure in a CA’s infrastructure, the CA remains fully responsible for the resulting non-compliance.

Could ANF AC please address the following:

(Q1) Can you share more detail related to how ANF AC is polling OCSP Watch? (For example, what does "periodically" mean - and how does this frequency ensure adherence to the TLS BRs, ANF AC's policies, and other policies across the ecosystem?)

(Q2) Aside from polling the OCSP Watch API, what internal, independent monitoring and alerting has ANF AC implemented (or plans to implement) to proactively detect OCSP service degradation, outages, or malformed responses?

(Q3) How is ANF AC monitoring the internal health of its OCSP infrastructure - such as database query performance, responder error logs, or network layer issues - so that the operations team is alerted before an external timeout or failure occurs?

(Q4) If the OCSP Watch service were to become unavailable tomorrow, what specific internal controls does ANF AC rely on to guarantee it would independently detect an OCSP issue?

Comment 4

[Andrew Ayer]

FYI, https://web.api.sslmate.com/ocspwatch/problems is an undocumented, internal API, and is not stable. The stable API is the collection of CSV feeds linked from https://sslmate.com/labs/ocsp_watch/. That said, OCSP Watch is operated without any warranties or guarantees and as Comment 3 says, is not a substitute for the CA's own monitoring.

Comment 5

[Yulier Nuñez]

Thank you for the feedback.

We acknowledge that OCSPWatch should not be considered the primary monitoring mechanism, but rather an additional external validation mechanism.

The OCSPWatch integration was implemented as an immediate corrective action to improve visibility of externally reported issues. The current implementation performs periodic checks every 5 minutes to ensure timely detection. As discussed below, we have also added further internal detective controls together with the corresponding Action Item.

Regarding your questions:

Q2 - Q4

ANF AC already performs internal operational monitoring of the OCSP infrastructure. This includes application error logging with automatic alerts, monitoring of service metrics (including CPU utilisation, memory consumption and response times), and monitoring of database health and performance to detect operational issues. Operational alerts are generated automatically when application errors occur or when monitored infrastructure metrics exceed defined thresholds.

These existing controls are intended to detect operational issues affecting the OCSP responder itself and would not have detected the error that occurred in this incident. The failure occurred earlier in the issuance workflow, where the serial number corresponding to the published precertificate was not present in the OCSP responder. The issuance request was subsequently retried, resulting in a new precertificate and the successful issuance of the final certificate with a different serial number. As a result, the OCSP responder continued to operate correctly, while the original precertificate remained orphaned.

To address this specific failure mode, we have added two complementary corrective actions. First, the issuance workflow will be modified so that the serial number is successfully registered in the OCSP responder before the corresponding precertificate is published to the CT logs.

Second, newly issued precertificates will be tracked in the issuance system and checked against the OCSP responder every 5 minutes until the corresponding serial number is verified as available through OCSP. If the serial number is not available within the expected timeframe, an operational alert will be generated.

In addition, a full consistency check of previously issued precertificates will be performed every 24 hours to detect any historical or later-introduced discrepancy.

In summary, three layers:

1. Internal operational monitoring of the OCSP infrastructure.
2. End-to-end verification that issued certificates and precertificates are correctly represented in the OCSP responder.
3. OCSPWatch as an external validation layer.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Modify the issuance workflow so that the serial number is successfully registered in the OCSP responder before the corresponding precertificate is published to the CT logs.	Prevent	Root Cause #1	A published precertificate cannot exist unless its corresponding serial number has already been registered in the OCSP responder.	2026-07-09	Planned
Implement independent end-to-end verification to ensure that issued certificates and precertificates are correctly registered and available in the OCSP responder, including periodic validation and alerting on discrepancies.	Detect	Root Cause #1	Automatic alerts are generated whenever a certificate or precertificate expected to be present in the OCSP responder is missing or returns an unexpected OCSP status.	2026-07-09	Planned

Comment 6

[Pablo Díaz]

We would like to provide an update regarding the Action Items of Comment 5 due today (2026-07-09).

Action Item 1:
This Action Item has been completed. The issuance workflow has been modified so that the precertificate is always registered in the OCSP responder before any publication attempt to CT logs is performed. As a result, a published precertificate can no longer exist without its corresponding serial number already being present in the OCSP responder.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Modify the issuance workflow so that the serial number is successfully registered in the OCSP responder before the corresponding precertificate is published to the CT logs.	Prevent	Root Cause #1	A published precertificate cannot exist unless its corresponding serial number has already been registered in the OCSP responder.	2026-07-09	Complete

Action Item 2:
Implementation is in its final stage. The independent end-to-end verification mechanism has been developed and is currently undergoing final validation.

The mechanism performs automated verification of certificates and precertificates expected to be present in the OCSP responder. Precertificates that have not yet resulted in an issued certificate are checked every 5 minutes, while issued certificates are verified daily. Any missing registration or unexpected OCSP status is treated as a discrepancy, generating an operational alert. When a missing OCSP registration is detected, the mechanism also attempts to automatically register the corresponding certificate in the OCSP responder.

Final deployment is expected tomorrow, after which this Action Item will be marked as completed.

Comment 7

[Pablo Díaz]

Update: Action Item 2 has been completed.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Implement independent end-to-end verification to ensure that issued certificates and precertificates are correctly registered and available in the OCSP responder, including periodic validation and alerting on discrepancies.	Detect	Root Cause #1	Automatic alerts are generated whenever a certificate or precertificate expected to be present in the OCSP responder is missing or returns an unexpected OCSP status.	2026-07-09	Complete

All Action Items associated with this incident have now been completed. We believe the incident has been fully addressed and remain available to provide any additional information or clarification if required.

Comment 8

[incident-reporting]

This report has gone stale. If it is ready for closure, please file a Closure Report.

Comment 9

[Pablo Díaz]

Report Closure Summary

• Incident description:
  OCSP inconsistency affecting a single CT precertificate following an interrupted issuance workflow. The original precertificate was published in CT without its corresponding serial number registered in the OCSP responder.

• Incident Root Cause(s):
  Our issuance workflow was not resilient to communication failures during SCT retrieval. After the timeout, the workflow treated the issuance attempt as completely failed and allowed a retry to generate a new serial number instead of recovering the existing precertificate. We also did not have an internal end-to-end control capable of detecting that a precertificate published in CT was missing from the OCSP responder.

• Remediation description:
  The affected precertificate was registered in the OCSP responder. The issuance workflow was modified to ensure that the certificate serial number is successfully registered in the OCSP responder before publication of the corresponding precertificate to the CT logs. An end-to-end verification mechanism was implemented to periodically verify the presence of issued certificates and precertificates in the OCSP responder, generate operational alerts upon discrepancies, and automatically register missing OCSP entries when applicable. Automated monitoring of OCSPWatch notifications was also implemented as an additional external validation layer.

• Commitment summary:
  There are no ongoing commitments beyond the completed Action Items.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 10

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-08-05.