Closed Bug 2053131 Opened 3 months ago Closed 2 months ago

TunTrust: OCSP responder "Unknown" of one Pre-certificate

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: pki, Assigned: pki)

Details

(Whiteboard: [ca-compliance] [ocsp-failure])

Preliminary Incident Report

Summary

  • Incident description:
    On July 5, 2026, at 23:55 UTC, we received a report identifying an OCSP issue affecting a Pre-Certificate (Serial Number: 77:14:79:2d:0e:69:33:b6:fd:45:0f:89:ec:a4:53:97:c2:a7:c0:f1). Following receipt of the report, we initiated an investigation, which confirmed that the Pre-Certificate had been successfully logged to Certificate Transparency. However, only one CT log successfully returned a Signed Certificate Timestamp (SCT), and the required SCT validation threshold was not met. As a result, the certificate issuance workflow was aborted and the final end-entity certificate was not issued.
    Additionally, the Pre-Certificate was not published and its status was "unknown" in the OCSP responder.
    We then generated a second Pre-Certificate (Serial Number: 41:f1:ff:f0:53:71:7d:8f:30:c2:3d:a3:2a:65:ea:e6:c1:89:d4:4f) and submitted it to the CT logs. Again, we did not receive enough SCTs, so the issuance process was halted. However, unlike the first Pre-Certificate, this one was properly registered in our OCSP system. It returned a "good" OCSP status, even though the final certificate was not issued.
    Finally, we generated a third Pre-Certificate (Serial Number: 2e:46:b3:95:6b:95:52:88:a2:2a:87:43:76:6a:b9:7f:90:9b:38:23). This time, we successfully received SCTs from three CT logs. The issuance process completed normally, resulting in a valid final certificate.
    We have reviewed our issuance logs and confirmed that only this one Pre-Certificate, with the "Unknown" status, was affected by this issue. The incident is contained. We have ensured that the first Pre-Certificate is now properly registered in the OCSP responder to provide an authoritative status. We are currently performing a root cause analysis to understand why the OCSP registration step was bypassed for the first Pre-Certificate and will provide a Full Incident Report within 14 days.

  • Relevant policies:
    CA/Browser Forum TLS Baseline Requirements v2.2.8, Section 4.9.9 (On-line revocation/status checking availability)
    TunTrust PKI CP/CPS v06.1.3, Section 4.9.9 (On-line revocation/status checking availability) [Link: https://www.tuntrust.tn/sites/default/files/Ressources/CPCPS-TunTrustPKI.pdf]

  • Source of incident disclosure:
    Third-party reported.

Assignee: nobody → pki
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ocsp-failure]

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000785

  • Incident description:
    On July 5, 2026, at 23:55 UTC, TunTrust received a third-party notification from SSLMate OCSPWatch reporting that the OCSP responder was returning the following error for a precertificate:
    "OCSP responder does not know this certificate."
    Following receipt of the report, TunTrust immediately initiated an internal investigation.
    The investigation confirmed that the affected precertificate (Serial Number: 77:14:79:2d:0e:69:33:b6:fd:45:0f:89:ec:a4:53:97:c2:a7:c0:f1) had been successfully generated and submitted to the configured Certificate Transparency (CT) logs. However, only one CT log returned a valid Signed Certificate Timestamp (SCT), which was insufficient to satisfy the configured SCT validation threshold. Consequently, the certificate issuance process was correctly aborted, and the corresponding end-entity certificate was not issued.
    Although the certificate issuance process was aborted, the precertificate should still have been registered in the OCSP responder so that an authoritative OCSP response would be available. Instead, the precertificate was not registered in the OCSP responder, causing OCSP requests for the precertificate to return an "unknown" status.
    To continue the certificate issuance process, a second precertificate (Serial Number: 41:f1:ff:f0:53:71:7d:8f:30:c2:3d:a3:2a:65:ea:e6:c1:89:d4:4f) was generated and submitted to the configured CT logs. Again, the required SCT validation threshold was not met, and the issuance process was aborted. During this second issuance attempt, however, the precertificate was successfully registered in the OCSP responder and returned an authoritative "good" status, despite the absence of a final end-entity certificate.
    A third precertificate (Serial Number: 2e:46:b3:95:6b:95:52:88:a2:2a:87:43:76:6a:b9:7f:90:9b:38:23) was subsequently generated. This issuance successfully obtained Signed Certificate Timestamps from three CT logs, allowing the certificate issuance process to complete successfully and resulting in the issuance of the final end-entity certificate.
    Based on our investigation, we identified that this incident is related to a recent upgrade performed on the application hosting the CA.
    TunTrust reviewed all certificate issuance records generated following the June 23, 2026 application upgradeand confirmed that only this single precertificate experienced the OCSP publication failure. No additional precertificates exhibiting the same behavior were identified. This incident affected OCSP availability for one aborted precertificate only. No end-entity certificate was incorrectly issued, no subscriber certificates required revocation, and no subscribers certificate were impacted. The incident has therefore been contained.

  • Timeline summary:

    • Non-compliance start date: June 30, 2026
    • Non-compliance identified date: July 5, 2026,
    • Non-compliance end date: July 6, 2026,

  • Relevant policies:
    CA/Browser Forum TLS Baseline Requirements v2.2.8, Section 4.9.9 (On-line revocation/status checking availability)
    TunTrust PKI CP/CPS v06.1.3, Section 4.9.9 (On-line revocation/status checking availability)

-Source of incident disclosure:
Third-party reported.

Impact

  • Total number of certificates: One Pre-Certficate

  • Affected end-entity certificates: 0

  • Subscribers impacted: 0

  • Certificates requiring revocation: 0

  • Total number of "remaining valid" certificates: 0

  • Affected certificate types: This incident affects an OV SSL Pre-Certificate

  • Incident heuristic: The link to the affected certificate on crt.sh : https://crt.sh/?id=27587191963

  • Was issuance stopped in response to this incident, and why or why not?: No. Certificate issuance was not suspended because the defect affected the OCSP publication of an aborted precertificate rather than certificate validation or successful certificate issuance. Following identification of the issue, TunTrust reviewed subsequent certificate issuances, verified OCSP registration for the affected precertificate, and monitored the generation and OCSP publication of every subsequently generated precertificate.

  • Analysis: N/A

  • Additional considerations: N/A

Timeline

Date Description
June 30, 2026 07:17 UTC Precertificate (Serial Number: 77:14:79:2d:0e:69:33:b6:fd:45:0f:89:ec:a4:53:97:c2:a7:c0:f1) was generated and submitted to the configured Certificate Transparency (CT) logs. The precertificate was not published to the OCSP responder and its OCSP status was "Unknown".
July 5, 2026 23:55 UTC TunTrust received an email notification from a third party reporting the issue.
July 6, 2026 06:30 UTC TunTrust initiated an internal investigation to identify the causes of the problem.
July 6, 2026 13:23 UTC The precertificate (Serial Number: 77:14:79:2d:0e:69:33:b6:fd:45:0f:89:ec:a4:53:97:c2:a7:c0:f1) was successfully published to the OCSP responder. The error reported by SSLMate OCSPWatch was no longer observed.
July 7, 2026 14:39 UTC TunTrust opened a Mozilla Bugzilla incident report.

TunTrust received the third-party notification on 2026-07-05 at 23:55 UTC and opened the public Mozilla Bugzilla incident on 2026-07-07 at 14:39 UTC, approximately 39 hours after receiving the report. During this period, TunTrust confirmed the scope of the incident, restored an authoritative OCSP response for the affected precertificate, and verified that no additional precertificates were affected before publicly disclosing the incident.

Related Incidents

Bug Date Description
Bug 2047952 2026-06-16 KIR: OCSP responder does not return status for precertificate
Bug 1844514 2023-07-20 MICROSEC: Incident report - No OCSP status response for 2 Precertificates
Bug 1858965 2023-10-13 VikingCloud: Incorrect OCSP Response

Root Cause Analysis

Contributing Factor #1: Insufficient monitoring

  • Description:
    At the time of the incident, TunTrust does not monitor the pre-certificate process.
  • Timeline:
    This monitoring gap existed prior to the incident.
  • Detection:
    The occurrence of this incident allowed us to identify a gap in the monitoring of this process.
  • Interaction with other factors:
    The lack of monitoring allowed the publication failure caused by Contributing Factor #2 to remain undetected until it was identified by an external party.

Contributing Factor #2 Application Upgrade Follow-up

  • Description:
    This incident occurred following a recent upgrade of the application hosting the CA. The investigation determined that the service responsible for processing aborted precertificate issuances did not successfully complete the precertificate process. In addition, the change management process did not include a post-upgrade follow-up of the aborted precertificate workflow. Consequently, the issue remained undetected after deployment until it was reported by a third party.
  • Timeline:
    Since the application upgrade on June 23rd 2026.
  • Detection:
    The issue was identified through analysis of the application log files.
  • Interaction with Other Factors:
    The application upgrade, combined with the lack of monitoring for pre-certificate publication, contributed to the failure to detect the issue.

Lessons Learned

  • What went well : The issue was promptly investigated following the third-party notification. TunTrust confirmed that the incident was limited to a single aborted precertificate, restored an authoritative OCSP response for the affected precertificate, and verified that no additional precertificates generated after the application upgrade were affected.
  • What didn't go well:
    The absence of monitoring for the precertificate process prevented the issue from being detected internally before it was reported by a third party. In addition, the post-upgrade follow-up activities did not include verification of the aborted precertificate workflow.
  • Where we got lucky: The issue was limited to a single aborted precertificate generated immediately following the application upgrade. No end-entity certificate was incorrectly issued, no subscribers were impacted, and no certificate revocation was required.
  • Additional: N/A

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Publish the affected precertificate to the OCSP responder and verify that the OCSP responder returns an authoritative status. Review historical issuance records to confirm that no additional precertificates were affected. Correct Contributing Factor #2 The affected precertificate is successfully registered in the OCSP responder, an authoritative OCSP response is returned, and the review confirms that no additional precertificates were impacted. July 6, 2026 Complete
Implement a monitoring of the Pre-certifcate process. Prevent Contributing Factor #1 Monitoring implemented and validated by successfully generating alerts for simulated publication failures. July 17, 2026 Complete
Update the Change Management Procedure to include a post-upgrade follow-up step following update to the application hosting the CA. Prevent Contributing Factor #2 No recurrence of similar incidents. July 24, 2026 Ongoing

UPDATE: The status of the last Action Item has been updated to "Complete".

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000785

  • Incident description:
    On July 5, 2026, at 23:55 UTC, TunTrust received a third-party notification from SSLMate OCSPWatch reporting that the OCSP responder was returning the following error for a precertificate:
    "OCSP responder does not know this certificate."

    Following receipt of the report, TunTrust immediately initiated an internal investigation.

    The investigation confirmed that the affected precertificate (Serial Number: 77:14:79:2d:0e:69:33:b6:fd:45:0f:89:ec:a4:53:97:c2:a7:c0:f1) had been successfully generated and submitted to the configured Certificate Transparency (CT) logs. However, only one CT log returned a valid Signed Certificate Timestamp (SCT), which was insufficient to satisfy the configured SCT validation threshold. Consequently, the certificate issuance process was correctly aborted, and the corresponding end-entity certificate was not issued.

    Although the certificate issuance process was aborted, the precertificate should still have been registered in the OCSP responder so that an authoritative OCSP response would be available. Instead, the precertificate was not registered in the OCSP responder, causing OCSP requests for the precertificate to return an "unknown" status.

    To continue the certificate issuance process, a second precertificate (Serial Number: 41:f1:ff:f0:53:71:7d:8f:30:c2:3d:a3:2a:65:ea:e6:c1:89:d4:4f) was generated and submitted to the configured CT logs. Again, the required SCT validation threshold was not met, and the issuance process was aborted. During this second issuance attempt, however, the precertificate was successfully registered in the OCSP responder and returned an authoritative "good" status, despite the absence of a final end-entity certificate.

    A third precertificate (Serial Number: 2e:46:b3:95:6b:95:52:88:a2:2a:87:43:76:6a:b9:7f:90:9b:38:23) was subsequently generated. This issuance successfully obtained Signed Certificate Timestamps from three CT logs, allowing the certificate issuance process to complete successfully and resulting in the issuance of the final end-entity certificate.

    Based on our investigation, we identified that this incident is related to a recent upgrade performed on the application hosting the CA.

    TunTrust reviewed all certificate issuance records generated following the June 23, 2026 application upgrade and confirmed that only this single precertificate experienced the OCSP publication failure. No additional precertificates exhibiting the same behavior were identified. This incident affected OCSP availability for one aborted precertificate only. No end-entity certificate was incorrectly issued, no subscriber certificates required revocation, and no subscribers were impacted. The incident has therefore been contained.

  • Timeline summary:

    • Non-compliance start date: June 30, 2026
    • Non-compliance identified date: July 5, 2026
    • Non-compliance end date: July 6, 2026

  • Relevant policies:

  • Source of incident disclosure:
    Third-party reported.

Impact

  • Total number of certificates: One Pre-Certificate
  • Affected end-entity certificates: 0
  • Subscribers impacted: 0
  • Certificates requiring revocation: 0
  • Total number of "remaining valid" certificates: 0
  • Affected certificate types: This incident affects an OV SSL Pre-Certificate
  • Incident heuristic: The link to the affected certificate on crt.sh: https://crt.sh/?id=27587191963
  • Was issuance stopped in response to this incident, and why or why not?
    No. Certificate issuance was not suspended because the defect affected the OCSP publication of an aborted precertificate rather than certificate validation or successful certificate issuance. Following identification of the issue, TunTrust reviewed subsequent certificate issuances, verified OCSP registration for the affected precertificate, and monitored the generation and OCSP publication of every subsequently generated precertificate.
  • Analysis: N/A
  • Additional considerations: N/A

Timeline

Date Description
June 30, 2026 07:17 UTC Precertificate (Serial Number: 77:14:79:2d:0e:69:33:b6:fd:45:0f:89:ec:a4:53:97:c2:a7:c0:f1) was generated and submitted to the configured Certificate Transparency (CT) logs. The precertificate was not published to the OCSP responder and its OCSP status was "Unknown".
July 5, 2026 23:55 UTC TunTrust received an email notification from a third party reporting the issue.
July 6, 2026 06:30 UTC TunTrust initiated an internal investigation to identify the causes of the problem.
July 6, 2026 13:23 UTC The precertificate (Serial Number: 77:14:79:2d:0e:69:33:b6:fd:45:0f:89:ec:a4:53:97:c2:a7:c0:f1) was successfully published to the OCSP responder. The error reported by SSLMate OCSPWatch was no longer observed.
July 7, 2026 14:39 UTC TunTrust opened a Mozilla Bugzilla incident report.

TunTrust received the third-party notification on 2026-07-05 at 23:55 UTC and opened the public Mozilla Bugzilla incident on 2026-07-07 at 14:39 UTC, approximately 39 hours after receiving the report. During this period, TunTrust confirmed the scope of the incident, restored an authoritative OCSP response for the affected precertificate, and verified that no additional precertificates were affected before publicly disclosing the incident.

Related Incidents

Bug Date Description
Bug 2047952 2026-06-16 KIR: OCSP responder does not return status for precertificate
Bug 1844514 2023-07-20 MICROSEC: Incident report - No OCSP status response for 2 Precertificates
Bug 1858965 2023-10-13 VikingCloud: Incorrect OCSP Response

Root Cause Analysis

Contributing Factor #1: Insufficient monitoring

  • Description:
    At the time of the incident, TunTrust does not monitor the pre-certificate process.

  • Timeline:
    This monitoring gap existed prior to the incident.

  • Detection:
    The occurrence of this incident allowed us to identify a gap in the monitoring of this process.

  • Interaction with other factors:
    The lack of monitoring allowed the publication failure caused by Contributing Factor #2 to remain undetected until it was identified by an external party.

Contributing Factor #2: Application Upgrade Follow-up

  • Description:
    This incident occurred following a recent upgrade of the application hosting the CA. The investigation determined that the service responsible for processing aborted precertificate issuances did not successfully complete the precertificate process. In addition, the change management process did not include a post-upgrade follow-up of the aborted precertificate workflow. Consequently, the issue remained undetected after deployment until it was reported by a third party.

  • Timeline:
    Since the application upgrade on June 23rd 2026.

  • Detection:
    The issue was identified through analysis of the application log files.

  • Interaction with Other Factors:
    The application upgrade, combined with the lack of monitoring for pre-certificate publication, contributed to the failure to detect the issue.

Lessons Learned

  • What went well:
    The issue was promptly investigated following the third-party notification. TunTrust confirmed that the incident was limited to a single aborted precertificate, restored an authoritative OCSP response for the affected precertificate, and verified that no additional precertificates generated after the application upgrade were affected.

  • What didn't go well:
    The absence of monitoring for the precertificate process prevented the issue from being detected internally before it was reported by a third party. In addition, the post-upgrade follow-up activities did not include verification of the aborted precertificate workflow.

  • Where we got lucky:
    The issue was limited to a single aborted precertificate generated immediately following the application upgrade. No end-entity certificate was incorrectly issued, no subscribers were impacted, and no certificate revocation was required.

  • Additional:
    N/A

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Publish the affected precertificate to the OCSP responder and verify that the OCSP responder returns an authoritative status. Review historical issuance records to confirm that no additional precertificates were affected. Correct Contributing Factor #2 The affected precertificate is successfully registered in the OCSP responder, an authoritative OCSP response is returned, and the review confirms that no additional precertificates were impacted. July 6, 2026 Complete
Implement a monitoring of the Pre-certifcate process. Prevent Contributing Factor #1 Monitoring implemented and validated by successfully generating alerts for simulated publication failures. July 17, 2026 Complete
Update the Change Management Procedure to include a post-upgrade follow-up step following update to the application hosting the CA. Prevent Contributing Factor #2 No recurrence of similar incidents. July 24, 2026 Complete

This report has gone stale. If it is ready for closure, please file a Closure Report.

Flags: needinfo?(pki)

Report Closure Summary

  • Incident description:
    On July 5, 2026, TunTrust received a third-party notification indicating that the OCSP responder returned the status "OCSP responder does not know this certificate" for an aborted OV SSL precertificate. The affected precertificate was subsequently published to the OCSP responder on July 6, 2026, restoring an authoritative OCSP response. A comprehensive review of all certificate issuance records generated after the June 23, 2026 application upgrade confirmed that no additional precertificates were affected. No end-entity certificates were incorrectly issued, no certificates required revocation, and no subscribers were impacted. The incident has therefore been contained.

  • Incident Root Cause(s):
    The root cause was an application issue introduced following the June 23, 2026 upgrade of the CA application. The component responsible for processing aborted precertificate issuances did not successfully complete the OCSP publication step for one aborted precertificate.
    A contributing factor was that the existing monitoring capabilities did not include supervision of the precertificate publication workflow. In addition, the Change Management Procedure did not include verification of the aborted precertificate workflow following application upgrades, allowing the issue to remain undetected until it was reported by an external party.

  • Remediation description:
    The affected precertificate was successfully published to the OCSP responder, restoring an authoritative OCSP status response. A complete review of all certificate issuances performed after the application upgrade confirmed that this was an isolated incident affecting only a single aborted precertificate. Monitoring of the precertificate publication workflow has been implemented to automatically detect publication failures. The Change Management Procedure has also been updated to require verification of both successful and aborted precertificate workflows following any upgrade of the application hosting the CA.

  • Commitment summary:

  1. Continue strengthening internal monitoring capabilities to improve the early detection of anomalies affecting certificate lifecycle operations, including precertificate processing and OCSP publication.
  2. Continue improving internal audit methodologies by incorporating periodic verification of precertificate processing and OCSP publication as part of routine compliance assessments.
  3. Reinforce staff awareness through periodic knowledge-sharing sessions on certificate lifecycle requirements, CA/Browser Forum Baseline Requirements, and lessons learned from operational incidents.
Flags: needinfo?(pki)

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-08-18.

Whiteboard: [ca-compliance] [ocsp-failure] → [close on 2026-08-18] [ca-compliance] [ocsp-failure]
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Whiteboard: [close on 2026-08-18] [ca-compliance] [ocsp-failure] → [ca-compliance] [ocsp-failure]
You need to log in before you can comment on or make changes to this bug.