2048370 - Sectigo: Delay in some OCSP response signing due to application restart loop

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Preliminary Incident Report

Summary

• Incident description:

On June 15th, 2026 we noticed spikes within our OCSP replication system. While investigating this, we found the cause to be due to our OCSP signing application (CertStatus) k8s pod restarting every minute.

Within the restart loop, we noticed that any backlog of OCSP responses to be signed for newly issued certificates was being processed timely. The replication lag never made it above roughly 5 minutes. As such, the immediate issue was resolved, but no formal incident was declared based on the data available.

On June 17th at 01:43 UTC we received a support ticket noting an “unauthorized” OCSP response was received for a certificate issued on June 15th. Furthermore we received a CPR at 05:40, also reporting “unauthorized” OCSP responses.

Preliminary investigation shows that OCSP responses for these certificates were at least partially processed by CertStatus and can confirm OCSP responses were signed by our HSMs, but some of these never made it to publication.

We have a failsafe for such a case, which is our continues refresh signing of OCSP responses. This should under normal circumstances sign a new response within a few minutes. However, due to the restart loop of the application, a large backlog in this queue had been raising over the past few days, further delaying the signing and publication of responses.

We are currently further investigating this incident.

• Relevant policies:

  • TLS Baseline Requirements version 2.2.6 - Section 4.9.9

• Source of incident disclosure: Third Party Reported

Comment 1

[Tim Callan]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000016
• Incident description:

On June 15th, 2026 we noticed spikes within our OCSP replication system. Investigation determined the cause to be that our OCSP signing application (certstatus-manager) k3s pod was restarting every few minutes.

During the time this restart loop was occurring, we observed that the system kept building a short-term OCSP response signing backlog for newly issued certificates. We initially perceived that the slowdown was a relatively minor inconvenience that did not create a formal incident. Once we had fixed the root cause of the slowdown, we believed the episode to be over.

On June 17th at 01:43 UTC we received a support ticket noting an “unauthorized” OCSP response for a certificate issued on June 15th. We subsequently received a CPR at 05:40, also reporting “unauthorized” OCSP responses.

Upon investigation of these reports, we determined that OCSP responses for some newly issued certificates had not been signed successfully. This was our first indication of an intermittent signing problem that had occurred during the slowdown from June 13 to June 16.

Sectigo already has a ”backstop” mechanism in place for this scenario, which is our OCSP refresher signing process. However, due to the large number of restarts of the signing application, the refresher process had gained a significant backlog that it could not fully process each time prior to the next restart. This refresh backlog affected signing of fresh OCSP responses for older certificates as well as signing of OCSP responses for the very small portion of newly issued certificates where the first OCSP response signing attempt had failed. Because we have several days to sign fresh OCSP responses for existing certificates, the building backlog did not create an incident for these certificates. However, with only 15 minutes to sign OCSP responses for new certificates, the processing delay created an incident in this case. As the large number of fresh OCSP responses for older certificates makes up the overwhelming majority of this queue, this volume masked the presence of the noncompliant delays in signing OCSP responses for the relatively few new certificates affected.

• Timeline summary:

  • Non-compliance start date: 2026-06-13 – 03:21:44 UTC.
  • Non-compliance identified date: 2026-06-17 – 07:39 UTC.
  • Non-compliance end date: 2026-06-17 – 16:22 UTC.

Note: The Non-compliance start date lists the date and time when the certstatus-manager pod started the described restart loop. We are unable to determine the precise moment of the first delay beyond 15 minutes in signing a new certificate’s OCSP response. Therefore, we’ve outlined the earliest and latest conceivable timestamps date and time of non-compliance.

• Relevant policies:

  • TLS Baseline Requirements version 2.2.6 - Section 4.9.9

• Source of incident disclosure: Third Party Reported

Impact

• Total number of certificates: N/A. Certificates were not directly affected by this incident, only their OCSP responses.
• Total number of "remaining valid" certificates: N/A.
• Affected certificate types: TLS, S/MIME and Code Signing Certificates were indirectly impacted.
• Incident heuristic: The restart loop of our certstatus-manager application was resolved on 2026-06-15 at 16:45 UTC. Some OCSP responses for certificates issued between the incident start time and this time were affected. Our logs indicate that for the large majority of certificates issued within this timeframe, OCSP responses were provided within the required timeframe.
• Was issuance stopped in response to this incident, and why or why not?: Issuance was not halted. At the moment we became aware of the issue, other actions we had taken had already resolved the issue.
• Analysis: N/A.
• Additional considerations: N/A.

Timeline

All times in UTC.

2026-06-13:

• 03:21:44 The certstatus-manager application, responsible for signing OCSP responses, is shut down automatically by our k3s cluster due to an inability to reach the liveliness endpoint.
• 03:21:45 A new instance of the certstatus-manager application starts automatically. The new instance is shut down after an additional 3 minutes.

From this moment until 2026-06-15 at 16:45, the pod is restarted regularly, counting 534 restarts in total. Logs confirm signings have continued within that time span, but with fewer signings occurring than in normal circumstances. A backlog of OCSP refresh signings starts to grow.

2026-06-15:

• 08:32 We notice regular, minor, replication lags affecting the CertStatus database, occurring every few minutes. The issue is flagged to the responsible teams. This triggers no immediate escalation as the replication lag remains within the allowed limits.
• 08:51 The operations team acknowledges the issue.
• 09:54 We discover the certstatus-manager application is in a “CrashLoopBackOff” state and has been restarting regularly over the last 2 days.
• 10:23 We note the restarts have been occurring due to a liveliness probe from our k3s cluster, which is designed to automatically restart a pod if it is unable to reach the pod itself.
• 10:45 We decide to turn off the liveliness probe on the application to determine if the application will run for a longer period without the check.
• 10:59 The liveliness probe is disabled. The application now remains stable.
• 13:00 We have not yet discovered a direct root cause for the liveliness probe failures. We turn the liveliness probe back on to conduct testing. The pod again starts a restart loop.
• 13:44 We once again turn off the liveliness probe until the underlying issue is resolved.

2026-06-17:

• 01:43 We receive a support request noting OCSP unauthorized responses on a certificate.
• 05:40 We receive a CPR noting OCSP unauthorized responses on two certificates.
• 07:39 We escalate the issue internally, acknowledging unauthorized OCSP responses are seen specifically for a limited number of certificates issued on June 15th.
• 08:14 We reply to the support request noting the issue.
• 10:53 We reply to the CPR confirming the issue.
• 12:28 Our incident analysis concludes that the previous weekend’s slowdown had caused a small portion of OCSP response signatures for newly issued certificates to not be delivered within the expected timeline.

2026-06-22:

• 19:49 Our internal investigation determines the root cause and prevention plan for the restart loop that caused the slowdown.

2026-06-26:

• 20:10 Patching of the bug that created the pod restart loop is complete (second listed action item).

2026-06-29:

• 10:55 Deployment of update to prioritize OCSP response signing for newly issued certificates in the refresh queue (third listed action item).

Related Incidents

Bug	Date	Description
1895312	2024-05-06	The incident relates to a complete unavailability of CRL and OCSP endpoints.
1903823	2024-06-20	The incident happened due to a race condition triggered by simultaneous requests in the database.
1905419	2024-06-28	The incident shows correlation in that delays were identified between signings and publication of OCSP responses.
1905446	2024-06-28	The issue was determined to be due to a missing OCSP response in the CA’s database.
1909203	2024-07-22	The incident relates to a complete unavailability of CRL and OCSP endpoints.
1933353	2024-11-25	A CA deployment caused an issue leading to a timezone difference, causing a discrepancy leading to missing OCSP responses.
1931636	2024-12-02	A failure to complete all sub-tasks of setting up new issuing CAs led to missing OCSP responses.
1946927	2025-02-08	Previous replication delays led to delays in publishing OCSP responses. While improvements were made and have shown to be effective, they did not consider a complete site being out of sync.
1954580	2025-03-17	An SSH host key change led to Postgres WALs no longer being retrieved, causing out-of-date and late OCSP responses
1957140	2025-03-28	High-rate certificate requests lead to intermittent failures to insert newly issued certificates into the database.
1963778	2025-05-01	The incident relates to a complete unavailability of CRL and OCSP endpoints.
1965828	2025-05-12	A combination of configuration settings of various components of the OCSP responder service which in certain runtime situations could lead to the OCSP service being completely unresponsive.
2014590	2026-02-04	OCSP was not functioning for four cross signed ICAs due to missing configuration.

Root Cause Analysis

Contributing Factor #1: Failed storage cleanup, leading to cascading failure

• Description: A bug previously unknown to us caused our third-party storage array not to perform expected cleanup. This led to disk use reaching 100%, which caused a cascading failure across other systems, manifesting as a slowdown in core CA systems and intermittent failure to sign OCSP responses.
• Timeline: 2026-06-13 to 2026-06-15.
• Detection: Internal reports
• Interaction with other factors: Contributing factor #2 prevented our existing systems from gracefully handling this slowdown.

Contributing Factor #2: OCSP refresh incapable of processing backlog in repeat restart scenario

• Description: Due to the size of the OCSP refresh backlog, caused by the frequent system restarts, existing processes were heavily delayed in making their way back to earlier failed signing events. This is in part because our OCSP refresher code did not treat the retry of failed signings as a priority over the regular OCSP refresh signing backlog.
• Timeline: 2026-06-13 to 2026-06-15.
• Detection: Postmortem analysis
• Interaction with other factors: Prevented “backstop” capabilities from processing our backlog in these specific conditions.

Contributing Factor #3: No alert for OCSP signing delay beyond acceptable limits

• Description: Though we were aware of the performance slowdown and took action to address it, the resulting failure to sign some OCSP responses on time did not trigger an alert or warning, and so this failure went undetected until we received external reports.
• Timeline: 2026-06-13 to 2026-06-17.
• Detection: Postmortem analysis
• Interaction with other factors: Prevented earlier detection of noncompliance incident.

Lessons Learned

• What went well:
  • We were able to address the root cause problem quickly and restore performance to expectations.
  • By the time we became aware of the OCSP response signing issue, we already had resolved its root cause.
• What didn’t go well:
  • Existing functionality designed to process OCSP response signing backlog was incapable of prioritizing this specific delay scenario.
  • Though we resolved the original performance slowdown, we were unaware at that time of the resulting noncompliance.
  • Our primary SME had been summoned to jury duty and was out of the office. Though we had redundant coverage, this hampered our ability to fully understand the impact of this failure as quickly as we otherwise would have.
• Where we got lucky:
  • The vast majority of OCSP responses for newly issued certificates were signed promptly, unaffected by the intermittent signing problem and/or the backlogged "backstop."

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Determine reason for frequent restart of certstatus-manager pods	Prevent	Contributing Factor # 1	Understanding the root cause allowed us to determine an immediate solution and establish a go-forward plan.	2026-06-17	Complete
Deploy vendor update to prevent storage disk cleanup failure	Prevent	Contributing Factor # 1	The original root cause of the cascading failure was a bug in our third-party storage solution. We have deployed a patch for this bug.	2026-06-26	Complete
Prioritize “backstop” signing backlog	Prevent	Contributing Factor # 2	Repeated and rapid restarts led to “backstop” signings being stuck behind a large backlog of OCSP response “refresh” signings. Prioritizing “backstop” signing eliminates this possibility.	2026-06-29	Complete
Investigate expanded monitoring and alerting options	Detect	Contributing Factor # 3	We have begun an investigation into how to add monitoring and alerting that would have called our attention to the OCSP response signing delay earlier than the external reports we received.	2026-07-31	Ongoing

Comment 2

[Tim Callan]

While we work on our remaining Action Item, we request a next-update of 2026-07-31.

Comment 3

[Martijn Katerbarg]

As of last week, all our action items have been completed.

Report Closure Summary

• Incident description:
  Between June 13 and June 16, 2026, our OCSP signing application (certstatus-manager) k3s pod was restarting every few minutes, causing initial OCSP responses for some newly issued certificates to not be signed successfully.

  The backstop mechanism in place was itself suffering from a backlog of OCSP responses to sign, and was therefore unable to sign the previously failed signings within the required 15 minutes.

• Incident Root Cause(s):
  A bug in our storage environment led to a cascading failure within systems, eventually causing a growing OCSP response refresher backlog and failed signings for newly issued certificates.

• Remediation description:
  Improved monitoring and alerting to mitigate and detect future problems, as well as improved handling within certstatus-manager to prioritize signing of previously failed initial OCSP response signings over lower-priority, refresher signings.

• Commitment summary:
  We remain committed to researching further improvements to these systems.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 4

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-08-07.