1965828 - SwissSign: OCSP outage

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Roman Fischer]

Preliminary Incident Report

Summary

• Incident description: SwissSign OCSP responder for one CA was not reachable from 2025-05-12T09:23:58Z to 2025-05-12T10:13:56Z
• Relevant policies: TLS BR 4.9.9 states "Effective 2025‐01‐15, an authoritative OCSP response MUST be available (i.e. the responder MUST NOT respond with the “unknown” status) starting no more than 15 minutes"
• Source of incident disclosure: Self reported

A full incident report will be published no later than 2025-05-16.

Comment 1

[Roman Fischer]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000049
• Incident description: SwissSign OCSP responder for one CA platform (ocsp.swisssign.ch) was not reachable from 2025-05-12T09:23:58Z to 2025-05-12T10:13:56Z
• Timeline summary:
  • Non-compliance start date: 2025-05-12T09:23:58Z
  • Non-compliance identified date: 2025-05-12T09:23:58Z
  • Non-compliance end date: 2025-05-12T10:13:56Z
• Relevant policies: TLS BR 4.9.9 states "Effective 2025‐01‐15, an authoritative OCSP response MUST be available (i.e. the responder MUST NOT respond with the “unknown” status) starting no more than 15 minutes"
• Source of incident disclosure: Self reported

Impact

• Total number of certificates: 28 (14 Pre, 14 Final) TLS certificates and 115 S/MIME certificates (as of 2025-05-13) were issued during the OCSP outage
• Total number of "remaining valid" certificates: 143 (as of 2025-05-13)
• Affected certificate types: This incident affects TLS DV and OV and S/MIME MV and SV certificates.
• Incident heuristic: This could be identified by performing an OCSP status check on certificates issued during the incident start date/time until the incident end date/time. After the incident end date/time passed, it was no longer possible for a third party to assemble the full corpus of affected certificates and OCSP responses.
• Was issuance stopped in response to this incident, and why or why not?: No. The certificates themselves are not considered mis issued, merely indirectly affected. As such and due to the estimated short time window of the incident, issuance was not halted.
• Analysis: N/A
• Additional considerations: N/A

Timeline

2023-09-22:
- SwissSign starts productive use of it's new CA plattform

2024-03-30:
- A daily check for OCSP-Watch on sslmate.com is implemented, this did not detect the outage

2025‐01‐15:
- TLS BR 4.9.9 is updated to require OCSP responses to be available no more than 15 minutes after issuance of a certificate

2025-05-12:
- 09:23:58 Monitoring (checks run every minute) detects OCSP outage and sends out alerts to the operations team
- 10:03:00 Operations team informs Compliance about OCSP outage
- 10:13:56 Service is restored
- 13:45:00 Audit body informed
- 14:01:00 Preliminary incident report posted

2025-05-16:
- 11:29:00 Posting of Full incident report

All times are UTC.

Related Incidents

Bug	Date	Description
1622505	2020-03-14	GlobalSign: OCSP Status HTTP 530
1636544	2020-05-08	IdenTrust: OCSP Outage
1662137	2020-08-31	SwissSign: OCSP responder unreachable
1663953	2020-09-09	TunTrust: OCSP unreachable
1677234	2020-11-13	Apple: OCSP availability 2020-11-12
1754593	2022-02-09	IdenTrust: Unavailable CRL and OCSP Responders
1879529	2024-02-09	D-Trust: "unknown" OCSP response for issued certificates
1895312	2024-05-06	TunTrust: CRL and OCSP unavailable
1909203	2024-07-22	Asseco DS / Certum: CP/CPS, Revocation Requests Mechanism, Certificate Problem Report, CRL and OCSP disruption
1954580	2025-03-17	Sectigo: Temporary failure to publish OCSP responses for newly issued certificates

Root Cause Analysis

Contributing Factor 1: Delayed response of OCSP pods to kubelet on health-checks

• Description: The OCSP responder in our CA platform, introduced in 2023, is deployed on Kubernetes. It provides health probe endpoints that Kubernetes, specifically the kubelet, uses to assess the application's readiness and responsiveness. The kubelet performs regular health checks using the configured endpoints of the OCSP responder, automatically restarting pods if they fail to receive a response within a specified timeframe. Log analysis revealed that, in the period leading up to the incident, some OCSP pods were restarted due to delayed responses exceeding the configured timeout threshold.
• Timeline: N/A
• Detection: SwissSign has monitoring and alerting in place around our Kubernetes environment. However, there was no specific alert for this scenario.
• Interaction with other factors: The restart of OCSP pods triggered the other contributing factors described below.
• Root Cause Analysis methodology used: 5-why

Contributing Factor 2: Open connection count from OCSP pods to HSM too high

• Description: When an OCSP responder starts, it establishes a predefined number of connections to the HSMs during startup. This approach functions as a connection pool, enhancing efficiency and reducing latency. In the configuration, each OCSP responder was set to establish 20 connections at startup.
• Timeline: This configuration was present since introduction of the CA platform.
• Detection: N/A
• Interaction with other factors: Once all available connections to the HSMs were in the state "open", newly started OCSP pods could not open their connections. They retried, which is the contributing factor 3.

Contributing Factor 3: Number of retries of OCSP pods to HSM too high

• Description: The OCSP responders were configured to attempt connecting to the HSMs up to five times before entering an error state and terminating. Combined with contributing factors 1 and 2, this led to an increase in the number of open connections to the HSMs.
• Timeline: This configuration was present since introduction of the CA platform.
• Detection: N/A
• Interaction with other factors: Quick and often restart of OCSP pods lead to an overly large number of connections to the HSMs being open but not used.

Contributing Factor 4: Exhaustion of connection pool from OCSP pods to HSMs

• Description: Our Luna HSMs can handle 800 concurrent open connections per HSM. We currently have 2 HSMs in our production cluster.
• Timeline: N/A
• Detection: Our HSMs do not have the option to configure alerts if the number of open connections reaches a threshold. This prevents us from preventatively detecting a similar situation in the future.
• Interaction with other factors: Once the maximum number of open connections was exhausted, new connections were not accepted and lead to the retries described in contributing factor 3.

Lessons Learned

• What went well:
  • Monitoring picked the outage up very quickly (within 1 minute)
• What didn’t go well:
  • Our alerting didn't cover the scenario of increased OCSP pod restarts
• Where we got lucky: N/A

Action Items

Action Item	Kind	Due Date
#1.1 Deep-dive investigation to determine reason for delayed response of OCSP pods (this may result in additional action items)	Prevent	2025-06-20
#1.2 Increase number of concurrent OCSP pods from 6 to 10	Mitigate	Done
#1.3 Configure alerting for scenario of OSCP pods becoming unresponsive	Detect	Done
#2.1 Re-configure OCSP pods to only open 10 connections at startup	Mitigate	Done
#3.1 Re-configure OCSP pods to only retry 2 times	Mitigate	Done
#4.1 Check if alternate HSM firmware with 4000 concurrent connections can be used (depending on FIPS and Common Criteria certification) (this may result in additional action item if firmware can be used)	Mitigate	2025-06-20
#4.2 Check with manufacturer of HSMs if alerting for connection limit is available (if available, this will result in additional action item)	Prevent	2025-06-20
#4.3 Deep-dive investigation with manufacturer of HSMs to determine reason for hanging connections (this may result in additional action items)	Prevent	2025-06-20

Appendix

Details of affected certificates

N/A

Comment 2

[Roman Fischer]

Scheduled update

Our deep-dive investigations and discussions with vendors are ongoing. We will post updates as soon as we have results from these investigations.

Kind regards
Roman

Comment 3

[Roman Fischer]

No update this week.

Kind regards
Roman

Comment 4

[Roman Fischer]

No update this week.

Kind regards
Roman

Comment 5

[Roman Fischer]

No update this week.

Kind regards
Roman

Comment 6

[Roman Fischer]

Scheduled update

Update to action item #1.1 Deep-dive investigation to determine reason for delayed response of OCSP pods

Our investigations are ongoing. We are building up a special test facility to replicate the behaviour (can't do load tests on the productive instance). We are also looking into enhancing the details in the produced logs. As there are several components (Kubernetes, OCSP microservice, HSM client library, HSM) involved, replicating the prod environment and finding the reason for the delayed response will take some more time.

Next update: 2025-07-18 latest

Update on action item #4.1 Check if alternate HSM firmware with 4000 concurrent connections can be used

The newer firmware is FIPS 140-3 certified, which is necessary for BR and ETSI compliance. But it's not Common Criteria certified which is required for our eIDAS signature business using the same HSMs. The manufacturer of the HSMs does not have a timeline for Common Criteria certification of the newer firmware. This means that we can't upgrade to the newer firmware at this time. We will continue to monitor the situation but due to the lack of manufacturer timeline, we close this action item here.

Update on action item #4.2 Check with manufacturer of HSMs if alerting for connection limit is available

Alerting functionality is not available for the HSMs we are using. Therefore we close this action item.

Update on action item #4.3 Deep-dive investigation with manufacturer of HSMs to determine reason for hanging connections

The manufacturer of the HSMs denied our requests for supporting the deep-dive investigation based on the firmware version we are using. Taking into consideration the situation described in #4.1, we are now trying to find the reason using the tests described in #1.1.

Next update: 2025-07-18 latest

Updated Action Items

Action Item	Kind	Due Date
#1.1 Deep-dive investigation to determine reason for delayed response of OCSP pods (this may result in additional action items)	Prevent	2025-07-18
#1.2 Increase number of concurrent OCSP pods from 6 to 10	Mitigate	Done
#1.3 Configure alerting for scenario of OSCP pods becoming unresponsive	Detect	Done
#2.1 Re-configure OCSP pods to only open 10 connections at startup	Mitigate	Done
#3.1 Re-configure OCSP pods to only retry 2 times	Mitigate	Done
#4.1 Check if alternate HSM firmware with 4000 concurrent connections can be used (depending on FIPS and Common Criteria certification)	Mitigate	Done
#4.2 Check with manufacturer of HSMs if alerting for connection limit is available (if available, this will result in additional action item)	Prevent	Done
#4.3 Deep-dive investigation with manufacturer of HSMs to determine reason for hanging connections (this may result in additional action items)	Prevent	2025-07-18

Kind regards
Roman

Comment 7

[Roman Fischer]

Scheduled update

Update to action item #1.1 Deep-dive investigation to determine reason for delayed response of OCSP pods

Our investigations show that all components function as designed. The problem only occurs because of the way the components interact:

• The OCSP microservice uses internal queueing to accept incoming OCSP signing requests. It also uses a thread pool to process the requests from this queue. Each thread makes use of a library provided by the HSM manufacturer to connect and multiplex several connections to the HSMs.
• Kubernetes uses readyness and lifeness checks on the OCSP pods to determine the status of the pod. These checks are calls to a REST API of the OCSP microservice. The calls to the REST API go into the same internal queue as the OCSP signing requests.
• If the load on one pod increases, the readyness and lifeness checks get processed too slow and the pod gets restarted (even though the pod is actually alive!).
• If one pod is restarted then the load on the remaining pods increases and soon they fail one after another.
• When a new pod with an OCSP microservice is started, initializing the thread pool with its multiplexed connections to the HSM takes a certain time. During that time requests are already coming in and get queued.

We are currently testing with differnt number of pods, virtual CPUs per pod, number of threads inside the pods and readyness and lifeness timeouts.

Next update: 2025-07-31 latest

Update on action item #4.3 Deep-dive investigation with manufacturer of HSMs to determine reason for hanging connections

Our tests did not show conclusively wheter the hanging connections are due to an error in the library that is used by the OCSP microservice to connect to the HSMs or in the way Kubernetes or the underlaying OS is handling the stopping of the OCSP pod. As we can't get any support from the HSM manufacturer as described in #4.1 in the last update, further investigation is not possible.

Therefore we close this action item.

Updated Action Items

Action Item	Kind	Due Date
#1.1 Deep-dive investigation to determine reason for delayed response of OCSP pods (this may result in additional action items)	Prevent	2025-07-31
#1.2 Increase number of concurrent OCSP pods from 6 to 10	Mitigate	Done
#1.3 Configure alerting for scenario of OSCP pods becoming unresponsive	Detect	Done
#2.1 Re-configure OCSP pods to only open 10 connections at startup	Mitigate	Done
#3.1 Re-configure OCSP pods to only retry 2 times	Mitigate	Done
#4.1 Check if alternate HSM firmware with 4000 concurrent connections can be used (depending on FIPS and Common Criteria certification)	Mitigate	Done
#4.2 Check with manufacturer of HSMs if alerting for connection limit is available (if available, this will result in additional action item)	Prevent	Done
#4.3 Deep-dive investigation with manufacturer of HSMs to determine reason for hanging connections (this may result in additional action items)	Prevent	Done

Kind regards
Roman

Comment 8

[Roman Fischer]

Scheduled update

Update to action item #1.1 Deep-dive investigation to determine reason for delayed response of OCSP pods

After running a large series of tests / simulations with different number of pods, number of threads in each pod, readiness- and liveness-check timeouts and retries, we are now confident that we understand the dependencies of the components and where the bottleneck arose that lead to the outage: We were using a configuration with too few pods and too many retries per thread in the pods. This lead to the OCSP requests being queued. At the same time, the readiness checks had a too long timeout which prevented new incoming OCSP requests to be routed to other pods. The requests were then dropped. Because the pods where seen as not responding, they were restarted, opened too many connections upon startup to the HSM and thus exhausting the HSMs connection pool.

With the new configuration, we successfully tested up to three times the peak OCSP load we've seen so far, without the problem re-occurring. At about 170% of the peak OCSP load, the OCSP responder with the improved configuration starts to drop single OCSP requests without restarting pods.

We added periodic checks on average and peak load on our OCSP responder as part of regular capacity planning to accommodate future growth of business and in turn OCSP requests.

This concludes our investigation of contributing factor #1 and thus action item #1.1

We will monitor this bug for a week or so to allow for questions from the community before we will post the Closure Summary.

Updated Action Items

Action Item	Kind	Due Date
#1.1 Deep-dive investigation to determine reason for delayed response of OCSP pods (this may result in additional action items)	Prevent	Done
#1.2 Increase number of concurrent OCSP pods from 6 to 10	Mitigate	Done
#1.3 Configure alerting for scenario of OSCP pods becoming unresponsive	Detect	Done
#2.1 Re-configure OCSP pods to only open 10 connections at startup	Mitigate	Done
#3.1 Re-configure OCSP pods to only retry 2 times	Mitigate	Done
#4.1 Check if alternate HSM firmware with 4000 concurrent connections can be used (depending on FIPS and Common Criteria certification)	Mitigate	Done
#4.2 Check with manufacturer of HSMs if alerting for connection limit is available (if available, this will result in additional action item)	Prevent	Done
#4.3 Deep-dive investigation with manufacturer of HSMs to determine reason for hanging connections (this may result in additional action items)	Prevent	Done

Kind regards
Roman

Comment 9

[Roman Fischer]

Report Closure Summary

• Incident description:
  SwissSign OCSP responder for one CA platform (ocsp.swisssign.ch) was not reachable from 2025-05-12T09:23:58Z to 2025-05-12T10:13:56Z

• Incident Root Cause(s):
  We identified 4 contributing factors that in combination lead to the OCSP outage. Deep technical investigations and large series of tests and simulations showed that the true root cause was a combination of configuration settings of various components of the OCSP responder service that in certain runtime situations could lead to the OCSP service being completely unresponsive.

• Remediation description:
  The configurations of the various components of the OCSP responder was changed and load/stress tests showed that this new configuration can handle up to three times the peak load on OCSP we have ever observed. Additionally, periodic checks on average and peak load on our OCSP responder were added as part of regular capacity planning to increase OCSP resources if needed.

• Commitment summary:
  SwissSign is committed to provide highly available certificate status information services.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 10

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-08-18.