1931636 - SSL.com: Delay in publishing OCSP responses

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

Preliminary Incident Report

Summary

On 2024-11-13, SSL.com's support team received a report of a possible issue affecting OCSP responders. Certificates issued from two identified subCAs gave unexpected errors when queried. While OCSP response times was not explicitly required in the current BRs when this event occurred; SSL.com is aware of the recent ballot SC-76, that was adopted on 2024-11-14 and maintains a general expectation that a OCSP response must be available for all certificates. Therefore, we are treating this event as an incident.

The report includes two subCAs affected by this potential issue:

• https://crt.sh/?identity=%25&iCAID=372373

• https://crt.sh/?identity=%25&iCAID=372375

In response to this report, SSL.com performed the following:

• Our Security Auditing team was informed, and the Incident Management Process was initiated. The Security Auditing team began to gather evidence to examine the issue, investigate how this occurred, and determine the impact.

• The Certificate Problem Report (CPR) process was initiated immediately, once the potential issue was confirmed. SSL.com identified four subCAs were generated on 2024-09-26 that were not configured for the validation authority (VA) module of EJBCA correctly.

• After identifying the four subCAs (including the two identified by the reporter), our team corrected the configuration within the VA module responsible for serving the OCSP responses.

SSL.com identified and remediated the issue quickly, so there was no need to temporarily stop the issuance of certificates. We will continue our investigation and Root Cause Analysis (RCA) and will submit a final incident report on or before 2025-11-27.

Comment 1

[Yongmin Hong [:revi]]

(In reply to Rebecca Kelley from comment #0)

will submit a final incident report on or before 2025-11-27.

I assume this is a typo and you meant 2024-11-27?

Comment 2

[Rebecca Kelley]

Correction: SSL.com will submit a final incident report on or before 2024-11-27.

Comment 4

[Rebecca Kelley]

Final Incident Report

Summary

On 2024-11-13, SSL.com's support team received a report of a possible issue affecting OCSP responses. Certificates issued from two identified subCAs gave unexpected errors when queried. These two subCAs were found to be misconfigured to not serve the pre-signed OCSP responses. This does not meet the expectation of section 4.9.9 of the SSL.com CP/CPS, that an OCSP response must be available without undue delay for all certificates with an OCSP URL.

Impact

Upon receiving the report and after conducting an initial investigation, four CAs were found to be affected (see Appendix). On the same day, once all affected CAs were identified, the respective Validation Authority (VA) configuration was corrected.

The issue affected the abovementioned CAs from 2024-11-07 (date of first leaf certificate issuance) to 2024-11-13 (date of remediation). During this period, the status of certificates was available through CRL.

Timeline

All times are UTC.

2024-09-26

Four (4) new CA certificates were issued

2024-09-28

Post CA Ceremony actions: configuration of the VA component; failure to complete all sub-tasks of the VA component to serve pre-generated OCSP responses for the new CAs.

2024-11-07

Started issuance of leaf certificates chaining to the subCAs in question.

2024-11-13

• 04:35 SSL.com support is informed about a potential issue affecting OCSP responders

• 06:16 The operations and compliance teams are internally informed of the potential issue

• 06:39 The issue is confirmed, and an investigation is initiated by the operations team

• 07:50 The operations team identified a misconfiguration affecting 4 CAs on the VA components responsible for serving the OCSP responses

• 07:54 The misconfiguration was corrected

• 13:09 The event is analyzed by the compliance team, resulting in a decision to escalate to an incident and treat accordingly

2024-11-15

• 22:52 SSL.com posted a Preliminary Incident Report to Bugzilla Bug 1931636 - SSL.com: Delay in publishing OCSP responses

• 23:06 Compliance officer notified Root Stores independently about this Bugzilla bug

2024-11-16 to 2024-11-26

Collection of evidence surrounding the events that led to the incident, Root Cause Analysis and determination of the remediation actions.

2024-11-27

Compilation and submission of the Final Incident Report (this report).

Root Cause Analysis

This Root Cause Analysis (RCA) includes separate branches to address (A) the weaknesses that led to the problem manifestation, and (B) the weaknesses that led to its non-detection.

(A) Problem Statement for the failure of the preventing controls:

OCSP response returned “unauthorized”, i.e. was not available, for leaf certificates of the four subCAs listed in the Appendix.

1. Why was the OCSP response not available for these certificates?

The VA component was not aware of the pre-signed OCSP responses that the CA was promptly and continuously generating.

2. Why was the VA component not aware of the existence of the pre-signed OCSP responses?

The relevant setting in the VA configuration for these CAs was not enabled.

3. Why was the relevant setting in the VA configuration for these CAs not enabled?

The relevant configuration task was not completed successfully, and the peer review process did not detect the issue.

Note: At this stage, the RCA further breaks into two separate paths (a, b)

4a. Why was one of the tasks not completed successfully?

The multi-step task was marked as done without one of its steps being performed.

4b. Why did the peer review process not detect the issue?

The tooling used in the peer review process does not sufficiently cover the VA component configuration.

5a. Why was the multi-step task marked as done without all steps being performed?

The post-ceremony actions process does not require confirmation of task completion at this granularity. (i.e., per step).

5b. Why is the tooling used in the peer review process not sufficiently cover the VA component configuration?

The configuration report generated by the CA software does not include all settings of the VA component.

(B) Problem Statement for the failure of the detective controls:

The OCSP unavailability issue was not detected by our systems.

1. Why was the OCSP unavailability issue not detected?

At the time the problem was reported, the monitoring tools did not include the four new CAs in the configuration to check OCSP availability.

2. Why did the monitoring tools not include the four CAs in the configuration to check OCSP availability?

The post-ceremony action to configure OCSP monitoring for these CAs was not completed within an acceptable amount of time.

3. Why was this post-ceremony task not completed soon enough?

The task requires manual intervention upon issuing the first leaf certificate per CA , or else the monitoring tool produces a false alarm.

Lessons Learned

What went well

• The issue was corrected within 4 hours from receiving the report

• Detailed RCA for both the prevention and the detection of the issue

What didn't go well

• Did not properly follow stated procedures

• Post ceremony action tasks were not followed through to completion

Where we got lucky

• Although the OCSP responses of those CAs were not available in a timely manner, the status of the leaf certificates could be determined using the CRLs.

Action Items

#	Action Item	Kind	Due Date
1	Improve the post-ceremony process tools to enable explicit confirmation of task completion at the sub-task level	Prevent	2024-12-06
2	Improve the peer review tools to fully cover all configuration settings related to post-ceremony actions	Prevent	2024-12-20
3	Improve the OCSP monitoring tools to automatically cover any new CA upon the issuance of the first leaf certificate without human intervention	Remediate/Detect	2024-12-20
4	Retroactive verification of post-ceremony actions for all CAs	Remediate	2024-12-13
5	Update the post-ceremony processes to require monitoring of the action completion by a separate person	Remediate	2024-12-06

Appendix

https://crt.sh/?id=14748158300

https://crt.sh/?id=14748158301

https://crt.sh/?id=14748167217

https://crt.sh/?id=14748167216

Comment 5

[Rebecca Kelley]

This is an update to report our progress with remediation actions.

The following action items have already been implemented:

#	Action Item	Kind	Due Date
1	Improve the post-ceremony process tools to enable explicit confirmation of task completion at the sub-task level	Prevent	Completed
5	Update the post-ceremony processes to require monitoring of the action completion by a separate person	Remediate	Completed

Comment 6

[Rebecca Kelley]

Action Item #4 is still in progress and we will provide an update next week.

Comment 7

[Rebecca Kelley]

This is an update to report our progress with remediation actions.
The following action items have already been implemented:

#	Action Item	Kind	Due Date
2	Improve the peer review tools to fully cover all configuration settings related to post-ceremony actions	Prevent	Completed
4	Retroactive verification of post-ceremony actions for all CAs	Remediate	Completed

Action Item number 3 will be completed next week.

Comment 8

[Rebecca Kelley]

We have made progress to Action Item #3, but it is not complete. We will post another update next week.

Comment 9

[Rebecca Kelley]

This is an update to report our progress with remediation actions.

Implementation of the following action item has now been completed:

#	Action Item	Kind	Due Date
3	Improve the OCSP monitoring tools to automatically cover any new CA upon the issuance of the first leaf certificate without human intervention	Remediate/Detect	Complete

Comment 10

[SSL.com]

At this time all action items have already been completed. SSL.com will continue to monitor this bug for questions.

Comment 11

[Rebecca Kelley]

If there are no other questions or concerns at this time, we ask that this bug be closed.

Comment 12

[Ben Wilson]

Hi,
Even though this has not yet been officially formalized as a bug-closure requirement, could you please provide a closing summary?
Thanks,
Ben

A closing summary should briefly:

• describe the incident, its root cause(s), and remediation;
• summarize any ongoing commitments made in response to the incident; and
• attest that all Action Items have been completed.

Here is a markdown template if needed:

Incident Report Closure Summary

• Incident Description: [Two or three sentences summarizing the incident.]
• Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
• Remediation Description: [Two or three sentences summarizing the incident's remediation.]
• Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 13

[Rebecca Kelley]

Incident Report Closure Summary

• Incident Description: Two subCAs were found to be misconfigured and did not meet the expectation of section 4.9.9 of the SSL.com CP/CPS that an OCSP response must be available without undue delay for all certificates with an OCSP URL.

• Incident Root Cause(s): The relevant Validation Authority (VA) configuration task was not completed successfully, and the peer review process did not detect the issue. The post-ceremony actions process did not require confirmation of the task completion at this granular level. Finally, a restriction of our CA software regarding the VA configuration report did not allow verification of all VA component settings.

• Remediation Description: We have made improvements to both our post ceremony process and peer review tools. Additionally, we have implemented an automated OCSP monitoring for any new CA upon the issuance of the first leaf certificate.

• Commitment Summary: SSL.com will continue to identify ways to improve and enhance our internal ceremony processes and tools.

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 14

[Ben Wilson]

If there are no additional comments or concerns, I will close this on Wed. 12-Feb-2025.