1962809 - SSL.com: Expired certificate for a “Valid” Test Website

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Rebecca Kelley]

Preliminary Incident Report

Summary

Incident description: One of our test websites for a valid certificate was expired crt.sh/?id=12305750195 (test-ev-ecc.ssl.com). The certificate has now been replaced with a valid certificate. An investigation has been initiated into our preventive and detective controls.

Relevant policies: This is a violation of section 2.2 of the TLS BRs:

2.2 Publication of information:

The CA SHALL host test Web pages that allow Application Software Suppliers to test their software with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber Certificates that are

i. valid,
ii. revoked, and
iii. expired.

Source of incident disclosure: Third Party Reported: A Certificate Problem Report was submitted to notify us of the issue on 2025-04-24 at 20:39 UTC.

SSL.com will post our Full Incident Report on or before 2025-05-08.

Comment 1

[Luis Cervantes]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A002038

• Incident description: One of our test websites for a valid certificate was expired crt.sh/?id=12305750195 (test-ev-ecc.ssl.com). The certificate has now been replaced with a valid certificate.

• Timeline summary:

  • Non-compliance start date: 2025-04-05
  • Non-compliance identified date: 2025-04-24
  • Non-compliance end date: 2025-04-24

• Relevant policies: This is a violation of section 2.2 of the TLS BRs:

  2.2 Publication of information:

  The CA SHALL host test Web pages that allow Application Software Suppliers to test their software with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber Certificates that are

  i. valid,
  ii. revoked, and
  iii. expired.

• Source of incident disclosure: Third Party Reported: A Certificate Problem Report was submitted to notify us of the issue on 2025-04-24 at 20:39 UTC.

Impact

• Total number of certificates: This incident is not related to mis-issuance of certificates.
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: N/A
• Incident heuristic: N/A
• Was issuance stopped in response to this incident, and why or why not?: N/A
• Analysis: N/A - No revocation delay.
• Additional considerations: N/A

Timeline

All times are in UTC.

2024-06-18:

• Added test certificates including test-ev-ecc.ssl.com into our monitoring system and confirmed email alerts were being processed as expected.

2025-02-18:

• An email alert was generated regarding an expiration notice for test-ev-ecc.ssl.com.

2025-03-05:

• An additional email alert was generated regarding an expiration notice for test-ev-ecc.ssl.com.

2025-04-05:

• 18:27: Certificate for test-ev-ecc.ssl.com expired.

  2025-04-24:

• 20:39: A Certificate Problem Report (CPR) was submitted to notify us of the expired certificate on test site for test-ev-ecc.ssl.com.

• 22:07: A new valid certificate was placed on the test website for test-ev-ecc.ssl.com.

2025-04-25:

• 15:21: Internal ticket registered by the Compliance Business Unit (CBU) in accordance with our Incident Management Policy, confirms this as a violation of section 3.2.2.4 of our CPS and declares an incident.
• 19:31: Bug 1962809 was generated

2025-04-26 to 2025-04-29:

• CBU team conducts an internal investigation and reviews possible remediation actions.
  2025-04-30:

• 16:30: Added preventive control by adding test websites to an external monitoring system. Alerts will be sent to an external channel and will be repeated frequently until the issue is resolved.

2025-05-01 to 2025-05-07:

• Collection of evidence surrounding the events that led to this bug was conducted along with a Root Cause Analysis and discussion of corrective action plan.
• Preparation of the Full Incident Report (this report).

2025-05-08:

• Submission of the Full Incident Report (this report).

Related Incidents

Bug	Date	Description
Bug 1957140	2025-03-28	There are similar contributing factors in which monitoring alerts were not processed in a timely manner.
Bug 1927532	2024-10-24	There are similar contributing factors in which monitoring alerts were incorrectly filtered in the support ticketing system.

SSL.com acknowledges the need to adopt a more efficient monitoring and alerting system, which will be addressed as part of our long-term commitment plans. In the interim, we are addressing this in a review of all alert types in our action item number four, which is listed below.

Root Cause Analysis

Contributing Factor #1: Monitoring alerts are sent to a high-volume email inbox without proper categorization.

• Description: An alert reporting mechanism was created to monitor test certificates for early detection of any upcoming expiring certificates. Unfortunately, the alerts were sent to a high-volume email inbox and were overlooked.
• Timeline:
  • 2024-06-18: Added test certificates including test-ev-ecc.ssl.com into our monitoring system.
  • 2025-02-18: An email alert was generated regarding an expiration notice for test-ev-ecc.ssl.com.
  • 2025-03-05: An additional email alert was generated regarding an expiration notice for test-ev-ecc.ssl.com.
  • 2025-04-05: Certificate for test-ev-ecc.ssl.com expired.
  • 2025-04-24: A Certificate Problem Report (CPR) was submitted to notify us of the expired certificate on test site for test-ev-ecc.ssl.com.
• Detection: Despite the email alerts being sent out as configured, it was not until the investigation of a CPR that we identified the issue.
• Interaction with other factors: Increased the time until detection.
• Root Cause Analysis methodology used: 5-Whys

Lessons Learned

• What went well: We were able to correct the “valid” test certificate within two (2) hours of it being reported.
• What didn’t go well: Issue was not detected from internal controls.
• Where we got lucky: This issue only affected one of our certificate test websites.
• Additional: N/A

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Add test websites to an external monitoring system. Alerts will be routed to an external channel and will be repeated frequently until the issue is resolved.	Prevent/Detect	Contributing Factor #1	Test against non-production certificates and confirm alerts are triggered as expected.		Completed
Create a master list of all certificates owned by the company and their expiration dates to ensure full coverage and tracking.	Prevent	Contributing Factor #1	Compare against all SSL.com’s public facing server certificates, including those required by BR or CP/CPS Policies.	2025-06-06	In-Progress
Update our process to require creation of work items for the renewal of certificates in advance.	Prevent	Contributing Factor #1	Confirm process is updated and work item tickets are created for all certificates in the master list.	2025-06-20	Open
Review and categorize email alerts to actionable priorities	Detect / Prevent	Contributing Factor #1	Set an SLA for each category and track it.	2025-07-11	Open

Appendix

This incident is not related to mis-issuance of certificates.

Comment 2

[chrome-root-program]

The Root Cause Analysis in Comment 1 could benefit from additional detail.

Given the “5-Whys” methodology was used, we’d appreciate SSL.com considering the following:

(1) Why were the test websites not previously considered for the external monitoring system described in the Actions Item table?

(2) Why does SSL.com expect the external reporting channel described in the Action Items table will prevent future recurrence of this issue?

(3) Why did SSL.com not previously account for “all certificates owned by the company and their expiration dates” as part of its own certificate lifecycle management strategy?

(4) Why does SSL.com expect that categorizing emails sent to the “high-volume” email inbox will prevent future recurrence of this issue? If all messages sent to the inbox are properly categorized, how does that focus attention on any particular type of issue?

(5) Why aren’t SSL.com’s test certificates relying on an automation solution?

Comment 3

[Luis Cervantes]

(In reply to chrome-root-program from comment #2)

Hi Chrome, thank you for taking the time to review our incident, please see our answers to your questions below.

The Root Cause Analysis in Comment 1 could benefit from additional detail.

Given the “5-Whys” methodology was used, we’d appreciate SSL.com considering the following:

(1) Why were the test websites not previously considered for the external monitoring system described in the Actions Item table?

Test websites were monitored through the internal monitoring system, which delivers alerts to a designated email address. In our analysis, the issue is more on how the delivery of alerts is executed rather than the system used to produce the alerts itself. The external monitoring system allows for more effective delivery of the alerts to our team collaboration platform.

The investigation of the repetitious alerting/monitoring incidents SSL.com has accrued over the last several months, revealed that using email as the delivery method of actionable items entails risks especially in case of high-volume email inboxes. For this reason, we recognize the need to adopt a more efficient monitoring and alerting system going forward. In the interim, we are reviewing all alert types in our internal monitoring system to ensure delivery to the proper channels where actionable procedures will take place (see our 4th action item).

(2) Why does SSL.com expect the external reporting channel described in the Action Items table will prevent future recurrence of this issue?

The new external reporting channel uses a specific collaboration channel dedicated to monitoring expiring public certificates utilized by SSL.com. The alerts from this channel will be directed to multiple people across many business units and continues its alerting frequently until the expiring certificate is resolved. This is similar to how the seat belt buzzers work, until passengers are ‘forced’ to buckle up.

The new external reporting mechanism operates independently of the SSL.com infrastructure; this minimizes the likelihood of both procedures failing at the same time.

With the implementation of our 3rd action item (creation of work items), this external reporting channel will shift to a detective control as work items will be put in place to renew certificates 30 days prior to expiration with our external reporting channel providing alerts if a certificate gets to 15 days of expiration.

(3) Why did SSL.com not previously account for “all certificates owned by the company and their expiration dates” as part of its own certificate lifecycle management strategy?

At the time of discovery, the inventory of certificates was fragmented and managed amongst different departments. We are now moving towards a single inventory across the company with a dedicated channel, health dashboard, and response team.

(4) Why does SSL.com expect that categorizing emails sent to the “high-volume” email inbox will prevent future recurrence of this issue? If all messages sent to the inbox are properly categorized, how does that focus attention on any particular type of issue?

The intent of categorizing the emails being sent to the high-volume email box is to take each alert type and route them to dedicated channels where next steps can be taken and followed up until completion.

SSL.com is in the process of filtering compliance-related emails and re-routing them to instant messaging and/or monitoring tools that are frequently used throughout our daily operations to provide better alerting and tracking abilities to the responding teams. We expect that collaborative tools like this approach will allow for better identification of high severity alerts, compared to uncategorized emails in overloaded inboxes.

(5) Why aren’t SSL.com’s test certificates relying on an automation solution?

Currently our ACME endpoints only support DV certificates as OV and EV certificates require identity (re-)validation. Therefore, the current EV test certificates rely on a manual process.

As part of our ongoing plan to minimize reliance on vendors, we are implementing an in-house ACME server solution that will also support OV and EV certificates by interacting with the in-house developed RA. At this time our ACME server is actively being tested. We expect to utilize the new ACME server as soon as it becomes available in production.

SSL.com is also considering the use of our latest Certificate Lifecycle Management (CLM) API for EV Certificate renewals. With the passing of SC081 and the gradual reduction of certificate validity, both these in-house tools (AMCE Server, CLM API) will allow us to implement automation in all certificate management processes. SSL.com supported SC081 and is preparing a strategy that includes tools, documentation and content pieces related to certificate lifecycle management automation for its customers and internal use.

Comment 4

[chrome-root-program]

Thank you for engaging with these additional “why” questions in detail. We appreciate SSL.com not only stating what you will do, but for explaining why the previous approach failed and why the new measures are expected to be more effective. We consider this a good reflection of learning and continuous improvement.

Comment 5

[Luis Cervantes]

SSL.com continues to monitor this bug for additional questions. We request the next update for this bug be set to 2025-06-06, as we continue to focus on completion of the action items.

Comment 6

[SSL.com]

This is an update to report our progress with remediation actions.

The following action item has been completed:

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Create a master list of all certificates owned by the company and their expiration dates to ensure full coverage and tracking.	Prevent	Contributing Factor #1	Compare against all SSL.com's public facing server certificates, including those required by BR or CP/CPS Policies.	2025-06-06	Completed

SSL.com continues to be on track with the remaining actions items and requests that our next update to be set for 2025-06-20.

Comment 7

[SSL.com]

This is an update to report our progress with remediation actions.

The following action item has been completed:

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Update our process to require creation of work items for the renewal of certificates in advance.	Prevent	Contributing Factor #1	Confirm process is updated and work item tickets are created for all certificates in the master list.	2025-06-20	Completed

SSL.com continues to be on track with the remaining action items and requests for our next update to be set for 2025-07-03.

Comment 8

[SSL.com]

SSL.com continues to monitor this bug for any comments or questions. We are still working on our final action item and expect it to be completed by 2025-07-11.

Comment 9

[SSL.com]

This is an update to report our progress with remediation actions.

The following action item has been completed:

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Review and categorize email alerts to actionable priorities	Detect / Prevent	Contributing Factor #1	Set an SLA for each category and track it.	2025-07-11	Completed

All Action Items have been completed as described. We continue to monitor this bug for any comments or questions, and we will provide our Incident Closure Summary next week.

Comment 10

[SSL.com]

Report Closure Summary

Incident description: The "valid" TLS Certificate on one of our test websites expired and was not immediately replaced. After receiving a notification from a third party, SSL.com replaced the expired certificate with a non-revoked, non-expired, trusted certificate.

Incident Root Cause(s): An alert reporting mechanism is in place to monitor test certificates for early detection of any upcoming expiring certificates. Unfortunately, the alerts were sent to a high-volume email inbox and were overlooked.

Remediation description: SSL.com added test websites to an external monitoring system designed for higher visibility and increased frequency of sticky notifications. These sticky notifications are now delivered to designated channels monitored by multiple teams to ensure prompt action. In addition, we compiled a master list of all certificates owned by the company. That way, based on their approaching expiration dates, new work requests are created for replacement of the certificate.

Commitment summary:

The following ongoing initiatives are part of our commitments for this bug:

• Adopt a more efficient monitoring and alerting system, by investing collaborative tools that allow for better identification of high severity alerts

• Move towards a single inventory across the company with a dedicated channel, health dashboard, and response team.

• Extend automation in all certificate management processes, including OV and EV certificates by integrating the ACME service with our RA portal.

All Action Items disclosed in this report have been completed as described, and we request closure of this bug.

Comment 11

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-25.