1974325 - ANF AC: Test Certificates Non-Compliance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Derek R. Greene]

Preliminary Incident Report

Summary

• Incident description:

  1. A TLS certificate (https://crt.sh/?q=09531316A000852D2E764B1FB5B90493AC3958E2EEC113C4479AFEFABB6CA643) which expired on 6-17-25 08:37:31 GMT is being served on the domain https://testvalidsslev.anf.es/. This domain is represented in CCADB as a "Test Website - Valid" URL and is associated with the ANF Secure Server Root certificate SHA-256: FB8FEC759169B9106B1E511644C618C51304373F6C0643088D8BEFFD1B997599

  2. A TLS certificate (https://crt.sh/?q=ab004a6d278d8b2a748155358ddb0c7c40609081d0e723dafa0156f813159778) which expired on 6-17-25 08:37:26 GMT is being served on the domain https://testrevokedsslev.anf.es/. This domain is represented in CCADB as a "Test Website - Revoked" URL, however as the certificate has expired, the revocation becomes irrelevant. Associated with the ANF Secure Server Root certificate SHA-256: FB8FEC759169B9106B1E511644C618C51304373F6C0643088D8BEFFD1B997599

  Section 2.2 of the BRs defines that the "Test Website - Valid" URL must host a valid, unrevoked, and unexpired TLS certificate at all times.
  Section 2.2 of the BRs defines that the "Test Website - Revoked" URL must host a revoked, unexpired TLS certificate at all times.

• Relevant policies:

  • Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates v2.1.5
    • Section 2.2 - Publication of information:

      "The CA SHALL host test Web pages that allow Application Software Suppliers to test
      their software with Subscriber Certificates that chain up to each publicly trusted Root
      Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber
      Certificates that are
      i. valid,
      ii. revoked, and
      iii. expired."

• Source of incident disclosure:
  Third Party Reported

Comment 1

[Pablo Díaz]

Update: Both certificates are updated. We are working on the full incident report.

Comment 2

[incident-reporting]

Please note the CA Owner response in Comment 1 is inconsistent with the expectations described on CCADB.org which state:

Within 72 hours of a CA Owner becoming aware of an incident (i.e., the “initial incident disclosure”) or an audit finding not previously disclosed in an Incident Report, the CA Owner MUST either:

• disclose a Preliminary or Full Incident Report; or
• respond to a Preliminary Incident Report previously created for the incident by a third party reporter.

In its initial report (i.e, Preliminary or Full Incident Report) or reply to a third-party report, the CA Owner MUST:

• accurately disclose the impact of the incident (e.g., the corpus of then-known mis-issued certificates); and
• describe whether the incident should be considered contained (e.g., because certificate issuance was stopped) or ongoing.

Comment 3

[Pablo Díaz]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000269
• Incident description: On June 26, 2025, it was reported that our certificates for our Valid (https://crt.sh/?q=09531316A000852D2E764B1FB5B90493AC3958E2EEC113C4479AFEFABB6CA643) and Revoked (https://crt.sh/?q=ab004a6d278d8b2a748155358ddb0c7c40609081d0e723dafa0156f813159778) Test Websites had expired, violating section 2.2 of the Baseline Requirements. The certificate have now been replaced with a non-expired certificate.
• Timeline summary:
  • Non-compliance start date: 2025-06-17 with the expiration of both certificates.
  • Non-compliance identified date: 2025-06-26
  • Non-compliance end date: 2025-06-30
• Relevant policies:
  Baseline Requirements v2.1.5, Section 2.2 (Publication of Information)

2.2 Publication of information:
The CA SHALL host test Web pages that allow Application Software Suppliers to test their software with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber Certificates that are
i. valid,
ii. revoked, and
iii. expired.

• Source of incident disclosure: Reported by third party.

Impact

• Total number of certificates: 0
• Total number of "remaining valid" certificates: 0
• Affected certificate types: This incident has not led to any misissued certificates.
• Incident heuristic: N/A
• Was issuance stopped in response to this incident, and why or why not?: No. The incident did not impact active certificate issuance processes.
• Analysis: N/A - No revocation delay.
• Additional considerations: N/A

Timeline

All times are in UTC.

Date	Event
2025-05-20 07:00	Internal migration of monitoring infrastructure initiated.
2025-06-02 12:07	Audit case 00002433 submitted to CCADB ("Add/update Root Request") and test URLs passed automated validation.
2025-06-17 08:37	Test Website certificates expired.
2025-06-23 17:04	CCADB Case 00002433 was closed.
2025-06-26	Preliminary Incident Report published in Bug 1974325 by Third Party. Not yet assigned.
2025-06-27	Bug 1974325 asigned to POC.
2025-06-30 09:14	Test Website certificates replaced. Internal review and consideration of possible corrective measures.
2025-06-30 10:00	testvalidsslev.anf.es and testrevokedsslev.anf.es re-added into the monitoring system. We made sure alert emails were working as intended.
2025-06-30 to 2025-06-01	Collection of the events that caused this incident, Root Cause Analysis and review of possible corrective actions. Preparation of the Incident Report
2025-06-02	Submission of the Full Incident Report (this report).

Related Incidents

Bug	Date	Description
1731887	2021-09-21	Test Website Certificates Expired.
1962809	2025-04-25	Test website for valid certificate expired.

Root Cause Analysis

Contributing Factor #1: Test Website URLs excluded during monitoring system reconfiguration

• Description: As part of an infrastructure migration, changes were made to our internal monitoring system. The service configurations were version-controlled in a Git repository; however, certain checks—specifically those monitoring the Valid and Revoked Test Website URLs—had previously been added directly on the live monitoring instance and were not committed to the central configuration repository.
  As a result, when the monitoring infrastructure was rebuilt from the repository during the migration, these specific test domain checks were omitted.
• Timeline:. Configuration changes occurred prior to June 2025.
• Detection:. Detected on June 26, 2025, via a third-party report.
• Interaction with other factors: The migration process correctly rebuilt the monitoring environment from the available repository. However, the absence of these test URLs in the version-controlled configuration represents a configuration management failure. No validation process was in place to ensure that all compliance-related monitoring items were correctly restored.

Contributing Factor #2: Dependency on audit timing for renewal of test certificates

• Description: In previous years, the renewal of test website certificates coincided with the annual audit update process and occurred just before submission to CCADB. This year, due to a change in the auditor, the audit was conducted approximately two months earlier than usual. As a result, the certificates were still valid during the submission to CCADB and passed all checks. The issuance of these test certificates remains a manual process, historically aligned with audit preparation activities.
• Timeline: Audit calendar shifted in 2025
• Detection: No expiration alert triggered due to missed renewal process.
• Interaction with other factors: The earlier audit decoupled the timing from the usual renewal process, and that, plus the Root cause 1, caused the expired certificates to go unnoticed.

Lessons Learned

• What went well: The audit preparation process ensured that the Test Websites were valid at submission.
• What didn’t go well:
  • The incident was not detected by our internal controls.
  • Monitoring system was not properly validated after migration. Renewal timing relied too heavily on audit schedule.
• Where we got lucky: Issue was detected by external party.
• Additional: N/A

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Re include test domains in monitoring system	Prevent	Root Cause #1	Test URLs covered in monitoring dashboard with expiration alerting	2025-06-30	Complete
Add renewal reminders for test website certificates to compliance calendar	Prevent	Root Cause #2	Task tracked and confirmed by responsible role	2025-06-30	Complete
Create a centralized list of all certificates owned by ANF AC along with their expiration date.	Prevent	Root Cause #2	Centralized list created, includes expiration dates and owners, and is used in monthly compliance checks.	2025-07-04	In Progress
Define test website management as a recurring compliance task independent of audit schedule. Add control in GRC Software	Prevent	Root Cause #2	SOP updated and version-controlled	2025-07-07	Planned
Add post-migration validation checklist for monitoring system changes	Prevent	Root Cause #1	Checklist implemented and used in infrastructure change processes	2025-07-09	Planned

Appendix

This incident is not related to mis-issuance of certificates.

Comment 4

[Yulier Nuñez]

The following action has been completed:

Create a centralized list of all certificates owned by ANF AC along with their expiration date.

Comment 5

[Yulier Nuñez]

The following action has been completed:

Define test website management as a recurring compliance task independent of audit schedule. Add control in GRC Software

Comment 6

[Yulier Nuñez]

The following action has been completed:

Add post-migration validation checklist for monitoring system changes

Comment 7

[Pablo Díaz]

Report Closure Summary

• Incident description: Two EV certificates for test websites (Valid and Revoked) expired on 2025-06-17, , violating section 2.2 of the Baseline Requirements.

• Incident Root Cause(s):

  • The test domain monitoring checks had not been included in the version-controlled configuration repository and were lost during the infrastructure migration.
  • The renewal of test certificates had been historically tied to the audit calendar. A change in auditor advanced the audit timeline, causing the usual renewal process to be skipped unnoticed.

• Remediation description: see Action Items table. As a summary:

  • Certificates were renewed.
  • Monitoring was restored with alerts.
  • SSL Certificate inventory was centralized.
  • Renewal tasks decoupled from audit cycle and added to compliance calendar.
  • Post-migration validation checklist implemented.

• Commitment summary:

  • Include test website monitoring in periodic internal audits.
  • Maintain test website management not tied to audit timing.

All Action Items have been completed as described, and we request the closure of this report.

Comment 8

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-16.