1966006 - KIR: Intermediate CA - SZAFIR Trusted CA3 - revocation status not changed in CCADB

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Waldemar Brzozowski]

Preliminary Incident Report

Summary

• Incident description:
  KIR has revoked subordinate certificate - SZAFIR Trusted CA3. It’s status has not been updated in the CCADB within the required timeline.

SZAFIR Trusted CA3 correct status (revoked) is already set in CCADB.

A full incident report will be provided no later than Monday May 19th 2025.

• Relevant policies:
  According to Section 6 of the Chrome Root Program Policy states Chrome Root Program Participants MUST: “Disclose revocation of all subordinate CA certificates capable of validating to a certificate included in the Chrome Root Store or associated with a Root Inclusion Request to the CCADB within 7 calendar days of revocation.”

• Source of incident disclosure:
  Third Party Reported

Comment 1

[Waldemar Brzozowski]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000251
• Incident description:
  The revocation of subordinate CA certificate Szafir Trusted CA3 was not disclosed to the CCADB within timeline required by Section 6 of the Chrome Root Program Policy and CCADB Policy (section 4. Subordinate CA Certificates).
  The issue was first noticed and reported by The Chrome Root Program (CRP) Team.
• Timeline summary:
  • Non-compliance start date: March 13, 2025
  • Non-compliance identified date: May 09, 2025
  • Non-compliance end date: May 12, 2025
• Relevant policies:
  According to Section 6 of the Chrome Root Program Policy states Chrome Root Program: “Disclose revocation of all subordinate CA certificates capable of validating to a certificate included in the Chrome Root Store or associated with a Root Inclusion Request to the CCADB within 7 calendar days of revocation.”
  According to Section 4 CCADB Policy: "If a subordinate CA certificate is revoked, the CCADB MUST be updated to mark it as revoked, including the reason for revocation, within seven calendar days of revocation."
• Source of incident disclosure:
  Third Party Reported - The Chrome Root Program (CRP) Team

Impact

• Total number of certificates: 1
• Total number of "remaining valid" certificates: 0
• Affected certificate types: SubCA
• Incident heuristic:
  Subordinate CA certificate SZAFIR TRUSTED CA3 - https://crt.sh/?sha256=ec036c294f512dd28c5666c2d53ec0dcf6f397fed6f8703a7c7532da3e02de8c
  Status disclosure violation in the CCDAB record for SZAFIR Trusted CA3. The revocation status has not been updated in CCADB
• Was issuance stopped in response to this incident, and why or why not?:
  The incident did not involve certificate issuance.
• Analysis:
  The dalay affected CCADB status update.
• Additional considerations:

Timeline

March 05, 2025 – 12:55 UTC – Revocation of Szafir Trusted CA3 certificate according to the closure plan of Szafir Trusted CA3
March 13, 2025 – 12:55 UTC – Expected CCADB disclosure deadline ( 7 days )
May 09, 2025 – 18:27 UTC – The Chrome Root Program (CRP) Team posts an email message to the individual contact addresses disclosed to the CCADB.
May 12, 2025 – 07:00 UTC – WebPKI team begin a preliminary investigation.
May 12, 2025 – 09:30 UTC - The correct status has been set in CCADB.
May 12, 2025 – 09:46 UTC – Piotr Grabowski of the KIR WebPKI team responds to The Chrome Root Program (CRP) team, reporting that the correct status (revoked) is already set in CCADB.
May 13, 2025 – 09:30 UTC – Operational procedure for disclosing/updating CCADB was updated. Section for changing certificate statuses (revocation) and related actions has been added.
May 13, 2025 – 07:00 UTC – Preliminary Incident Report

Related Incidents

Root Cause Analysis

• Contributing Factor #1: CCADB update failure
• Description: In preparing an internal procedure for disclosing/updating CCADB, we focused on recording new certificates for subCAs. Our operating procedure for disclosing/updating CCADB was too general and did not directly address updating the status of revoked subCA's in CCADB. We did not clearly describe that in the case of revocation, the revocation status must be also updated within 7 days. After the revocation of Szafir Trusted CA3 certificate, which was carried out as planned, we generated a CRL, while the revocation was not reported to CCADB.
• Timeline: We have already modified our operating procedure to include detailed steps for updating the status of subCA certificates in CCADB
• Detection: Third Party Reported - The Chrome Root Program (CRP) Team
• Interaction with other factors:
• Root Cause Analysis methodology used: 5-Whys

Lessons Learned

• What went well:
  Prompt action was taken to update the CCADB status upon notification

• What didn’t go well:
  The internal procedure did not include sufficient checks to ensure timely CCADB updates in case of revocation

• Where we got lucky:

• Additional:

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Oprational procedure for disclosure/updating statuses to CCADB was updated	Prevent	Root Cause # 1		2025-05-13	completed
Training for WebPKI team in use of updated operational procedure	Prevent	Root Cause # 1		2025-05-19	Ongoing

Based on Incident Reporting Template v. 3.0

Comment 2

[chrome-root-program]

Thank you for providing this report. However, it should be updated in several areas.

(1) The “Relevant Policies” Section intends to describe the policy name(s), applicable version(s), and corresponding section(s) that result in this problem being diagnosed as an incident. This report lists the Chrome Root Program and CCADB Policies, but omits other Root Program policies, which should also be considered. Clearly identifying all of the relevant policies impacted in the incident creates opportunities for consolidating and/or clarifying source language.

(2) The “Timeline” Section appears to omit the original CPR submitted by the Chrome Root Program Team on May 2, 2025 (~14:09 UTC) using the KIR problem reporting address disclosed to the CCADB. Further, on May 5, 2025 (~13:07 UTC) we sent communication to the CA Email Alias address disclosed to the CCADB to draw attention to the CPR sent the week prior. These notifications should be added to the timeline of this report. The observed CPR response timeline also violates the expectations of the TLS BRs.

(3) The “Related Incidents” Section is incomplete. A quick (and probably incomplete) search highlights several related incidents, including one from KIR failing to disclose an intermediate certificate within 7 days. This section of the report should be populated in accordance with the CCADB IRGs.

(4) In the “Root Cause Analysis” Section we would like to better understand how this incident avoided detection with consideration of the action items detailed in 1921596 where operational procedure and training updates were described Action Items intending to Prevent CCADB disclosure failures from happening. Also, it seems relevant to highlight that training alone is typically an insufficient response to compliance failures. This has been stated in multiple incident reports over the years [1][2][3][4][5] and is also representative of 1921596 and this report.

(5) The “Action Items” Section seems to repeat the actions from 1921596 and offers little confidence that this type of non-compliance will not recur. We would encourage a more robust set of changes, possibly after completing a more detailed root cause analysis, that aligns with the CCADB IRGs (e.g., actions for each of the three kinds and detailed evaluation criteria for how the CA Owner and the public can measure the effectiveness and impact.)

Comment 3

[Waldemar Brzozowski]

Thank you for the submitted comments

1. We added a reference to the Mozilla Root Store Policy

• Relevant policies:
• According to Section 6 of the Chrome Root Program Policy states Chrome Root Program: “Disclose revocation of all subordinate CA certificates capable of validating to a certificate included in the Chrome Root Store or associated with a Root Inclusion Request to the CCADB within 7 calendar days of revocation.”
• According to Section 4 CCADB Policy: "If a subordinate CA certificate is revoked, the CCADB MUST be updated to mark it as revoked, including the reason for revocation, within seven calendar days of revocation."
• According to section 4 Mozilla Root Store Policy : "CA operators with certificates in Mozilla’s root store MUST use the CCADB, and are bound by the latest published version of the CCADB Policy."

2. Timeline
   We have verified it. We were able to determine that the first notification by the Chrome Root Program Team was in fact sent on May 2.
   Due to internal problem the notification was sent by our Contact Center on the 12th May. This was due to the large number of cases in the ContactCenter. We examined this case deeply and registered a new incident. We have not received an e-mail which you mentioned dated May 5. Until now it could not be traced. Staff managing the email server are verifying this.

Updated Timeline below

March 05, 2025 – 12:55 UTC – Revocation of Szafir Trusted CA3 certificate according to the closure plan of Szafir Trusted CA3
March 13, 2025 – 12:55 UTC – Expected CCADB disclosure deadline ( 7 days )
May 02, 2025 – 14:09 UTC – The Chrome Root Program (CRP) Team posts an email message to the general KIR problem reporting address. Notification to the PKI Team was redirected May 12.
May 05, 2025 – 14:09 UTC - The email was not found in KIR's systems. We asked email server administrators to check.
May 09, 2025 – 18:27 UTC – The Chrome Root Program (CRP) Team posts an email message to the individual contact addresses disclosed to the CCADB.
May 12, 2025 – 07:00 UTC – WebPKI team begin a preliminary investigation.
May 12, 2025 – 09:30 UTC - The correct status has been set in CCADB.
May 12, 2025 – 09:46 UTC – Piotr Grabowski of the KIR WebPKI team responds to The Chrome Root Program (CRP) team, reporting that the correct status (revoked) is already set in CCADB.
May 13, 2025 – 09:30 UTC – Operational procedure for disclosing/updating CCADB was updated. Section for changing certificate statuses (revocation) and related actions has been added.
May 13, 2025 – 07:00 UTC – Preliminary Incident Report
May 13, 2025 – 11:21 UTC – Incident Report

3. Following you instruction we have updated KIR incidents related to this incident.

• Related Incidents

Bug	Date	Description
1921596	2024-09-28	No detailed steps in the procedure to disclose new certificates in the CCADB database
1921598	2024-09-28	The reason for revocation of Szafir Trusted CA3, resulting in the failure to update the revocation entries in CCADB

4. We recognize that training and changing one of the points alone in procedure is not a sufficient preventive measure, as confirmed by similar incidents. We have analyzed the causes once again and added one item to the action plan. This is a regular review of CCADB entries by the PKI Team (once a month) and immediately after each significant change in the PKI structure. This is a preventive factor and mitigating potential problems with the current status of data in CCADB

5. We have updated Action Items related to this incident.

• Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Oprational procedure for disclosure/updating statuses to CCADB was updated	Prevent	Root Cause # 1		2025-05-13	completed
Training for WebPKI team in use of updated operational procedure	Prevent	Root Cause # 1		2025-05-19	completed
Regular review of entries in the CCADB database (once a month) and after any major change in the PKI system	Prevent / Mitigate	Root Cause # 1		2025-05-21	completed

Comment 4

[Waldemar Brzozowski]

All action items have been completed.
We have no further updates here.

Comment 5

[incident-reporting]

If requesting closure of this report, please follow the guidance on CCADB.org.

Comment 6

[Waldemar Brzozowski]

Report Closure Summary

• Incident description:
  The intermediate certificate Szafir Trusted CA3 was revoked on March 5, 2025, in accordance with KIR's planned decommissioning schedule.
  However, the revocation status of the certificate was not updated in the CCADB within 7-day window required by Section 6 of the Chrome Root Program Policy and CCADB Policy.

• Incident Root Cause(s):
  Internal procedure for disclosing/updating CCADB was too general and did not directly address updating the status of revoked subCA's in CCADB. As a result, no team member performed the necessary update in CCADB.

• Remediation description:

KIR updated its operational procedure for disclosure/updating statuses to CCADB on May 13, 2025. WebPKI team members received training by May 19, 2025.
Routine monthly checks of CCADB entries and after any significant PKI updates were introduced by May 21, 2025.

• Commitment summary:
  KIR confirms that all action items have been implemented.
  KIR undertakes to regular review of entries in the CCADB database and after any major change to ensure compliance.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 7

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-01.