Closed Bug 1983267 Opened 1 year ago Closed 5 months ago

PKIoverheid: TSP KPN Findings in 2025 ETSI Audit - Incident Report #7 – Change Management

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: pkioverheid, Assigned: pkioverheid)

Details

(Whiteboard: [ca-compliance] [audit-finding])

Preliminary Incident Report

Summary

  • Incident Description:
    • Minor Non-conformity: Change Management
  • Relevant Policies:
    • ETSI 319 401 (REQ-7.7-03, -04)
  • Source of incident disclosure:
    • Annual ETSI Audit
Assignee: nobody → pkioverheid
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [audit-finding]

Full Incident Report - ETSI Finding #7 - Change Management

Summary

  • CA Owner CCADB unique ID: A000068

  • Incident description: The CAB noted that the established change management procedures weren’t always followed for system configuration changes. For Ansible changes a separate procedure had been devised but this hadn’t been formalized within KPN at the time of the audit. This was filed as a minor non-conformity on the audit statement.

  • Timeline summary:

    • Non-compliance start date: 11-Jul-2025

    • Non-compliance identified date: 11-Jul-2025

    • Non-compliance end date: Ongoing

  • Relevant policies:

    • ETSI 319 401 REQ-7.7-03: Change control procedures shall be applied for releases, modifications and emergency software fixes of any operational software and changes to the configuration which applies the TSP's security policy

    • ETSI 319 401 REQ-7.7-04: The procedures shall include documentation of the changes.

  • Source of incident disclosure: Finding by CAB during annual ETSI audit

Impact

  • Total number of certificates: N/A

  • Total number of "remaining valid" certificates: N/A

  • Affected certificate types: N/A

  • Incident heuristic: N/A

  • Was issuance stopped in response to this incident, and why or why not?: N/A (see point below)

  • Analysis: N/A

  • Additional considerations: KPN only operates legacy S/MIME-capable CAs, which have not issued any S/MIME certificates since 1 August 2023. At that time this was changed due to updated S/MIME regulations, under which email addresses and the EKU emailProtection were no longer included in publicly trusted certificates.

Timeline

  • Sep-2024: During an assessment it was determined devops changes should be registered in service management tooling.

  • 11-Jul-2025: Auditor identifies finding

  • 17-Jul-2025: Created Corrective Action Plan

  • 12-Aug-2025: Corrective Action Plan Approved by auditor

Related Incidents

N/A

Root Cause Analysis

Contributing Factor 1: API interface of service management tooling not yet operational

  • Description: During an assessment in 2024 it was determined devops changes should be registered in the main service management tooling. KPN was not aware that this was a noncompliance, as the devops changes are implemented via a controlled procedure. Because manually registering all of these changes was deemed undesirable due to the chance of errors or omission, a solution was devised in which the changes would be automatically forwared from Ansible to the main service management tooling. However, this would require significant time. For this, an internal exception was granted by KPN management, but the CAB noted that the current implementation was not in line with the ETSI requirements in EN 319 401.

  • Timeline: See main timeline

  • Detection: Audit finding by CAB.

  • Interaction with other factors: No.

  • Root Cause Analysis methodology used: N/A

Lessons Learned

  • What went well: Devops changes are implemented via a controlled procedure and this was registered with KPN management

  • What didn’t go well: N/A

  • Where we got lucky: N/A

  • Additional: N/A

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Implement API to maintain DevOps changes. Mitigate/Detect Root Cause # 1 Implementation of API and report back 2026-01-11 In progress

Appendix

N/A

PKIoverheid is monitoring this bug and we're open for additional questions or remarks people might have. Currently we don't have any updates with regards to the Action Items.

PKIoverheid is (still) monitoring this bug and we're open for additional questions or remarks people might have. Currently we don't have any updates with regards to the Action Items.

PKIoverheid is (still) monitoring this bug and we're open for additional questions or remarks people might have. Currently we don't have any updates with regards to the Action Items.

Could you confirm whether the NCSSRs were impacted by this bug and why they weren’t listed under ‘Relevant Policies’?

Hi Dustin, thanks for your reply. The NCSSRs were in scope for this audit (specifically version 2.0.3). The audit criteria under "relevant policies" bullet in this bug were directly taken from the audit report (statement of non-conformity). As to why the CAB didn't include the NCSSRs for this specific finding, we're not exactly sure, so KPN has asked the CAB (BSI) for clarification. This might take a few days. Thanks.

Thank you for confirming that NCSSRs v2.0.3 were in scope. For transparency, please clarify whether this finding relates to a non‑conformity with the NCSSRs and update the “Relevant Policies” section to reflect that.

I understand the CAB may not have listed the NCSSRs directly. At the same time, it is important for the CA to show its own awareness of these requirements, rather than relying only on the auditor’s report.

As a reminder, the CCADB Incident Reporting Guidelines set an expectation for timely engagement with the community. A request was made in Comment 7 and does not appear to have been responded to.

Flags: needinfo?(pkioverheid)

Hi all,

Apologies for the delay. We have gotten a response from BSI (the CAB) regarding the applicability of NetSec in general with regards to this audit finding and together with KPN have made an assessment about the applicability of specific NCCSR requirements. Our questions to BSI and their answer are posted below verbatim accompanied by our our assessment below that.

1. Should NCSSRs be explicitly mentioned in the report for this type of finding?

The audit was conducted as an assertion-based audit, where KPN confirmed compliance with the applicable requirements outlined in the Statement of Applicability and the Overview of Applicability. These requirements include ETSI standards, NCSSRs, and the PKIoverheid Programme of Requirements, among others. For reporting purposes, each non-conformity is referenced against the requirement that most directly addresses the issue. In this case, ETSI EN 319 401 REQ-7.7-01 fully covers the observed condition (unsupported software). Referencing ETSI ensures clarity and consistency in the report, while all applicable criteria were evaluated during the audit.

2. Do you consider this non-conformity a breach of NCSSRs, or is reporting under ETSI sufficient?

The non-conformity also falls under NCSSR obligations (e.g., timely remediation of unsupported components) and PKIOverheid requirements. The audit team reviewed all applicable frameworks as part of the full-scope audit required by, for example, Mozilla Root Store Policy and Microsoft Trusted Root Certificate Program requirements. For reporting purposes, the finding was mapped to ETSI EN 319 401 because it most directly addresses the issue. This does not diminish the relevance of other frameworks, which remain part of the audit scope. In general, a TSP should ensure that any corrective action plan addresses compliance across all obligations.

As to the specific NCCSR criteria applicable to this finding, we’ve concluded that "1.3 – Change Management" is applicable in this case. We’ll update this bug accordingly.

Flags: needinfo?(pkioverheid)

As indicated in our previous post we hereby restate our incident report, now fully reflecting the applicable audit criteria including the NCSSRs. All other information is unchanged compared to the original report (comment #1)

Full Incident Report - ETSI Finding #7 - Change Management

Summary

  • CA Owner CCADB unique ID: A000068

  • Incident description: The CAB noted that the established change management procedures weren’t always followed for system configuration changes. For Ansible changes a separate procedure had been devised but this hadn’t been formalized within KPN at the time of the audit. This was filed as a minor non-conformity on the audit statement.

  • Timeline summary:

    • Non-compliance start date: 11-Jul-2025

    • Non-compliance identified date: 11-Jul-2025

    • Non-compliance end date: Ongoing

  • Relevant policies:

    • ETSI 319 401 REQ-7.7-03: Change control procedures shall be applied for releases, modifications and emergency software fixes of any operational software and changes to the configuration which applies the TSP's security policy

    • ETSI 319 401 REQ-7.7-04: The procedures shall include documentation of the changes.

    • Network and Certificate System Security Requirements section 1.3 - Change management: [...]The CA MUST ensure that all changes are completed in accordance with such a change management process for: [...] CA Infrastructure."

  • Source of incident disclosure: Finding by CAB during annual ETSI audit

Impact

  • Total number of certificates: N/A

  • Total number of "remaining valid" certificates: N/A

  • Affected certificate types: N/A

  • Incident heuristic: N/A

  • Was issuance stopped in response to this incident, and why or why not?: N/A (see point below)

  • Analysis: N/A

  • Additional considerations: KPN only operates legacy S/MIME-capable CAs, which have not issued any S/MIME certificates since 1 August 2023. At that time this was changed due to updated S/MIME regulations, under which email addresses and the EKU emailProtection were no longer included in publicly trusted certificates.

Timeline

  • Sep-2024: During an assessment it was determined devops changes should be registered in service management tooling.

  • 11-Jul-2025: Auditor identifies finding

  • 17-Jul-2025: Created Corrective Action Plan

  • 12-Aug-2025: Corrective Action Plan Approved by auditor

Related Incidents

N/A

Root Cause Analysis

Contributing Factor 1: API interface of service management tooling not yet operational

  • Description: During an assessment in 2024 it was determined devops changes should be registered in the main service management tooling. KPN was not aware that this was a noncompliance, as the devops changes are implemented via a controlled procedure. Because manually registering all of these changes was deemed undesirable due to the chance of errors or omission, a solution was devised in which the changes would be automatically forwared from Ansible to the main service management tooling. However, this would require significant time. For this, an internal exception was granted by KPN management, but the CAB noted that the current implementation was not in line with the ETSI requirements in EN 319 401.

  • Timeline: See main timeline

  • Detection: Audit finding by CAB.

  • Interaction with other factors: No.

  • Root Cause Analysis methodology used: N/A

Lessons Learned

  • What went well: Devops changes are implemented via a controlled procedure and this was registered with KPN management

  • What didn’t go well: N/A

  • Where we got lucky: N/A

  • Additional: N/A

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Implement API to maintain DevOps changes. Mitigate/Detect Root Cause # 1 Implementation of API and report back 2026-01-11 In progress

Appendix

N/A

Implementation is on schedule. No further updates.

This report has gone stale. As a reminder, CA Owners may request the “Next update” Whiteboard field be set by a Root Store Operator to align with a specific date related to an open Action Item.

Flags: needinfo?(pkioverheid)

PKIoverheid is monitoring this bug and we're open for additional questions or remarks people might have. Currently we don't have any updates with regards to the Action Items.

KPN has implemented the required changes in a staging (test) environment.

Unfortunately, to bring the changes to production cooperation from another department within the organization is needed (this is outside the control of the TSP unit) which has led to delays. Currently the estimate is that this will be completed by March 20, 2026 at the lastest. Of course, effort will be made to shorten this implementation time where possible. We'll update in the meantime if there are any developments.

Flags: needinfo?(pkioverheid)

Could the "next Update" flag for this bug be set to the new due date for action item #1? (2026-03-20). Thanks!

Whiteboard: [ca-compliance] [audit-finding] → [ca-compliance] [audit-finding] Next update 2026-03-20

KPN has reported that execution of the sole action item has been completed ahead of schedule. As such, remediation is complete. We'll be filing a closure request and summary shortly.

We're awaiting your closure request.
Thanks.

Whiteboard: [ca-compliance] [audit-finding] Next update 2026-03-20 → [ca-compliance] [audit-finding]

Apologies, please find the closure statement below.

Report Closure Summary

  • Incident description: The CAB noted that the established change management procedures weren’t always followed for system configuration changes. For Ansible changes a separate procedure had been devised but this hadn’t been formalized within KPN at the time of the audit. This was filed as a minor non-conformity on the audit statement.
  • Incident Root Cause(s): During an assessment in 2024 it was determined devops changes should be registered in the main service management tooling. KPN was not aware that this was a noncompliance, as the devops changes are implemented via a controlled procedure. Because manually registering all of these changes into the main service management tooling was deemed undesirable due to the chance of errors or omission, a solution was devised in which the changes would be automatically forwarded from Ansible to the main service management tooling. However, this would require significant time. For this, an internal exception was granted by KPN management, but the CAB noted that the current implementation was not in line with the ETSI requirements in EN 319 401 and noted this down as an minor non-conformity.
  • Remediation description: KPN has implemented a connection between Ansible changes and an API connected to the main service tooling, as such, future Ansible changes will automatically update records in the main service management tooling and an integrated overview is available of all changes.
  • Commitment summary: KPN is committed to consistently applying change management procedures for system configuration changes, including automated registration of DevOps changes. The process covers registration, authorization, execution, evaluation, and closure of changes. These improvements reduce the likelihood of recurrence.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-03-30.

Whiteboard: [ca-compliance] [audit-finding] → [close on 2026-03-30] [ca-compliance] [audit-finding]
Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Whiteboard: [close on 2026-03-30] [ca-compliance] [audit-finding] → [ca-compliance] [audit-finding]
You need to log in before you can comment on or make changes to this bug.