1999296 - Telia: Findings in 2025 ETSI Audit - Incident Report #1 – Vulnerability management

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Antti Backman]

Preliminary Incident Report

Summary

• Incident description:
  • Non-conformity: There is no evidence that critical vulnerabilities identified in penetration tests from 13.10.2025 were addressed in due time.
  • Identified in audit session conserning Telia CA's Swedish RA responsible for S/MIME certificate issuance that identified critical vulnerabilities on Apache HTTP server serving S/MIME certificate management portal were not properly addressed in 48 hours as required by ETSI EN 319 401.
• Relevant policies: ETSI EN 319 401 v3.1.1, REQ-7.9.2-09X
• Source of incident disclosure: Telia CA Annual Audit 2025

Full incident report will be submitted no later than Friday the 21st of November 2025.

Comment 1

[Antti Backman]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000055

• Incident description:

  • Non-conformity: There is no evidence that critical vulnerabilities identified in penetration tests from 13.10.2025 were addressed in due time. Identified in audit session conserning Telia CA's Swedish RA responsible for S/MIME certificate issuance that identified critical vulnerabilities on Apache HTTP server serving S/MIME certificate management infrastructure patch download and distribution were not properly addressed in 48 hours as required by ETSI EN 319 401.

• Timeline summary:

  • Non-compliance start date: 2025-10-18
  • Non-compliance identified date: 2025-11-05
  • Non-compliance end date: 2025-11-11

• Relevant policies:

  • ETSI EN 319 401, REQ-7.9.2-09X

• Source of incident disclosure:

  • Telia CA Annual Audit 2025

Impact

• Total number of certificates: N/A
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: N/A
• Incident heuristic: N/A
• Was issuance stopped in response to this incident, and why or why not?: N/A
• Analysis: N/A
• Additional considerations: N/A

Timeline

• 2025-10-13: Vulnerability scan was performed.
• 2025-10-15: Review of vulnerability scan report.
• 2025-10-18: Non-conformity start date
• 2025-11-05: Auditor initially identify findings. Discussion about findings.
• 2025-11-10: Non-Compliance verified date after discussion with auditors.
• 2025-11-10: Initiating work for identifying and remedying root causes of the non-conformity
• 2025-11-10: Preliminary incident report submitted
• 2025-11-11: Immediate remediation plan created to address the affected server for decommissioning.
• 2025-11-11: Shuts down the affected server and make a request to install a new server.
• 2025-11-11: Begin detailed analysis and preparation of Full Incident Report
• 2025-11-12: Installation preparations started for new installation of server
• 2025-11-19: Auditors updated on the completed remediation of the non-conformity.
• 2025-11-21: Submit Full Incident Report

Related Incidents

Bug	Date	Description
Bug 1983272	2025-08-15	Unpatch / outdated software
Bug 1972158	2025-06-13	Detection of vulnerability and identifying the rating of vulnerability
Bug 1906028	2024-07-02	Addressing vulnerabilities, plans and timelines
Bug 1955721	2025-03-21	Addressing vulnerabilities within required timeframes

Root Cause Analysis

Contributing Factor #1: Manual process for software update management

• Description:

  • The process for patching software versions and ensuring consistent deployment across all systems is not automated, contributing the affected server not being patched as required.

• Timeline:

  • Existing practice up until the identified action item to include all servers in the automated routine completed 2025-11-21.

• Detection:

  • Audit 2025

• Interaction with other factors: #4

• Root Cause Analysis methodology used: Internal investigation and review with required personnel to identify Root Causes for the non-conformity.

Contributing Factor #2: Lack of knowledge regarding the vulnerability requirements

• Description:
  • The vulnerability was not properly addressed in 48 hours as required by ETSI EN 319 401 due to lack of knowledge about the regarding requirements. Person responsible of reviewing and acting on the vulnerabilities was not aware that the 48-hour rule applied to support systems that do not expose their interface to the internet
• Timeline:
  • From the appointment of the new responsible person to 2025-05-11 (audit session where the non-conformity identified)
• Detection:
  • Audit 2025
• Interaction with other factors: #3
• Root Cause Analysis methodology used: Internal investigation and review with required personnel to identify Root Causes for the non-conformity.

Contributing Factor #3: Lack of proper handover process

• Description:
  • Due to recent changes and reorganization, the appointed staff who took over responsibility for addressing critical vulnerabilities did not receive sufficient training or handover from previous personnel leaving the company.
• Timeline:
  • From the appointment of the new responsible person to 2025-05-11 (audit session where the non-conformity identified)
• Detection:
  • Audit 2025
• Interaction with other factors: #2
• Root Cause Analysis methodology used: Internal investigation and review with required personnel to identify Root Causes for the non-conformity.

**Contributing Factor #4: Missing support server the inventory of servers included in recurring patching **

• Description:
  • The server was not included in the regular patching scheme. The server was inadvertently excluded from the patch list due to an oversight in our inventory validation process. At the time of planning, the system was not properly registered in the patch scope documentation.
• Timeline:
  • Existing practice up until the identified action item to include all servers in the automated routine completed 2025-11-21
• Detection:
  • Audit 2025
• Interaction with other factors: #1
• Root Cause Analysis methodology used: Internal investigation and review with required personnel to identify Root Causes for the non-conformity.

Lessons Learned

• What went well:

  • Vulnerability scanning / penetration testing on itself proven capable of detecting vulnerabilities as expected.

• What didn’t go well:

  • Vulnerabilities identified in penetration tests from 2025-10-13 were not addressed in time leading up to non-conformity finding in our audit.
  • The server was not included in the regular patching scheme.

• Where we got lucky:

• Additional:

  • We continue to treat all findings with due diligence, the isolation of the environment provides a strong mitigating factor against the identified risks.
  • Critical vulnerabilities were found on Apache HTTP server serving patches to other servers and is only used when new patches are needed to be installed. The webserver does not face internet and is a part of an isolated infrastructure which significantly limits the potential for exploitation. Access to this environment requires multiple layers of physical and logical security controls, making direct exploitation of the identified vulnerabilities unlikely under current conditions (MFA, 4 eyes principle).

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Make sure that all servers (including support systems, like dedicated Apache/repository server) are included in the inventory of recurring patching to ensure timely updates	Prevent	Root Cause #4	Criteria	2025-11-21	Complete
Update the internal vulnerability practices to secure adherence to the requirements	Prevent	Root cause #2	The practice documentation updated, approved and distributed to relevant personnel	2025-11-13	Complete
Conduct training to ensure that the updated practice is understood by all relevant people	Mitigate	Root causes #2 and #3	Training completed	2025-12-02	Ongoing
Shut down and decommission the affected Apache server related to the vulnerability	Prevent	Root cause #2	Server shut down performed and verified	2025-11-11	Complete
Deploy new Apache/repository server	Mitigate	Root cause #2	New repositry server deployed and used in automated patching successfully	2026-01-01	Ongoing
Interim solution: Direct server patching is in place until the new repository server goes live	Mitigate	Root cause #2	Servers configured for automated patching	2025-11-21	Complete
Review our internal inventory for patch management procedures (recurring job) to ensure all servers are correctly identified, categorized in the inventory before each maintenance window	Prevent	Root cause #4	All servers (physical and virtual) are listed in the asset management system. Review process documented and followed before each maintenance window	2025-11-21	Complete

Appendix

Comment 2

[Dustin Hollenback (Apple)]

Since this is an S/MIME CA, the S/MIME Baseline Requirements incorporate the NCSSRs, which require timely remediation of critical vulnerabilities. Could you confirm whether the NCSSRs were impacted by this bug and why they weren’t listed under ‘Relevant Policies’?

Comment 3

[Antti Backman]

Hi Dustin

In response to comment #2.

We believe your question is related to :

NCSSR v1.7 section 4 j) requiring to address critical vulnerabilities in 96 hours.

Confirming that this deadline was also missed.

Why the relevant policies section did not include the NCSSR requirement?

The identified non-conformity was to ETSI EN 319 401, REQ-7.9.2-09X, which does not reference the normative reference [i.3 CA/B NCSSR] in ETSI 319 401, therefore the Relevant Policies section only reference the particular ETSI requirement.

We do admit that we could have included the NCSSR reference also and thank you for pointing that out by your observation.

Comment 4

[Antti Backman]

This is our weekly update on progress of pending action items.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Deploy new Apache/repository server	Mitigate	Root cause #2	New repositry server deployed and used in automated patching successfully	2025-11-27	Complete

Last pending action item is in schedule to be completed 2025-12-02 as planned.

Comment 5

[chrome-root-program]

Thank you for filing this report. We have a few questions:

Q1: Can you please help clarify the timeline presented in Comment 1?

• The “Timeline summary” states the non-compliance was identified on 2025-11-05.
• This report was opened on 2025-11-10.

Should we interpret this to mean that this report disclosure was delayed, considering the requirements included in the CCADB Incident Reporting Guidelines?

Q2: Can you help us understand the delay between the following “Timeline summary” items?

• 2025-11-05: Auditor initially identify findings. Discussion about findings.
• 2025-11-10: Non-Compliance verified date after discussion with auditors.

Specifically, what took place between these dates?

Q3: Can you describe whether the qualified auditor(s) considered the timeliness of all vulnerability management activities performed by Telia over the audit period related to in-scope systems, or just the period where they were actively involved in evaluating Telia’s policies and practices (often described as the "audit dates" in the ACAB’c AAL template)?

Comment 6

[Antti Backman]

In response to comment #5 by Chrome root program.

Q1 and Q2, We thank you for your detailed observation, the timeline presented is missing on relevant event for November the 7th. We received the written audit conclusion from the auditors on the Friday the 7th confirming their findings that were verbally discussed on the last audit onsite day the 5th of November. The conclusions verified and provided details of the identified non-conformity. After reviewing the conclusions preliminary on Friday the 7th we had final confirming meeting internally on Monday the 10th of November prior submitting our preliminary audit report. The delivery of the conclusions started our countdown of three days to submit the preliminary incident report as required by the IRGs in CCADB Incident Reporting Guidelines.

We understand that the missign date / event from the timeline raised question of the possible delay, but with above amendments and details we hope being able to clarify the course of events.

Q3, The audit was performed for all in-scope systems for the whole audit period from November the 1st 2024 to October the 31rd 2025 including all vulnerability management activities. This audit is also our re-certification audit for ETSI certification.

Comment 7

[Antti Backman]

This is to update the status of the last pending action item as completed.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Conduct training to ensure that the updated practice is understood by all relevant people	Mitigate	Root causes #2 and #3	Training completed	2025-12-02	Complete

We'll start preparing for the closure report and while doing so we'll follow this incident for question and comments.

Comment 8

[chrome-root-program]

[In response to Comment 6]

Q1 and Q2, We thank you for your detailed observation, the timeline presented is missing on relevant event for November the 7th. We received the written audit conclusion from the auditors on the Friday the 7th confirming their findings that were verbally discussed on the last audit onsite day the 5th of November. The conclusions verified and provided details of the identified non-conformity. After reviewing the conclusions preliminary on Friday the 7th we had final confirming meeting internally on Monday the 10th of November prior submitting our preliminary audit report. The delivery of the conclusions started our countdown of three days to submit the preliminary incident report as required by the IRGs in CCADB Incident Reporting Guidelines.

Thank you for this explanation. However, the CCADB IRG’s state (emphasis ours):

“Within 72 hours of a CA Owner becoming aware of an incident (i.e., the “initial incident disclosure”) or an audit finding not previously disclosed in an Incident Report…”

We interpret the timeline provided and the explanation offered in response to our questions to indicate Telia became aware of this issue on 05 November. Given the verbal discussion on 05 November, and acknowledging we do not know the details of that discussion, it seems Telia could have independently concluded the non-compliance instead of waiting for the auditor’s written report to start the disclosure clock. Receiving written confirmation from the auditor should not alter the facts of the technical non-compliance, nor does it change the date on which Telia became aware of those facts.

Waiting for a final or written artifact from an auditor that “officially” communicates a finding could arbitrarily delay the disclosure process (in some cases upward of 90 days) - which is why the IRGs specifically consider the clock starting when the CA becomes aware.

Comment 9

[Antti Backman]

This is our weekly update and response to comment #8.

We thank you Chrome Root Program to provide further clarity to expectations to meet set time frames in CCADB IRGs.

Per the clarification we have submitted this incident report for delayed submission of preliminary audit incident report.

For our weekly update, there's no updates at this time as we have completed all action items. We're preparing the closure report.

As all action items are completed, we request the NextUpdate to be set to Friday the 19th of December 2025.

Comment 10

[Antti Backman]

Report Closure Summary

• Incident description:

  • Non-conformity: There is no evidence that critical vulnerabilities identified in penetration tests from 13.10.2025 were addressed in due time. Identified in audit session conserning Telia CA's Swedish RA responsible for S/MIME certificate issuance that identified critical vulnerabilities on Apache HTTP server serving S/MIME certificate management infrastructure patch download and distribution were not properly addressed in 48 hours as required by ETSI EN 319 401.

• Incident Root Cause(s):

  • #1: Manual process for software update management

    • The old repository sever was shutted down as soon as we have been aware of the vulnerability. A new repository server deployed and used in automated patching successfully.

  • #2: Lack of knowledge regarding the vulnerability requirements

    • Training was conducted to ensure that the updated practice is understood by all relevant people. Remediation timeframe and CVSS-scroing was presented during the training. Also, the practice documentation have been updated, approved and distributed to relevant personnel to ensure that everyone is aware of the requirements.

  • #3: Lack of proper handover process

    • Practice documentation have been updated, approved and distributed to relevant personnel.

  • #4: Missing support server the inventory of servers included in recurring patching

    • Server added to the inventory of servers and are now patched and will be patched reguarly as standard procedure.

• Remediation description:

  • Processes and practices have been updated with relevant information regarding way of working for specific roles and on a team level. Also, an inventory of the current patching schedule has been done to confirm that all relevant servers are included. Training for the whole team was held to review practices and train the new updates to the practice.

• Commitment summary:

  • Telia plans to review and update solutions that further strenghtens to automatically address vulnerabilities and actions around them to mitigate or prevent the risk of incidents arising and not being handled in a timely manner

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 11

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-12-29.