Closed Bug 2004668 Opened 10 months ago Closed 8 months ago

Telekom Security: Root-CA certificates published in PEM encoded format

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: stefan.kirch, Assigned: stefan.kirch)

Details

(Whiteboard: [ca-compliance] [policy-failure])

Preliminary Incident Report

Summary

  • Incident description:
    Telekom Security has linked to download points for the issuing Root CA certificates in the AIA of some Sub-CAs, but the Root CA certificates held there are encoded with PEM instead of DER as required.
  • Relevant policies:
    RFC 5280 Section 4.2.2.1
  • Source of incident disclosure:
    Third Party Reported
Assignee: nobody → stefan.kirch
Status: NEW → ASSIGNED
Whiteboard: [ca-compliance] [policy-failure]

Full Incident Report

Summary

  • CA Owner CCADB unique ID:
    A000059
  • Incident description:
    In 2008, Telekom Security (then T-Systems) issued two new Root CAs:
    T-TeleSec GlobalRoot Class 2
    T-TeleSec GlobalRoot Class 3
    These two new Root CA certificates have been available for download from Telekom Security's web servers ever since.
    In the Sub-CA certificates issued by these Root CAs, Telekom Security has linked the corresponding download point in the “caIssuers” attribute of the AIA extension.
    According to RFC 5280, the target file linked in “caIssuers” must be encoded in DER. However, the files offered by Telekom Security are PEM-encoded.
  • Timeline summary:
    • Non-compliance start date:
      2008-10-01
    • Non-compliance identified date:
      2025-12-06
    • Non-compliance end date:
      2025-12-08
  • Relevant policies:
    RFC 5280 Section 4.2.2.1
  • Source of incident disclosure:
    Third Party Reported

Impact

  • Total number of certificates:
    n/a, as there were no certificates misissued
  • Total number of "remaining valid" certificates:
    n/a, as there were no certificates misissued
  • Affected certificate types:
    n/a, as there were no certificates misissued
  • Incident heuristic:
    The potential impact of this incident could theoretically affect all instances that verify certificates issued by all Sub-CAs under the two Root CAs using the caIssuers within the AIA.
  • Was issuance stopped in response to this incident, and why or why not?:
    As there were no certificates misissued, issuance was not stopped.
  • Analysis:
    n/a as there is no delayed revocation of certificates
  • Additional considerations:
    As written above, the potential impact of this incident could theoretically affect all certificates under these two Root CAs using the caIssuers within the AIA. In practice, however, there has not been a single problem report in the last 17 years, so that the verifying instances are obviously able to process PEM-encoded files as well, and not just the DER-encoded files that are actually required.

Timeline

2008-10-01 Issuance and publication of the two Root-CAs in scope of this incident
2025-12-06, 00:32 Receipt of the problem report from a third party who identified the error based on research of affected CAs due to bug https://bugzilla.mozilla.org/show_bug.cgi?id=2004492
2025-12-06, 08:26 Forwarding the problem report within the Root/Compliance Team
2025-12-06, 12:00 Telephone conference within the Root/Compliance Team regarding this bug
2025-12-06, 12:43 Feedback via email to the reporting party
2025-12-08, 10:30 Telephone conference within the Root/Compliance Team regarding the Preliminary Incident Report as well as filing an incident within the incident management tool to resolve the issue
2025-12-08, 11:59 Resolution of the issue by replacing the target files on all affected servers
2025-12-08, 18:56 Filing the Preliminary Incident Report

Related Incidents

Our research indicates that the following recent bugs in Bugzilla are affected by the same issue:

Bug Date Description
2005567 2025-12-11 CA Certificates Published in PEM format
2005399 2025-12-10 caIssuers Returns PEM Encoded Certificate
2005149 2025-12-10 Certificate not published in DER Encoded Format
2004845 2025-12-08 CA Certificates Published in PEM format
2004733 2025-12-08 CA Certificate not published in DER Encoded Format
2004732 2025-12-08 AIA CA issuer field pointing to PEM encoded cert
2004699 2025-12-08 CA in AIA in PEM format
2004521 2025-12-05 CA Certificate not published in DER Encoded Format
2004492 2025-12-05 CA Certificate not published in DER Encoded Format

Root Cause Analysis

Since the error occurred over 17 years ago, it is difficult to determine the exact reasons for it today. We can therefore only assume that it was due to the fact that there was no explicit requirement in the work instructions to publish CA certificates in DER format, and there still isn't one. We are currently still researching whether we can find any old records on this and therefore have to postpone further statements on RCA for the time being.

Contributing Factor #1: Vague work instructions

  • Description:
    The work instructions do not contain an explicit requirement to publish CA certificates in DER format.
  • Timeline:
    From 2008 until now
  • Detection:
    We found the missing requirement when checking the work instructions due to this bug.
  • Interaction with other factors:
    n/a as it is difficult to determine further reasons leading to this bug after this long time
  • Root Cause Analysis methodology used:
    n/a

Lessons Learned

  • What went well:
    Although the error remained undetected for many years, it has apparently had no impact so far; we have not received a single error or problem report.
  • What didn’t go well:
    n/a as this incident apparently had no impact so far
  • Where we got lucky:
    n/a
  • Additional:
    n/a

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Updating the work instructions for the commissioning of CA certificates Prevent #1 Quality assurance of work instructions based on the dual control principle 2025-12-18 ongoing

Appendix

n/a

Action Item

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Updating the work instructions for the commissioning of CA certificates Prevent #1 Quality assurance of work instructions based on the dual control principle 2025-12-18 complete

Hello. I noticed that this bug is Third‑Party Reported. Since similar issues were disclosed by several other CAs last year, it’s unclear whether you are monitoring incident reports from the broader community. Public disclosures are intended to help all CA operators learn from each other’s findings.

Could you confirm whether you review other CAs’ incident reports as part of your internal processes? If so, a brief overview of how you track these reports and incorporate relevant lessons would be helpful. It would also be useful to understand how this issue was ultimately identified through a third‑party report rather than your internal monitoring.

Also, please be aware that your list of relevant incidents is missing several older similar incidents.

Dear Dustin, Thanks for your comments! Here's our feedback on it.

Could you confirm whether you review other CAs’ incident reports as part of your internal processes? If so, a brief overview of how you track these reports and incorporate relevant lessons would be helpful.

Yes, we review other CAs’ incident reports as part of our internal processes, as mentioned once in 1877388#c55:
Yes, we have a process to triage every bug in the "CA Certificate Compliance" component. Practically speaking, we check on a daily basis. Since we are on the mailing list, we receive an email for every new bug. Due to the time difference, however, we check most of the new bugs until the next morning. Formally, the review takes place at least every week (usually on Tuesdays). Each bug is documented within an Excel sheet and evaluated separately. This process has been in place and documented since the spring of 2018. The task is the responsibility of the "Root and Compliance Team".
It is common practice that there are actually 3 types of procedure:
a) The bug is classified as "not relevant" by the initial assessment.
b) The bug is potentially relevant and will be discussed and evaluated immediately or at least in our weekly "Root and Compliance Telco".
c) A bug remains open for a longer period in order to pay explicitly attention to whether something significant arises later.
The ongoing discussions in the bugs will be monitored in each of the 3 cases afterwards. We receive mails for every comment on every bug. These will all be processed by Tuesday at the latest and any new finding that is important to us will be documented in the comments field and re-evaluated accordingly.

It would also be useful to understand how this issue was ultimately identified through a third party report rather than your internal monitoring.

In this case, we did not have time to treat the bug through our normal process, as the bug mentioned in the certificate problem report was disclosed (from a German perspective) on Saturday morning (2025-12-06, 00:02 German time) and the certificate problem report reached us just two and a half hours later (2025-12-06, 02:32 German time). In other words, the certificate problem report alerted us more quickly over the weekend than the new bug in Bugzilla, which we would have dealt with during normal working hours the following week.

Also, please be aware that your list of relevant incidents is missing several older similar incidents.

We have researched the related bugs using the "Advanced search" in Bugzilla because we thought it was the safest way to search for all related bugs. When we searched bugs in the "CA Certificate Compliance" component that contain the strings "DER" or "PEM" and did not select a resolution status to find both fixed and open bugs, we found the bugs referred in comment#1. Obviously, Bugzilla doesn't return the fixed bugs when researching without a resolution status, so we should not have relied solely on this search, but we should have compared this search to the bug list in our Excel sheet
Renewed research has revealed that our list is missing the Bugs 1637093, 1637854, 1884461 and 1914466 from previous years. In total, this results in the following list:

Bug Date Description
2005399 2025-12-10 caIssuers Returns PEM Encoded Certificate
2005149 2025-12-10 Certificate not published in DER Encoded Format
2004845 2025-12-08 CA Certificates Published in PEM format
2004733 2025-12-08 CA Certificate not published in DER Encoded Format
2004732 2025-12-08 AIA CA issuer field pointing to PEM encoded cert
2004699 2025-12-08 CA in AIA in PEM format
2004521 2025-12-05 CA Certificate not published in DER Encoded Format
2004492 2025-12-05 CA Certificate not published in DER Encoded Format
1914466 2024-08-22 CA Certificates not published in DER Encoded Format
1884461 2024-03-08 CA Certificates not published in DER Encoded Format
1637854 2020-05-13 AIA CA Issuer field pointing to PEM encoded cert
1637093 2020-05-11 AIA CA Issuer field pointing to PEM encoded cert

In the end, the question arises as to why we didn't find our bug when we reviewed the bugs from previous years?
The bugs in 2020 and 2024 only concerned issuing CA certificates that were referenced in the AIA of end-entity certificates. In our records in the Excel sheet used to track the bugs, we noted at the time that we checked the CAs linked in the AIA in our end-entity certificates and found them all to be correct.
Therefore, we unfortunately did not take any further action to verify whether clear requirements were included in the relevant work instructions, nor did we check the Root CA certificates, which we should have done consequently, even though they were not part of the scope of the bugs.

We are monitoring this bug for feedback. Please let us know if there are any comments or questions.

We are monitoring this bug for feedback. Please let us know if there are any comments or questions.

All Action Items appear complete. If this report is ready for closure, please file a Closure Report.

Flags: needinfo?(stefan.kirch)

Report Closure Summary

  • Incident description:
    Telekom Security has linked to download points for the issuing Root CA certificates in the AIA of some Sub-CAs, but the Root CA certificates held there are encoded with PEM instead of DER as required.

  • Incident Root Cause(s):
    The people involved in the publication of the CA certificates were not aware of the requirement to publish CA certificates in DER format, as this was not explicitly mentioned in the work instructions.

  • Remediation description:
    The incident was remedied the next business day by replacing the target files on all affected servers. The Root Cause was remedied within 2 weeks by updating the work instructions for the commissioning of CA certificates.

  • Commitment summary:
    Since the requirements of BR, SBR, CCADB, Root Stores, ETSI, etc. are regularly reviewed due to ongoing changes, the requirements of older stable specifications, as the RFC5280 in this case, are not regularly reviewed again once they have been implemented. To this end, we will also regularly review aspects of the relevant RFCs and other stable specifications (ITU-T, ISO) in our training courses on current developments, in order to keep both existing staff and new colleagues sustainably well-informed.

Flags: needinfo?(stefan.kirch)

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-01-19.

Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [policy-failure] → [close on 2026-01-19] [ca-compliance] [policy-failure]
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Flags: needinfo?(incident-reporting)
Resolution: --- → FIXED
Whiteboard: [close on 2026-01-19] [ca-compliance] [policy-failure] → [ca-compliance] [policy-failure]
You need to log in before you can comment on or make changes to this bug.