1902042 - Siemens: meaningless characters in personal name fields

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[Thorsten Bergmann]

On Monday, 2024-06-10, 12:04 UTC, we received a certificate problem report from Entrust after a check on a sample of issued certificates under the Siemens SMIME CA Hierarchy.

The affected certificates depict meaningless characters such as ¨.¨ or ¨-¨ in the Common Name, Givenname or Surname fields.
Initial investigation showed that the issue was caused by Indian users with only one name in their passports.
Configuration was changed on 2024-06-12 to prevent problematic issuing.
We will provide more details in the full incident report.

• 342 certificates affected issued between 2023-09-08 and 2024-06-12 under the Legacy Profile of the SMIME BRs.
• 33 certificates have been revoked before the detection of the incident.
• There were no expired certificates affected.

All affected certificates will be revoked within the 5 day timeline by Friday, June 14.

A full incident report will be published latest by 2024-06-24, before 12:00 UTC.

Comment 1

[Marco Fechter]

The revocation of all affected certificates was completed.

Comment 2

[Thorsten Bergmann]

FINAL REPORT

Incident Report

Summary

On Monday, 2024-06-10, 12:04 UTC, we received a certificate problem report from Entrust after checking a sample of issued certificates under the Siemens S/MIME CA Hierarchy.

The affected certificates depict abstract characters such as ¨.¨ or ¨-¨ in the Common Name, Givenname or Surname fields.

After investigating the issue further, we were able to confirm that:

• 358 affected certificates issued between 2023-09-08 and 2024-06-14 under the Legacy Profile of the S/MIME BRs.
• 41 certificates had been revoked before the detection of the incident.
• There were no expired certificates affected.

After the preliminary report was published, we identified 16 additional certificates which were not included in the preliminary statistic.

These certificates were issued by the following CAs:

• Siemens Issuing CA EE Auth 2023: 166 certificates
• Siemens Issuing CA EE Enc 2023: 165 certificates
• Siemens Issuing CA Medium Strength Authentication 2023: 12 certificates
• Siemens Issuing CA EE Network Smartcard Auth 2023: 15 certificates

After the incident was confirmed, we started working on a mitigation patch which was deployed on 2024-06-12 16:00 UTC.
By 2024-06-14 06:30 UTC Continued monitoring of the system revealed that two more non-compliant certificates were issued despite the mitigation patch being in place. These certificates were revoked immediately. The error in the mitigation patch was identified and corrected by 2024-06-14 07:00 UTC. By 2024-06-14 09:00 UTC additional 14 certificates were identified which had been issued before 2024-06-12 16:00 UTC. This was due to a human error in the initial search query to retrieve the affected certificates for the preliminary statistic. By 2024-06-14 15:05 UTC all affected certificates had been revoked.

A list of the serial numbers and SHA-256 hashes of the affected certificates is attached to this bug.

Impact

Requirements: Special characters as placeholders are not compliant to section 7.1.4.2.2.a and 7.1.4.2.2.e of the S/MIME Baseline Requirements. As stated in the requirements, the given and surname fields should contain the names that are present on the official documents of the subscriber. In the case that a subscriber only has one legal name, that name should be stated in the surname field.
Not fulfilling the requirements, resulted in the issuing and revocation of 358 non-compliant certificates.

• 358 affected certificates issued between 2023-09-08 and 2024-06-14 under the Legacy Profile of the S/MIME BRs.
• 41 certificates had been revoked before the detection of the incident.
• There were no expired certificates affected.

Timeline

All times are UTC.

2024-06-10:

• 12:04 E-Mail received from Entrust to bring this incident to our attention
• 12:05 Start of investigation
• 17:33 Confirmation of the problem to Entrust
• 17:33 Start root cause analysis

2024-06-11:

• 10:00 Root cause identified by tracing the full end to end chain of connected systems and involved processes down to the data source.
• 10:15 Coordination of timeline for mitigation
• 10:20 Start working on mitigation patch
• 12:00 Started communication to the affected subscribers

2024-06-12:

• 15:50 Publication of preliminary incident report
• 16:00 Deployment of mitigation patch
• 16:10 Start extended monitoring by periodically checking newly issued certificates

2024-06-13:

• 10:38 Communication to the affected end-users, how their certificates can be renewed

2024-06-14:

• 06:30 Monitoring identified two additional non-compliant certificates issued
• 06:33 Revocation of those two affected certificates
• 07:00 Identification and correction of the error in the mitigation patch
• 09:00 Fixed error in initial search query identified 14 additional affected certificates issued before 2024-06-12 16:00 UTC
• 15:05 Completed revocation of all affected certificates
• Continued extended monitoring

Root Cause Analysis

The root cause of this incident was a combination of business logics in HR processes on how to handle naming conventions and cultural differences regarding single legal names. For users who have only one legal name, placeholders were used to fill the givenname or surname fields in the HR systems. Those data were used to issue the certificates.

In place pre-issuance linting and data quality checks did not detect the wrong data. Therefore, the certificates were issued with the same.
With the mitigation patch, the quality checks have been improved. The patch does identify placeholder characters in the givenname or surname fields and rejects the issuing request.

Lessons Learned

What went well

• The identification of the root cause and development of mitigation plan was quick.
• Communication and collaboration with Entrust was target-oriented and efficient.
• Publishing of preliminary report and revocation of affected certificates were completed within the expected timeframe.

What didn't go well

• Due to a human error in the first search query, the statistic in the preliminary report was incorrect. The initial query used a wrong search parameter, which led to the incorrect result. After this was corrected and the query updated, all affected certificates have been reported.
• We acknowledge that the implementation of the mitigation should have been faster than it was. Thus, we will review our incident response processes to make sure we can react faster and to deploy mitigations instantly in the future.
• Although we implemented zlint for pre-issuance linting, this issue was not detected. After further analysis we found out, that zlint does implement a check (https://github.com/zmap/zlint/blob/04d863f7660edfe0498162334524742397226fb2/v3/lints/cabf_br/lint_subject_contains_noninformational_value.go#L40)for noninformational characters. However, this is not used for the S/MIME BRs. We will address this topic to the open-source linters.

Action Items

• Improve certificate issuing process:
  We are working on a fix which will write the users' single legal name according to the S/MIME BRs, into the surname field and exclude the given name field from the certificate and ignore single character placeholders or empty fields.
  Clarification with vendor ongoing. Update of target due date will follow shortly.

• Improve incident response process:
  Review and update the internal incident response process to become faster and to react within the expected time.

• Report linting topic:
  Address linting topic regarding zlint to open source community.

Action Item	Kind	Due Date
Improve certificate issuing process	Prevent	TBD
Improve incident response process	Mitigate	2024-06-28
Report linting topic	Prevent	2024-06-28

Comment 3

[Thorsten Bergmann]

Attachment: List of affected certificates

Comment 4

[Thorsten Bergmann]

Update on Action items:

• Improve certificate issuing process (ongoing) - technical discussions with vendor almost completed. Next step is to align on the implementation timeline which results in the final due date.

• Improve incident response process (ongoing) - will be completed latest by June 28.

• Report linting topic (completed) - issue addressed to community: https://github.com/zmap/zlint/issues/863

Comment 5

[Thorsten Bergmann]

Update on Action items:

• Improve certificate issuing process (ongoing) - technical discussions completed. Alignment on implementation and deployment timeline ongoing. Next update on this topic including final due date latest by July 19th.

• Improve incident response process (completed) - Improvements implemented and active. We have reviewed our internal processes and identified 2 main areas of improvement:

1. Responsibilities: We have reviewed and assigned clear responsibilities (who will be informed and who needs to do what and by when) to the respective roles (service owner and manager) within our organization.
2. Tooling: We have onboarded this kind of incident to our internal incident management process based on ServiceNow. This guarantees that the responsible people are informed in time and can react immediately according to the expected timelines.