1945867 - Izenpe: Incorrect Unicode characters in Subject

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[David]

This is a preliminary report regarding Unicode replacement characters in Subject.
We are still investigating the root causes of this missisuance.

Incident Report

Summary

Izenpe has issued a certificate with Unicode replacement in Subject which violates RFC5280.

Impact

Only one certificate has been detected.

Timeline

All times are UTC.

2025-02-03:

• 12:06 A certificate with wrong encoding in Subject has been issued.

2025-02-04

• 15:10 An email from chrome root program warns us about this issue.
• 15:32 The certificate has been revoked.

Root Cause Analysis

Lessons Learned

What went well

What didn't go well

Where we got lucky

Action Items

Action Item	Kind	Due Date

Appendix

Details of affected certificates

https://crt.sh/?id=16481040023

Comment 1

[David]

Incident Report

Summary

On 2025-02-03 Izenpe issued a certificate containing Unicode Replacement characters in the "Locality" and "Organization" fields of Subject.
Those characters belong to spanish alphabet (such as 'ñ', 'ü'...) and should have been properly encoded as it had been done previously with certificates issued to the same Organization.

An email from Chrome Root program raised the problem and the certificate was immediately revoked.

Impact

• The impact has been limited to only one certificate.
• Certificates issued over the last 13 months have been linted with pkimetal and no incidents have been detected.
• Only EV/OV certificates, with organization or locality names containing these characters would have been affected.

Timeline

All times are UTC.

2024-07-22

• A new version of our web application for certificate requests was deployed.

2025-01-13

• Zlint 3.6.5 was deployed in our infrastructure.

2025-02-03:

• 10:06 A certificate with wrong encoding in Subject was issued.
• 10:10 An email with the pkimetal analysis was received.

2025-02-04

• 15:10 An email from chrome root program warns us about this issue.

2025-02-04

• 15:44 The certificate was revoked and we stopped issuing EV/OV certificates.

2025-02-05

• 14:00 A new version of our web application was deployed fixing the problem.

Root Cause Analysis

The 22nd of July a new version of the web application for TLS certificates enrollment was deployed. This version included some new screens using a different encoding that finally have caused the issue.
Although all characters were properly shown on the screens, the final encoding was not UTF8 when the PKCS10 file was regenerated and sent to the CA software.

At this point, ZLINT did not detect this issue but PKIMETAL post linting did. Automatically an email was sent to the issuing team reporting the bug:

Linter	Severity	Finding	Field	Code
ftfy	ERROR	Unicode replacement character(s) present

but it was not took into account or notified to the PKI Team.

Lessons Learned

What went well

• Pkimetal post linting task detected the issue and an email with the error was received. and we were in the process of configuring pkimetal as the prelinting tool.

What didn't go well

• A minor change in the application has led to this issue and has not been detected in 6 months. Our testings do not include certificates containing the special characters.
• The issuing team ignored the email from pkimetal showing the issue.
• The last version of Zlint did not detect this issue.

Where we got lucky

• The number of Organizations or Localities in our database with special characters is really low.
• Domain names have been correctly punnycoded and were not affected by the change which introduced the issue.

Action Items

Action Item	Kind	Due Date
Stop issuing	PREVENT	2025-02-04
FIX Web application	MITIGATE	2025-02-05
Email Issuing Team about pkimetal errors and actions	PREVENT	2025-02-06
Add new test cases to test in development environment	PREVENT	2025-02-07
Configure Pkimetal in testing environment	DETECT	2025-02-14
Configure Pkimetal in production environment	DETECT	2025-02-28

Appendix

Details of affected certificates

https://crt.sh/?id=16481040023

Comment 2

[David]

We have some delays regarding the configuration of Pkimetal on our testing environment but we plan to have it the next week.

Comment 3

[David]

Attachment: pkimetal_integration_development.jpg

Comment 4

[David]

We have succeeded installing and configuring the plugin for pkimetal in our development environment. The only exceptions for issuance that have been applied are those related to the TLD's and CT. It has been also tested with the certificate that raised this bug and it has been successfully detected.

Comment 5

[David]

Hi,
pkimetal is now running in our Production environment and linting the precertificates. It has been configured for the three profiles we issue according to the pkimetal profiles:

• tbr_leaf_tlsserver_dv_precertificate
• tbr_leaf_tlsserver_ov_precertificate
• tevg_leaf_tlsserver_ev_precertificate

Regards,

Comment 6

[Zacharias]

Both here and in 1948600 one of the things that didn’t go well was that emails got ignored. One of your action items is to send more emails. It seems to me that perhaps you have a case of email (alert) fatigue. I noticed that your website defaults to eh, not-English. Maybe there is a language barrier that needs overcoming.

Have you considered how you can ensure that important emails don’t get ignored in the future?

Comment 7

[David]

Attachment: screenshot_emal.jpg

Comment 8

[David]

Hi,
Thank you for your appreciation. In my opinion these are two different situations. In this bug an email like this was received by the team who issues certificates. I think the email is clear enough (or it should have been) to "detect" something has gone wrong no matter in which language it was written. The reason why it was ignored was mainly because it was the first time they received one like that.
Regarding the other Bug, once again I would not say there is a problem with the language...just in case there is something that is not understod you can alwasys use any translator to figure out its meaning. In that case, we just did not check out the ccadb links to find out which were the oudated documents as we though they were related to latest CP we had uploaded.
In the first situation, we have warned by written the issuing team to not ignore any message received from PKIMETAL (specially now that it has been installed in the pre issuance) and to stop issuing and notify the problem.
For the other situation, as we have done in other situations (root programms changes, new ballots) we will create an internal task in our JIRA to follow the problem and take the necessary measures if needed.
Regards.

Comment 9

[David]

Hi,
since the installation of pkimetal, we have issued 41 certificates including at least one of each type (DV, OV and EV) and all of them have been successfully prelinted.
Regards

Comment 10

[Jeremy]

Point of order - CAs aren't allowed to set the Next update field. At least, I was told this at one point. Ben - has this policy changed?

Comment 11

[Ben Wilson]

Jeremy - that's correct - only root program representatives are supposed to set the "Next Update" in the whiteboard, while CAs are supposed to request that the Next Update be set to some date related to the accomplishment of a milestone or CA action item.

Comment 12

[David]

Hi,
It was my mistake as I misunderstood the information received. Now it is clear to me who can set the Next Update date.
Regards,

Comment 13

[David]

Hi,
I would like to ask to set the Next Update for 2025-04-11
Regards,

Comment 14

[David]

Report Closure Summary

• Incident description:

  On 2025-02-03, a certificate was issued containing Unicode replacement characters in Locality and Organization Subject fields.

• Incident Root Cause(s):

  A change introduced in the application that manages the SSL requests, caused that the CSR fields sent to the issuing server lost the proper encoding when special characters appeared. This situation had not been tested when the change was introduced.

  Although postlinting tools were working at that time (Pkimetal and zlint), zlint did not detect it but pkimetal did and an email was sent warning about this. Although the mail was received by the issuing team, it was ignored and did not notify this issue to proceed with the revocation.

• Remediation description:

  Our main goal has been to implement pkimetal in the preissuance to avoid any missisuance. In the meantime, we warned the issuing team by email that any email coming from pkimetal tool had to be taken into account and notified. We will also introduce new tests when a new version is deployed to avoid failures like this one.

• Commitment summary:

  We have to pay more attention whenever a change in the code is done, no matter the impact it is supposed to have. We should not underestimate it and perform enough tests because everything can not relied upon pkimetal as it does not cover all the possible issues.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 15

[Ben Wilson]

I'll close this next Friday, 18-Apr-2025, unless there are questions or issues to discuss further.