1943604 - HARICA: TLS Server certificate issuance without proper validation

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Dimitris Zacharopoulos]

Steps to reproduce:

Preliminary Incident Report

Summary

On 2025-01-23, HARICA was informed by a Subscriber that a TLS Server certificate was issued without proper domain control validation. The Applicant accidentally typed an incorrect Domain Name containing part of the pre-validated Domain Name which caused an incorrect string matching, leading to the mis-issuance. After investigation, HARICA discovered that under certain conditions, a flaw in the validation reuse code for pre-validated Organizations, permitted unvalidated domains to reuse DCV evidence if the requested FQDN contained one of the Organization’s pre-validated domains names.

TLS Server certificate issuance was suspended until a fix was deployed later the same day. A detailed timeline will be provided in the final incident report.

The affected certificate was revoked within 24 hours and the subscriber was informed.

A full scan of logs and the certificate database revealed that no other certificate was affected by this bug.

A full incident report will be posted no later than 2025-01-31.

Comment 1

[Dimitris Zacharopoulos]

Incident Report (not final)

Summary

On 2025-01-23, HARICA was informed by a Subscriber that a TLS Server certificate was issued without proper domain control validation. The Applicant did a typo during a certificate request which contained part of their Organization’s pre-validated Domain Name which caused an incorrect string matching, leading to a mis-issuance. After investigation, HARICA discovered that under certain conditions, a flaw in the validation reuse code for pre-validated Organizations, permitted unvalidated domains to reuse DCV evidence if the requested FQDN contained one of the Organization’s pre-validated domains names.

Impact

A single TLS Server certificate was issued without proving Domain Control Validation (DCV) due to a problematic DCV re-use code.

TLS Server certificate issuance was suspended until a fix was deployed later the same day.

Timeline

All timestamps in EET (GMT+2)

• 2021-02-12 16:11 The problematic code was merged to development branch
• 2021-02-12 – 2021-02-16 The code was tested with a set of FQDNs belonging to the list of pre-validated domain names and a set of FQDNs that did not belong to the pre-validated domain names.
• 2021-02-16 17:40 After successful testing, the problematic code was merged to the production branch
• 2025-01-23 12:34 HARICA received an email from a Subscriber in our standard support email queue informing us that a certificate without proper domain validation has been issued.
• 2025-01-23 13:27 Reproduced the bug and stopped the certificate issuance of TLS Certificates
• 2025-01-23 22:22 A patch was deployed that fixed the bug
• 2025-01-23 22:58 TLS certificate issuance resumed
• 2025-01-24 09:00 Revoked the affected certificate and informed the Subscriber to request a replacement certificate
• 2025-01-24 16:52 Completed a scan of all non-expired certificates to identify additional affected certificates, without any more findings.

Root Cause Analysis

Here we present the results of the “5 whys” root cause methodology that was followed:

1. Why HARICA mis-issue a TLS Server certificate?

Because control of the FQDN was not proven before issuance.

2. Why was control of the FQDN not proven before issuance?

Because the DCV re-use code considered it as validated.

3. Why did the DCV re-use code consider it as validated?

Because there was a bug in the domain label identification and matching part of the code.

4. Why was there a bug in the domain label identification and matching part of the code?

Because the code didn’t properly handle the boundaries of a pre-validated domain.

5. Why did the code not properly handle the boundaries of a pre-validated domain?

Because there were no test cases to check the behavior when the pre-validated domain appeared in the middle of the FQDN. The tests should have included FQDN cases that included all the pre-validated Domain Name labels in other parts of the FQDN. For example, if “example.com” was the pre-validated Domain Name, the tests should have included “myexample.com”, “example.com.some-other-domain.net”, etc.

While each incident focuses on a specific issue, HARICA has identified similar root causes in recent incidents:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1943596 and
• https://bugzilla.mozilla.org/show_bug.cgi?id=1942130.

The investigation will continue in order to evaluate these incidents in a combined manner in search of a potential common root cause.

Lessons Learned

What went well

• HARICA responded quickly and fixed the bug

What didn't go well

• The bug was introduced four years ago
• QA didn’t catch the bug

Where we got lucky

• The bug affected only one TLS Server certificate
• The bug could be triggered only for pre-validated Organizations that were mapped to pre-validated domains
• The bug workflow was very specific and DCV could not be bypassed for any FQDN

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Implement Unit test for TLS Server certificates that checks the behavior when the pre-validated domain appears in the middle of the FQDN	Mitigate	2025-03-07
For pre-validated Organizations create code in a central location to prevent issuance of TLS Server Certificates that contain Domain Names not ending with all labels of the pre-validated Base Domain Names of Organizations	Mitigate	2025-04-08

Appendix

Details of affected certificates

Serial, SHA-256 fingerprint

• 0CB7F20B5C691AD1D3BB87ABFCFD6C05, https://crt.sh/?sha256=8622BCADDB4C8EC5F80090A82ED3FB78F5C2A61F87E4699C1B3BC05A485A7CC5

Based on Incident Reporting Template v. 2.0

Comment 2

[Dimitris Zacharopoulos]

Based on comment 1 we continued the investigation which revealed additional causes contributed to this and the other referenced incidents.

One additional cause was identified which is the code review process that lacks the depth and intimate knowledge required to properly check and verify the resulting code. Despite the change management procedures and feature reviews/testing by the QA team in place, we believe that some -if not all- of these cases would have been prevented if the code review were performed by a senior level software engineer/developer.

To prevent this, we plan to flag a set of functions in the RA code, associated with:

• validation (domain, emails, identity)
• authentication – authorization – accounting
• certificate request, issuance, revocation

and require code reviews to be performed exclusively by a senior software engineer/developer.

We are submitting the final incident report below.

Incident Report

Summary

On 2025-01-23, HARICA was informed by a Subscriber that a TLS Server certificate was issued without proper domain control validation. The Applicant did a typo during a certificate request which contained part of their Organization’s pre-validated Domain Name which caused an incorrect string matching, leading to a mis-issuance. After investigation, HARICA discovered that under certain conditions, a flaw in the validation reuse code for pre-validated Organizations, permitted unvalidated domains to reuse DCV evidence if the requested FQDN contained one of the Organization’s pre-validated domains names.

Impact

A single TLS Server certificate was issued without proving Domain Control Validation (DCV) due to a problematic DCV re-use code.

TLS Server certificate issuance was suspended until a fix was deployed later the same day.

Timeline

All timestamps in EET (GMT+2)

• 2021-02-12 16:11 The problematic code was merged to development branch
• 2021-02-12 – 2021-02-16 The code was tested with a set of FQDNs belonging to the list of pre-validated domain names and a set of FQDNs that did not belong to the pre-validated domain names.
• 2021-02-16 17:40 After successful testing, the problematic code was merged to the production branch
• 2025-01-23 12:34 HARICA received an email from a Subscriber in our standard support email queue informing us that a certificate without proper domain validation has been issued.
• 2025-01-23 13:27 Reproduced the bug and stopped the certificate issuance of TLS Certificates
• 2025-01-23 22:22 A patch was deployed that fixed the bug
• 2025-01-23 22:58 TLS certificate issuance resumed
• 2025-01-24 09:00 Revoked the affected certificate and informed the Subscriber to request a replacement certificate
• 2025-01-24 16:52 Completed a scan of all non-expired certificates to identify additional affected certificates, without any more findings.
• 2025-01-31 Incident report (not final) was submitted to Bugzilla
• 2025-02-03 – 2025-02-06 Investigation continued in search of possible common causes to the recent incidents (https://bugzilla.mozilla.org/show_bug.cgi?id=1943596, https://bugzilla.mozilla.org/show_bug.cgi?id=1942130 )
• 2025-02-07 Incident report (final) was submitted to Bugzilla

Root Cause Analysis

Here we present the results of the “5 whys” root cause methodology that was followed:

1. Why HARICA mis-issue a TLS Server certificate?

Because control of the FQDN was not proven before issuance.

2. Why was control of the FQDN not proven before issuance?

Because the DCV re-use code considered it as validated.

3. Why did the DCV re-use code consider it as validated?

Because there was a bug in the domain label identification and matching part of the code.

4. Why was there a bug in the domain label identification and matching part of the code?

Because the code didn’t properly handle the boundaries of a pre-validated domain.

5. Why did the code not properly handle the boundaries of a pre-validated domain?

Because there were no test cases to check the behavior when the pre-validated domain appeared in the middle of the FQDN. The tests should have included FQDN cases that included all the pre-validated Domain Name labels in other parts of the FQDN. For example, if “example.com” was the pre-validated Domain Name, the tests should have included “myexample.com”, “example.com.some-other-domain.net”, etc.

In addition, we identified a common factor applicable to this and the following two recent HARICA incidents:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1943596 and
• https://bugzilla.mozilla.org/show_bug.cgi?id=1942130

According to the current policy, the code review process may be performed by any software engineer/developer. In practice, HARICA has been using experienced software engineers/developers to review code but in some cases there was lack of depth and intimate knowledge of the entire RA system required to properly check and verify the resulting code in critical functions of the RA.

Lessons Learned

What went well

• HARICA responded quickly and fixed the bug

• HARICA reviewed recent incidents both independently and in a combined manner with other incidents

What didn't go well

• The bug was introduced four years ago
• QA didn’t catch the bug
• Code review didn’t catch the bug

Where we got lucky

• The bug affected only one TLS Server certificate
• The bug could be triggered only for pre-validated Organizations that were mapped to pre-validated domains
• The bug workflow was very specific and DCV could not be bypassed for any FQDN

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Implement Unit test for TLS Server certificates that checks the behavior when the pre-validated domain appears in the middle of the FQDN	Mitigate	2025-03-07
For pre-validated Organizations create code in a central location to prevent issuance of TLS Server Certificates that contain Domain Names not ending with all labels of the pre-validated Base Domain Names of Organizations	Mitigate	2025-04-08
Flag a set of critical functions in the RA code and require code reviews to be performed exclusively by a senior software engineer/developer	Mitigate	2025-05-07

Appendix

Details of affected certificates

Serial, SHA-256 fingerprint

• 0CB7F20B5C691AD1D3BB87ABFCFD6C05, https://crt.sh/?sha256=8622BCADDB4C8EC5F80090A82ED3FB78F5C2A61F87E4699C1B3BC05A485A7CC5

Based on Incident Reporting Template v. 2.0

Comment 3

[Dimitris Zacharopoulos]

Regarding the action item,

“Implement Unit test for TLS Server certificates that checks the behavior when the pre-validated domain appears in the middle of the FQDN,”

the team has created a list of positive and negative test cases that check for the correct string matching behavior on re-using completed and unexpired DCVs of pre-validated domain names.

We are on track with all action items.

Comment 4

[Dimitris Zacharopoulos]

Here is the update on our action items.

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Implement Unit test for TLS Server certificates that checks the behavior when the pre-validated domain appears in the middle of the FQDN	Mitigate	Completed 2025-02-20
For pre-validated Organizations create code in a central location to prevent issuance of TLS Server Certificates that contain Domain Names not ending with all labels of the pre-validated Base Domain Names of Organizations	Mitigate	2025-04-08
Flag a set of critical functions in the RA code and require code reviews to be performed exclusively by a senior software engineer/developer	Mitigate	2025-05-07

If there are no further questions or concerns with the incident report, we would like to kindly ask the Next Update to be set for 2025-03-21 so we can provide an update on the progress. If there is meaningful progress before that date, we will post an update sooner.

Comment 5

[Dimitris Zacharopoulos]

Here is the update on our action items.

We are on track to deliver the remaining action items as scheduled. We would like to set the next update to 2025-04-08.

Comment 6

[Jeremy]

How was the scan done to ensure no other certificates were impacted? What did you look for if a typo was the cause of the primary issue?

Comment 7

[Dimitris Zacharopoulos]

Thank you for the request.

As you can see in the affected certificate https://crt.sh/?sha256=8622BCADDB4C8EC5F80090A82ED3FB78F5C2A61F87E4699C1B3BC05A485A7CC5, the "typo" was in the FQDN. Instead of requesting "pre.vn-fh.potsdam.de", the Applicant requested "pve.vn-fh-potsdam.de" which triggered the buggy string matching process. HARICA had pre-validated the Domain Namespace "potsdam.de" which allowed the Applicant to request anything ending with "potsdam.de" but the code wasn't careful to check the "." before the Base Domain Name when subdomains were requested.

We started querying all the non-expired, non-revoked certificates linked to pre-validated TLS orders that contained 2 or more SAN dNSName entries.

Then, for each certificate, we extracted all dNSName values and excluded those that ended with all the labels of the pre-validated Base Domain Names of the corresponding Enterprise, careful to include the "." for any subdomains to the Base Domain Names.

Only one certificate came out of that query which was the impacted one.

We hope this answers the question, but if not, please let us know if we can add more details.

Comment 8

[Dimitris Zacharopoulos]

Ben, as mentioned in comment 5, if there are no other questions, we kindly ask setting the next update to 2025-04-08.

Comment 9

[Dimitris Zacharopoulos]

Here is a status update for the pending actions.

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Implement Unit test for TLS Server certificates that checks the behavior when the pre-validated domain appears in the middle of the FQDN	Mitigate	Completed 2025-02-20
For pre-validated Organizations create code in a central location to prevent issuance of TLS Server Certificates that contain Domain Names not ending with all labels of the pre-validated Base Domain Names of Organizations	Mitigate	Completed 2025-03-21
Flag a set of critical functions in the RA code and require code reviews to be performed exclusively by a senior software engineer/developer	Mitigate	2025-05-07

One action item remains to be completed. We kindly request setting the next-update flag to 2024-04-25. We will update the bug earlier if the last action is completed sooner.

Comment 10

[Dimitris Zacharopoulos]

We continue to work on the last open action item and expect to complete it as planned. Ben, can you please schedule the Next update to 2025-05-09?

Comment 11

[HARICA]

We would like to inform the community that HARICA will be using the account "public-incident-reports [at] harica [dot] gr" for managing HARICA-related incidents.

Here is a status update for the action items.

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Implement Unit test for TLS Server certificates that checks the behavior when the pre-validated domain appears in the middle of the FQDN	Mitigate	Completed 2025-02-20
For pre-validated Organizations create code in a central location to prevent issuance of TLS Server Certificates that contain Domain Names not ending with all labels of the pre-validated Base Domain Names of Organizations	Mitigate	Completed 2025-03-21
Flag a set of critical functions in the RA code and require code reviews to be performed exclusively by a senior software engineer/developer	Mitigate	Completed 2025-05-06

All action items have been completed. Please let us know if there are any further questions or concerns.

Dear Ben, with the recent changes in incident handling, is it reasonable to wait at least a week for possible additional questions or concerns before submitting a closure statement, or should we proceed with that sooner?

Comment 12

[HARICA]

Incident Closure Summary

• Incident description:
  On January 23, 2025, HARICA discovered that a TLS certificate had been misissued due to improper reuse of domain control validation (DCV) evidence. This occurred because the system incorrectly treated domains as validated when their FQDNs included a previously validated domain as a substring. The affected certificate was revoked within 24 hours.

• Incident Root Cause(s):
  The misissuance resulted from a flaw in HARICA’s domain validation logic, where DCV evidence was reused based on substring matching. Specifically, if a requested domain name included a pre-validated domain name as a substring, the validation system mistakenly assumed the full domain was validated.

• Remediation description:
  Upon detection, HARICA immediately suspended issuance of TLS certificates and implemented a patch the same day to prevent reuse of DCV evidence based on substring logic. The affected certificate was revoked, and the impacted subscriber was notified. HARICA also conducted a retrospective audit to confirm that no other certificates had been issued under similar circumstances.

• Commitment summary:

  • Develop and maintain additional automated test cases targeting domain matching logic and DCV edge cases.
  • Improved Change Management review procedure of critical functions of the RA code (associated with Certificate Lifecycle Management, Domain Validation, Authentication/Authorization/Accounting) which requires a senior software developer/engineer in the review process.

All action items have been completed as described, and we respectfully request closure of this incident.

Comment 13

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-05-23.