1747915 - Sectigo: Incorrect JOI Country value

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

1. How your CA first became aware of the problem

We received an email reporting a certificate with a seemingly incorrect subject:jurisdictionCountryName value.

2. Timeline

February 8, 2021 – 10:47 UTC and 11:05 UTC
Sectigo receives two orders for company name “DirectLease B.V.”, one for their .nl domain and one for their .de domain. Within the next hour we perform a verification phone call, on which the Certificate Approver states that the order for the .de domain should use different company information. We update the order and conduct a new verification.
However, in a then unknown software error, our automated system fails to update the subject:jurisdictionCountryName value originally assigned with the first set of order information.

December 17 – 16:54 UTC
A Sectigo compliance officer receives an email from an external party reporting a certificate possibly containing an incorrect country code in the subject:jurisdictionCountryName field.

December 17 – 17:14 UTC
The email is forwarded to the WebPKI Incident Response (WIR) team. Sectigo starts an internal investigation.

December 17 – 17:58 UTC
We identify the certificate and deem it misissued. We learn that the certificate was validated through the legacy validation system from Xolphin, which Sectigo acquired in 2020. A revocation event is planned and scheduled for December 21 at 16:00 UTC.

December 17 – 18:22 UTC
The customer impacted by this is informed of the scheduled revocation

December 17 – 19:16 UTC
We create an SSL Abuse ticket based on the original email. An initial response is sent to the original reporter confirming the incorrect JOI information in the certificate and informing them of the scheduled revocation date.

December 17 – 19:30 UTC
We run an initial scan for additional certificates affected by this on the Xolphin system. None are detected.

December 20 – 12:47 UTC
We create a development ticket for the Xolphin validation system to add a critical warning for requests where the subject:jurisdictionCountryNamefield value does not match the subject:countryName field value.

We also create an additional ticket for the validation system to update the subject:jurisdictionCountryName field, when a different QGIS is selected to be used for issuance.

This has the effect that in the future if a pending certificate request has a mismatch between the two fields on an EV certificate order, it can only be issued with the explicit approval of two senior validation staff members. This warning also shows the exact reason why explicit approval is required, making them more alert of the case.

December 20 – 18:00 UTC
Development and QA is completed on both tickets and the production systems are updated.

December 20 – 19:30 UTC
We perform a final scan of the Xolphin system to detect any additional certificates that may have been affected by the same root cause. We find none.

December 21 – 16:37 UTC
The certificate is revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

A patch has been applied to the Xolphin system resolving the issue.

4. Summary of the problematic certificates

1 certificate issued on February 8, 2021

5. Affected certificates

Serial Number	Certificate	Precertificate
33A64BDC57F66F15D6341775C9E8EB65	Certificate	Precertificate

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now

This order was validated by Sectigo’s Dutch team that formerly was part of Xolphin. Xolphin was a Sectigo RA prior to its acquisition by Sectigo in 2020.

Our Dutch validation office continues to use the legacy Xolphin systems for validating SSL certificates. We have an integration project underway to move these operations to our main Sectigo validation systems, and until that is complete, the previous RA systems continue to see use. Before the merger, Xolphin was an active RA for over 10 years with an excellent track record. During that time the company developed its own validation systems catering mainly to European customers, enabled with a great deal of automation. This automation however, is one root cause contributor to this misissuance.

It is worth noting that all of the pre-issuance checks that Sectigo has implemented over the past year as part of the Guard Rails project, are also in use for the certificates validated through the Xolphin validation system. Our recently “QGIS matching” release would have prevented this error. The certificate in question was issued in February of 2021, prior to the abovementioned release.

On February 8, 2021, Xolphin received two orders for company name “DirectLease B.V.”, one for their .nl domain, and one for their .de domain at 10:47 UTC and 11:05 UTC respectively.

At 11:21 UTC the QGIS source for both orders was found by the automated systems and verified by a validation staff member. At this point, the Xolphin systems automatically set the correct subject:jurisdictionCountryName value based on the QGIS in use. To avoid human error, the Validation Specialist never enters the JOI details manually into the Xolphin system.

As both orders were for the same organization, the verification step involving a phone call was combined for both orders. During the call, the Certificate Approver stated that the order for the .de domain should not use the same company information and provided us with the correct information. Due to this change in details, staff restarted the validation process for that specific order.

Starting at 11:33 UTC, validation staff updated the .de order and started a new validation, which included, due to the country switching from NL to DE, a different QGIS source.

Unbeknownst to the validation staff members however, this did not update the subject:jurisdictionCountryName value, making its way into the certificate. This is the previously unknown software bug mentioned above.

We performed a scan of the Xolphin system for any additional certificates that may have been affected by the same root cause and found none.

The act of changing the country of an EV certificate based on new information from the validation call is exceedingly rare, as attested to by the fact that we could discover no other affected certificates. Because this circumstance is very rare and only affects our legacy Xolphin systems, it went undetected before now.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future

Since the reporting and discovery of this bug, we have implemented changes to the Xolphin validation system requiring additional checks if there ever is a discrepancy between the values in the subject:jurisdictionCountryName and subject:countryName fields.

While it is uncommon, there are some cases where the two fields must have different values. Therefore, we cannot completely block issuance of this sort.

The Xolphin automated systems now update the JOI fields based on the selected QGIS source, even after one has initially been approved.

Comment 1

[Martijn Katerbarg]

Our initial report concludes our investigation and remediation into this incident. We are monitoring this bug for any questions and/or comments.

Comment 2

[Martijn Katerbarg]

It appears that there are no comments. As such we’d like to request closing this bug.

Comment 3

[Ben Wilson]

I'll close this on Friday, 14-Jan-2022.