1891245 - Sectigo: EV Certificate issuance with incorrect subject:serialNumber attribute value

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Preliminary Incident Report

Summary

On April 11, 2024 at 18:57 UTC, we received a Certificate Problem Report (CPR) for a single TLS certificate, inquiring about the validity of the subject:serialNumber attribute value.

We have since determined the certificate to have been misissued. Revocation of the affected certificate is scheduled for April 16, 2024 around 18:00 UTC.

Both the Subscriber and the entity who filed the CPR have been notified of this.

We are currently investigating the cause of this incident as well as the impact, determining if more certificates related to the same Subscriber are affected.

We expect to post a full incident report by Friday April 19, 2024.

Comment 1

[Martijn Katerbarg]

Attachment: 1891245 - Affected Certificated.csv

Comment 2

[Martijn Katerbarg]

Incident Report

Summary

We received a Certificate Problem Report (CPR) for a single TLS certificate, inquiring about the validity of the subject:serialNumber attribute value.

After investigating the validation performed on the organization, we come to the conclusion that, while the legal entity and organization details are valid, an incorrect subject:serialNumber attribute value has been validated and included in the certificate.

The subject:serialNumber attribute value included in the certificate is the registration number of a legal entity that is no longer incorporated but that shares the same name as the legal entity the certificate has been issued to.

The included value of “09003091”, should have been “30046259”. While it doesn’t do away with the mis-issuance, it may be worth mentioning that the Incorporating Agency shows that the entity with registration number 09003091 was de-listed by being incorporated within 30046259 instead, showing a legal relation between these two entities.

Impact

166 Unexpired certificates issued between 2023-04-24 and 2024-01-11.

Out of these 166 certificates, 33 have already been revoked by the Subscriber prior to the discovery of this incident.

Timeline

All times are UTC.

2022-11-16:

• 08:33:44 We receive a request to issue a certificate, create a pre-validation record and verify the Applicant’s organization.

2022-11-17:

• 18:23 We start the verification process for the organization.
• 18:27 We add a Registration QGIS document showing the name of the entity.
• 18:40 We confirm the details listed on the Registration QGIS document and verify these against the request details. The validation agent working on the order does not spot that the entity has been de-listed. This source belongs to registration number 09003091.
• 18:41 We add a second document and use it to verify Physical and Operational Existence, and address details. This source belongs to registration number 30046259.
• 18:51 Further validation steps are completed and the order is sent to our Second Approval team.
• 20:38 We complete the Second Approval process and issue the certificate.
• 20:38 The certificate is marked in our system as a pre-validation record. This means subsequent orders for the same account and same legal entity may rely on this order for document re-use (not including Domain Control) within the BR and EVG allowed re-use period.

2023-12-18:

• 09:03:51 We receive a new request to issue a certificate, create a pre-validation record and verify the Applicant’s organization.
• 15:46 We start the verification process of the organization.
• 15:49 We add a Registration QGIS document showing the name of the entity.
• 15:54 We confirm the details listed on the Registration QGIS document and verify these again the request details. While the source used is the same as the order the year before, the current document does not show whether or not the entity is de-listed. This source belongs to registration number 09003091.
• 16:00 We add a second document and use it to verify Physical and Operational Existence and address details. This source belongs to registration number 30046259.
• 16:24 Further validation steps are completed and the order is sent to our Second Approval team.
• 17:13 We complete the Second Approval process and issue the certificate.
• 17:13 The certificate is marked in our system as a pre-validation record (again not including Domain Control).

2024-01-16:

• 11:51:30 Our customer places a new request to issue a certificate, create a pre-validation record and verify the Applicant’s organization. While the existing record is still valid, our system allows for the creation of new records upon customer request. We are not aware of the reason this customer has for placing a new request.
• 12:37 We start the verification process of the organization.
• 12:40 We add a Registration QGIS document showing confirming the name of the entity, confirm the details listed on the Registration QGIS document and verify these again the request details. This time, the source document used belongs to registration number 30046259.
• 12:46 Further validation steps are completed and the order is sent to our Second Approval team.
• 13:26 We complete the Second Approval process and issue the certificate.
• 13:26 The certificate is marked in our system as a pre-validation record (again not including Domain Control).
• 14:25:14 The Subscriber requests revocation of the certificate issued on December 18, 2023. Revocation is completed automatically.

2024-04-11:

• 17:57 We receive a Certificate Problem Report (CPR) for a single TLS certificate.

2024-04-12:

• 09:39 We respond that we will initiate an investigation into the certificate.
• 12:02 As some of the members in our Compliance team have seen the CPR, we decide to request a report from our DBA team for all certificates issued sharing the same subject:organizationName attribute value.
• 13:23 The CPR is escalated to our Compliance team. The reporter is notified of this escalation.
• 14:00 Our twice-weekly WebPKI Incident Response (WIR) team’s scheduled call starts. We start the call by discussing the CPR received. During the call, we conclude that the certificate is indeed mis-issued.
• 14:34 We respond to the reporter that we have confirmed mis-issuance of the certificate and are launching further investigation. We notify the reporter that the certificate will be revoked on April 16, 2024 by 18:00 UTC.
• 15:33 We notify the Subscriber of the received CPR and our initial findings. We request for them to replace their certificate (which is for their main website) as soon as possible. They are notified about the scheduled revocation and subsequent continued investigation.
• 15:34 The DBA team receives the required SQL query details from our development team.
• 15:40 We proceed to call the Subscriber in order to expedite the message and express the urgency.
• 15:53 We open this incident report.

2024-04-13:

• 03:44 The query is run by our DBA team and the results are returned to the Compliance team.

2024-04-15:

• 07:30 We commence manual processing of the results.
• 08:05 We conclude our investigation of the results. Including the originally reported certificate, we find 166 unexpired certificates affected by this incident. Of those 166, 33 were previously revoked by the Subscriber.
• 08:13 We notify the Subscriber of the need to replace the 132 remaining unexpired and unrevoked certificates. Revocation is scheduled for April 19, 2024 at 21:00 UTC.
• 12:29 The Subscriber notifies us that they’ve started the replacement process and have already informed their internal teams and urged them to take action ASAP.

2024-04-16:

• 17:53 We revoke the initial reported certificate.

2024-04-19:

• 21:25 We revoke the remaining 132 certificates.

Root Cause Analysis

The search for QGIS and QIIS sources for this Applicant revealed a large enterprise with several registrations. The registration extract of the Subscriber’s correct Registration QGIS alone consists of 190 pages, including 94 trade names and 231 different locations within a single country. Both the valid and the de-listed Registration QGIS had overlapping name details.

While that itself should not result in a mis issuance event, our validation team did not notice that the selected report was marked as de-listed as all other documents on the order showed a valid registered company.

The document gathering for this case relied partially on automated collection of documents and partially on manual gathering. The documents that should not have been used, showing the incorrect registration number and the de-listed company, were collected manually.

This leads us to the conclusion that the Root Cause of this incident is an error in manual processing.
For the registration source used, we have recently integrated automated document collection, which we are encouraging our validation staff to use. It is our firm belief that such an automated collection would have prevented this incident from having occurred in the first place.

Lessons Learned

What went well

• We had clear and direct communication lines with this customer which helped escalate this matter quickly.

What didn't go well

• In the first of two validation cases, our validation staff did not spot that the registration document listing the registration number was marked as de-listed.
• In the second of two validation cases, the document containing the wrong registration number did not report whether or not the entity was de-listed.

Where we got lucky

• A new and correct pre-validation record was already issued in January, somewhat limiting the impact of this incident.

Action Items

Action Item	Kind	Due Date
Integrate the registration source within our validation system	Prevent	Completed
Continue with our automation objectives both for internal and external customers	Mitigate	Ongoing*

*We do not believe this Mitigation action item can have a Due Date set. Sectigo has an ongoing objective for moving manual tasks to automated tasks and improving these. We expect this will be an ongoing item for the foreseeable future.

Appendix

Details of affected certificates

The affected certificates are listed in attachment 9397673 [details].

Comment 3

[Martijn Katerbarg]

Our incident report completed our action related to this incident.

We are monitoring this bug for any comments and/or questions.

Comment 4

[amir]

Can you share how you accomplished the automation in this space? I believe this can be helpful for other CAs as well.

Comment 5

[Martijn Katerbarg]

(In reply to amir from comment #4)

Can you share how you accomplished the automation in this space? I believe this can be helpful for other CAs as well.

In this particular case, we utilize the APIs provided by the Dutch Chamber of Commerce.

The automated search results are mapped against fields that need to be verified by our vetting agents, showing a clear overview of what needs to be verified. The vetting agent then verifies if the mapped field values match the to-be-verified data.

We take a similar approach with other data sources. Not every data-source offers an API however, limiting our options.

Comment 6

[Martijn Katerbarg]

Ben, it looks like all comments have been addressed. Unless there are comments in the next few days, we would like to request this bug to be marked as resolved.

Comment 7

[Ben Wilson]

I'll close this on or about Monday, 13-May-2024.