1902748 - Sectigo: QWAC certificates issued with incorrect subject:organizationIdentifier attribute value

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Preliminary Incident Report

Summary

In comment #19 of bug 1897538, a suspected misissued certificate was reported. Our initial investigation shows that this certificate was indeed misissued.

The root cause of this misissuance is not related to items outlined in bug 1897538. As such, this bug will track our incident report and root cause pertaining to the misissued certificate.

We will post a full incident report no later than June 27th, 2024.

Comment 1

[Martijn Katerbarg]

Incident Report

Summary

In comment #19 of bug 1897538, a suspected misissued certificate was reported. Our initial investigation shows that this certificate was indeed misissued.

Our investigation found another 2 misissued certificates sharing the same root cause.

By allowing our customers to enter the organization identifier during the order process, errors were introduced in the submitted certificate requests. During the validation these errors were not rectified, causing the inclusion of incorrect subject:organizationIdentifier attribute values.

Impact

3 certificates issued between 2024-04-23 and 2024-06-13.

Timeline

All times are UTC.

2024-06-13:

• 17:00 Comment 19 on bug 1897548 is placed, informing us of a potential misissued certificate.

2024-06-14:

• 08:00 We start reviewing the certificate mentioned in comment 19 of bug 1897548.
• 08:59 We confirm the certificate is misissued. We start a revocation event. Revocation is scheduled for June 18, 2024.
• 14:30 We request a database export of all issued QWAC certificates to review if more certificates are affected.
• 16:17 We acknowledge Comment 19 on bug 1897548.
• 23:07 Our DBA team delivers the database export.

2024-06-17:

• 10:00 We start manually reviewing the orders in the database export.
• 10:10 Based on review of the first certificate, we identify that a customer-provided organization identifier was not corrected during the validation phase. We reach out to our retail development team and request to remove the organization identifier field from the order process.
• 11:57 We start internal discussions with the validation team to discuss improvements and further automation within our validation system around the organization identifier

2024-06-18:

• 15:25 The retail team starts work to remove the organization identifier field from the order process.
• 15:26 The first reported certificate is revoked within 5 days of being reported.

2024-06-19:

• 10:04 We complete a proposal for improvements in the validation system around the organization identifier and submit this to the validation and development teams.
• 11:19 We complete peer-reviewing potential misissued certificates and confirm 2 additional certificates. A revocation event is scheduled for June 24th, 2024 at 11:00 UTC.

2024-06-24:

• 11:14 We revoke the two additional certificates.

2024-06-25:

• 18:00 The changes to our retail order process are deployed.

Root Cause Analysis

Up until this week, we have allowed customers on our retail channel to provide the organization identifier during the order process. This data has subsequently been added into the order for validation purposes.

While validation agents are trained to review and verify the entered organization identifier, this incident has taught us that allowing customers to enter this during the order process more easily leads to mistakes made during the validation process, as opposed to letting the validation agent set the organization identifier themselves based on available data.

Due to just minor differences, validation agents have approved a supplied organization identifier. The absence of any real automation or guard rails pertaining to the organization identifier has led to these mistakes having gone unnoticed prior to certificate issuance.

Lessons Learned

What went well

• Our existing data-source by country setup allows for automation tasks to be added in relatively short term.

What didn't go well

• We did not yet have automation in place for assisting in the validation of organization identifiers.
• We allowed customers to enter data that, while it was being validated by us, led to mistakes being made more easily.

Where we got lucky

• Only a small number of certificates was affected.
• Because the number of certificates affected is very low, our internal audit did not catch this error. We are lucky that a community member noticed the error and called it to our attention so that we could investigate and put a programmatic solution in place.

Action Items

Action Item	Kind	Due Date
Remove the option for customers to enter organization identifiers during the order phase in our retail channel.	Prevent	Completed
Enhance our validation system by setting per-product and per-datasource formatting rules (using Regular Expressions) to specify which type and format of organization identifier is allowed with the specified source.	Prevent	2024-08-31

Appendix

Details of affected certificates

Serial Number	Certificate	Precertificate
009844DA27F0C5DF7F03F00328B6B31482	Certificate	Precertificate
00C6672D2330C2773B3635B8660835B48E	Certificate	Precertificate
00B74F2E0DF5E963065B63DA343A77E796	Certificate	Precertificate

Comment 2

[Tim Callan]

Ben, may we have a Next Update on 2024-08-31 to match our remaining Action Item?

Comment 3

[Martijn Katerbarg]

Continued investigation of this incident and our certificate base has revealed one additional certificate affected by this issue.

This error is dissimilar enough from the earlier one that our initial investigation missed it. As we continued management of this bug, we also have continued scrutinizing our certificate base for similar errors, and we found this one. In this certificate we discovered a mismatch within the organization identifier. It is a PSD2 based QWAC with valid NCA identifier, but the subsequent Registration Reference is that which the QGIS assigned to the Legal Entity, not the Registration Reference which the NCA assigned to the Legal Entity.

The certificate falls within the impacted timeline as specified in comment #1. We have scheduled revocation for 2024-07-21 around 16:00 UTC.

Serial Number	Certificate	Precertificate
00909DEF8889D6220A108F1E322E896CCB	Certificate	Precertificate

Comment 4

[Martijn Katerbarg]

On Saturday, August 24, 2024, we deployed the changes required to complete our final action item in this bug. With that, there are no further actions pending.

Ben, since there have not been any questions or comments, we would like to request closing this bug.

Comment 5

[Ben Wilson]

I will close this on or about Wed. 28-Aug-2024 unless there are additional comments, concerns, or questions.
Thanks,
Ben