1915883 - Sectigo: Missing data in cabfOrganizationIdentifier

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Preliminary Incident Report

Summary

On August 28, 2024, I received a private message through social media recommending we investigate 19 certificates.

While the (vague) description of the issue did not lead to the discovery of miss issued certificates, additional research on August 29, 2024 did find that 4 out of the 19 certificates did show an issue with the cabfOrganizationIdentifier extension.

Further internal research discovered one additional certificate, accounting for a total of 5 miss issued certificates.

A patch is currently being developed and is scheduled to be deployed this coming weekend. Revocation of these certificates is scheduled for September 3, 2024.

A full incident report will be posted no later than September 6, 2024.

Comment 1

[Martijn Katerbarg]

Incident Report

Summary

On August 28, 2024, I received a private message through social media recommending we investigate 19 certificates.

While the (vague) description of the issue did not lead to the discovery of mississued certificates, additional research did find that 4 out of the 19 certificates showed an issue with the cabfOrganizationIdentifier extension.

Further internal research discovered one additional certificate, accounting for a total of 5 mississued certificates.

Impact

5 Certificates issued between 2023-12-28 and 2024-08-13.

Timeline

All times are UTC.

2024-08-28:

• 14:33 I receive a personal message containing 19 certificate serial numbers and a vague note regarding the serial number field.
• 14:42 The message is acknowledged.
• 15:00 We start an investigation, reviewing included data in the subject:serialNumber attribute field.

2024-08-29:

• 10:00 We complete our review between the compliance and validation departments. We find no issues with the subject:serialNumber field in the provided certificates.
• 13:00 After internal discussion, we decide to make sure nothing else could be wrong with these certificates and perform a complete review.
• 13:30 We find that the cabfOrganizationIdentifier extension is missing part of the Registration Reference when a dash is included within the Registration Reference part of the subject:organizationIdentifier attribute value.
• 13:40 We schedule a revocation event for the discovered certificates and start customer outreach.
• 13:45 We schedule an internal call with development stakeholders for 15:30
• 13:52 We halt issuance of all QWAC-based certificates until further notice.
• 15:30 We hold an internal call with development stakeholders in order to determine what is causing the issue.
• 15:45 The issue is found to be related to how the organizationIdentifier is parsed and stripped into different pieces for inclusion in the cabfOrganizationIdentifier field.
• 16:00 All the required changes are scoped. Our development team sets out to building a patch for the issuance system. Our QA department is briefed and asked to make room for testing on an urgent patch.
• 19:08 The patch is released to our QA environment.

2024-08-30:

• 13:59 After requesting a further change, QA signs off.

2024-08-31:

• 23:00 We have a planned outage for a deployment to our issuance system. We deploy the patch during this window.

2024-09-01:

• 11:13 We resume QWAC-based certificate issuance, allowing for replacement certificates to be issued.

2024-09-03:

• 13:16 The affected certificates are revoked.

Root Cause Analysis

Back in 2020, we first implemented support for QWAC certificates within our issuance system. As part of this, code was added to parse the Organization Identifier and strip this into several pieces, such as the Registration Scheme and the Registration Reference. As part of this, a match on the hyphen-minus character (-) was implemented.

As was now revealed, the registrationReference field within the cabfOrganizationIdentifier, was only set to include characters after the last occurrence of the hyphen-minus character. While we have gone back to review the requirements of the original implementation, it remains unclear if this approach was a deliberate development choice, or rather a bug in the implementation itself.

The implemented approach, be it by choice or unintended bug, did not consider the possibility of the registration reference itself being able to contain one or more hyphen-minus characters, causing the miss issuance of certificates.

In development, a test case with the hyphen-minus character within the Registration Reference part of the Organization Identifier was never part of QA testing, which is why this was not spotted in the past.

Lessons Learned

What went well

• We were able to patch our systems within relative short time and provide replacement certificates to affected Subscribers.
• We were able to halt issuance during our investigation and until our patch was released.

What didn't go well

• Our investigation into mississued QWAC certificates in May of 2024, while focussed on a different issue, did not reveal the semi-related issue outlined here.
• Design/Implementation and QA did not consider that characters used to split strings might also be used within a valid value.

Where we got lucky

• Due to the small number of QWAC certificates with a hyphen-minus character in the Registration Reference, the number of mississued certificates was limited.

Action Items

Action Item	Kind	Due Date
Update our code for how the organization identifier is parsed prior to inclusion into the cabfOrganizationIdentifier extension.	Prevent	Completed
During the design phase of changes and implementations related to included Subscriber data, Validation Specialists and/or Validation Management will be included to assist in identifying potential issues. This has already been implemented, but was not the case in 2020.	Mitigate	Completed
Update incident investigation procedures, cast a wider net to try and identify semi-related issues in our issuance base	Detect	Completed

Appendix

Details of affected certificates

Serial Number	Certificate	Precertificate
00D0137C9458423653DA4AB13586AC330F	Certificate	Precertificate
00FD8C40BEC675B6047D9E97CD0186D4EB	Certificate	Precertificate
00F5C93EC1E66F1AC8AE389E859A4079FD	Certificate	Precertificate
3D111BBD105EA409D31B3A54E26A1CDA	Certificate	Precertificate
5EBB1205B2F4B450FF3BC77E919F338E	Certificate	Precertificate

Comment 2

[Martijn Katerbarg]

Comment 1 completes our incident response with all action items having been completed. We are monitoring this bug for any questions and/or comments.

Comment 3

[Martijn Katerbarg]

Ben, our incident response on this bug has been completed. We’d like to request closing this bug.

Comment 4

[Ben Wilson]

I'll close this on or about next Wed., 25-Sept-2024, unless there are additional questions or comments.