1813989 - Sectigo: Incomplete Subject organizationName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

We are opening this bug to inform the community of an incident where 6 issued certificates were discovered and reported to us, with an incomplete organizationName value in the subject.

These certificates were reported to us on January 23rd at 21:32 UTC and were revoked on January 28th at 21:29 UTC.

Our own investigation revealed an additional certificate. Revocation of this certificate is scheduled for January 31, 20:30 UTC.

We are investigating the incident and will provide a complete incident report once we have completed the investigation.

Comment 1

[Martijn Katerbarg]

Further investigation leads us to believe this incident affects more certificates than we discovered during our initial investigation.

We are currently investigating a number of certificates to see the exact impact of this incident and will, if required, perform additional revocations and include these in our report.

We aim to complete our investigation early next week.

Comment 2

[Martijn Katerbarg]

Further investigation has revealed 253 certificates in which the Subject organizationName is believed to be incorrectly abbreviated. We completed this investigation today at 15:45 UTC. Notifications are being sent out to affected parties and we have scheduled revocation of these certificates for February 19th at 10:00 UTC.

We are working on our complete incident report.

Comment 3

[Martijn Katerbarg]

On February 19th at 10:21 UTC, we completed revocation of 235 certificates.

For a total of 18 certificates, we have been asked for an extension of the revocation deadline. We decided to grant this delayed revocation. These 18 certificates are scheduled to be revoked before February 22nd, 04:00 UTC.

As we realize this constitutes a separate incident, we have opened bug 1818073 for this particular incident, including all relevant details.

Comment 4

[Martijn Katerbarg]

Attachment: 278 Revoked Certificates.xlsx

Comment 5

[Martijn Katerbarg]

1. How your CA first became aware of the problem

We received a Certificate Problem Report on January 23rd at 21:32 UTC.

2. Timeline

January 23, 2023 – 21:32 UTC
We receive a Certificate Problem Report, which identifies 6 certificates issued in 2022 that appear to have incomplete subject:organizationName values.

January 24, 2023 – 17:00 UTC
We confirm the certificates contain subject:organizationName values that have been abbreviated incorrectly. We initiate our revocation process.

January 24, 2023 – 21:16 UTC
We send out notifications of the upcoming revocations and schedule revocations for January 28, 21:00 UTC.

January 26, 2023 – 22:30 UTC
Further internal investigation reveals and confirms an additional certificate following the same pattern. Revocation of this certificate is scheduled for January 31st 16:00 UTC.

January 28, 2023 - 21:29 UTC
We revoke the initial 6 reported certificates.

January 31, 2023 – 16:42 UTC
The additional certificate we discovered is revoked.

February 3, 2023 – 15:00 UTC
During our twice-weekly WebPKI Incident Response (WIR) team call, we re-evaluate the methodology used for discovering the affected certificates.

The original Certificate Problem Report refers to certificates that have an open parenthesis in the subject:organizationName, but lack a corresponding close parenthesis. In our initial investigation, we focused on this detail. We decide to widen our scope in this subsequent investigation.

February 14, 2023 – 15:45 UTC
We complete our widened investigation and determine a total of 253 additional certificates to be revoked, as mentioned in comment 2. These are scheduled for revocation on February 19th at 10:00 UTC. As we do not have the full remediation plan deployed, we keep track of any new certificate issuance.

February 17, 2023 – 15:00 UTC
During our twice-weekly WIR team call, we review requests from 3 affected customers to delay revocation. Based on the details at hand, we decide to grant the delay. This delayed revocation is outlined in bug 1818073.

February 19, 2023 – 10:21 UTC
A total of 235 additional certificates are revoked based on our continued investigation. Another 9 certificates, which were issued after February 14, are revoked at the same time.

February 21, 2023 – 18:21 UTC
We revoke a further 9 certificates that were issued over the preceding 5 days. All of these have subject:organizationName values that were automatically approved based on previously issued certificates.

February 22, 2023 - 03:21 UTC
We revoke 18 certificates for which we had granted a delayed revocation.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

No further misissuance has been discovered. Until new technical controls have been developed and deployed, there is a component of manual vetting that could in theory cause a repeat occurrence. Additional awareness has been raised internally and while we do not expect further misissuance, we will actively monitor our certificate base and revoke any certificate deemed non-compliant.

4. Summary of the problematic certificates

278 certificates issued between January 11, 2022 and February 17, 2023

5. Affected certificates

Please see attachment 9320356 [details].

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now

All affected certificates were for organizations whose full names exceed 64 characters. Since including these full names in subject:organizationName fields is not possible due to the upper bound set in RFC 5280, our validation team abbreviated the names so they would not exceed the 64 character limit.

For the initial 7 certificates in this incident, the subject:organizationName contained an open parenthesis but lacked a corresponding close parenthesis, due to our validation staff cropping parts at the end of the organizationName value, leading to a violation of the allowed abbreviations stated in the BRs. During this investigation, we decided to widen the scope by looking not only at subject:organizationName values with an open parenthesis, but also at all subject:organizationName values that are precisely 64 characters long. We found additional certificates whose subject:organizationName fields we believe were incorrectly abbreviated by our validation team.

While reviewing, we did notice that there are inconsistencies between the rules for abbreviating / shortening the subject:organizationName value in the BRs vs the EVGs, where it appears that the EVGs allow for a less strict approach than the BRs. As the BRs apply to all issued TLS certificates, including EV certificates, we are interpreting that in the case of a conflict between the two (which now is the case), the stricter requirement should be applied. We want to highlight this conflict within the guidelines as a source of confusion that may have contributed to why certain abbreviations were accepted in the past. We plan to bring a change request to the CA/B Forum to sync the BRs and EVGs on these requirements and avoid further confusion across all CAs.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future

Correctly abbreviating organization names that exceed 64 characters in length is a manual process. At present, we do not see a path to fully automating this. However, we do want to take every precaution possible and are therefore adding an extra step in our system, whereby a senior validation staff member must review and approve the proposed abbreviation. Where native local expertise is available, potentially including Subscribers, we will give weight to the recommendations of the local expert.

We believe that this dual control before approval should provide enough assurances to avoid a repeat of this incident.

Comment 6

[Martijn Katerbarg]

We have no further updates at this time.

Ben, while we are working on implementing the remediation, we would like to request a next-update for 2023-04-17.

Comment 7

[Martijn Katerbarg]

Development on the remediation has been completed and is currently undergoing QA testing.

We have set a target deployment date of April 23rd, pending the completion of QA testing. We will provide an update next week on April 24th.

Comment 8

[Martijn Katerbarg]

We deployed the changes to our production systems on April 22, 2023, 23:36 UTC.

This deployment completes the remediation of this incident.

Comment 9

[Martijn Katerbarg]

Ben, this issue has been remediated and there appear to be no further questions. As such we would like to request closing this bug.