1934790 - GlobalSign: OV TLS certificate with incorrect countryName value for organization

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Christophe Bonjean]

Steps to reproduce:

GlobalSign received a Certificate Problem Report on 2 December 2024, 17:26 UTC for an OV TLS certificate with a subject:countryName value of “US” for an organization established in Turkey: https://crt.sh/?id=11196588366.

We are processing the revocation of the affected certificate.

Investigation has started on the root cause for the issue and we’ll provide a detailed incident report as soon as we have concluded our analysis, but no later than Monday 9 December 2024.

Comment 1

[Christophe Bonjean]

The affected certificate has been revoked on 6 December 2024, 19:52 UTC.

Comment 2

[Christophe Bonjean]

Incident Report

Summary

On 23 November 2023, GlobalSign issued the following certificate: https://crt.sh/?id=11196588366. This certificate was initiated as a renewal of https://crt.sh/?id=8029021050. During the placing of the order, the customer submitted a new CSR, which contained a subject:countryName value of “US”. The new certificate request was submitted for review by vetting. The vetting agent mistakenly accepted the new value.

Upon further review, we identified and planned the revocation of 11 additional certificates with incorrect C, ST or L combinations based on changes initiated by the customer during renewal.

Impact

We identified and planned the revocation of 12 certificates with incorrect C, ST and L combinations.

Timeline

All times are UTC.

Date (dd/mm/yyyy) - time UTC	Description
21/11/2022 17:43	Issuance of the original certificate (related to the reported, renewal certificate)
23/11/2023 15:11	Issuance of the reported, renewal certificate
02/12/2024 17:26	Certificate problem report (“CPR”) received
02/12/2024 18:58	CPR escalated to compliance team
02/12/2024 19:05	Compliance team picks up the CPR and confirms the issue
02/12/2024 21:35	Revocation scheduled for affected certificate, vetting management notified. Start of review pending certificate renewal requests and historic issuance of renewals.
03/12/2024 07:15	Update to all pending certificate requests with different subject information compared to the original order, for additional review.
03/12/2024 08:45	Response provided to reporter of CPR
07/12/2024 15:26	Scheduled revocation for reported certificate
09/12/2024 13:23	Scheduled revocation for additionally identified certificates

Root Cause Analysis

The original orders included correct and valid C, ST and L values. However, during the certificate renewal flow customers can update the certificate request (in case of changes). Despite an order being a renewal of an original order, the differences between the original order and the renewal are not highlighted to vetting agents. Additionally, the vetting workflow does not currently offer an automated way for cross-checking full address information for correctness.

The vetting agents missed the (incorrect) combinations of C, ST or L values as changed by the customer during renewal, however the limitations in terms of comparing and highlighting are also deemed factors contributing to this oversight.

Lessons Learned

What went well

• We were able to quickly review and identify other pending orders to prevent issuance of certificates with similar issues.

What didn't go well

• Differences between the subject information of the original order and the new order were not highlighted to vetting agents.
• The vetting workflow did not offer an automated way for cross-checking the full address information for correctness.

Where we got lucky

• A limited number of certificates were affected by this issue.

Action Items

Action Item	Kind	Due Date
Implement an update to the vetting interface to highlight differences between the original order and the renewal order to vetting agents, requiring an explicit approval of the changes prior to certificate issuance.	Prevent	06/01/2025
Deploy lookup feature to enable vetting agents to cross-check the full address information for correctness in an automated manner.	Prevent	06/01/2025

Appendix

Details of affected certificates

Link	Discovery	Revocation
https://crt.sh/?id=11196588366	02/12/2024 17:26	06/12/2024 19:52
https://crt.sh/?id=11343918352	03/12/2024 09:20	06/12/2024 15:29
https://crt.sh/?id=13366153318	06/12/2024 12:50	11/12/2024 (Scheduled)
https://crt.sh/?id=12089182187	09/12/2024 07:57	14/12/2024 (Scheduled)
https://crt.sh/?id=12666968192	09/12/2024 13:23	14/12/2024 (Scheduled)
https://crt.sh/?id=12687999271	09/12/2024 13:23	14/12/2024 (Scheduled)
https://crt.sh/?id=12742935791	09/12/2024 13:23	14/12/2024 (Scheduled)
https://crt.sh/?id=12742220751	09/12/2024 13:23	14/12/2024 (Scheduled)
https://crt.sh/?id=13351690773	09/12/2024 13:23	14/12/2024 (Scheduled)
https://crt.sh/?id=12836058352	09/12/2024 13:23	14/12/2024 (Scheduled)
https://crt.sh/?id=15061482626	09/12/2024 13:23	14/12/2024 (Scheduled)
https://crt.sh/?id=15052287454	09/12/2024 13:23	14/12/2024 (Scheduled)

Comment 3

[Pedro Fuentes]

Have you considered discarding the identity information in the CSR, so the data inserted in the certificate is the validated information of the subscriber and not what coming in the CSR?

Comment 4

[Christophe Bonjean]

(In reply to Pedro Fuentes from comment #3)

Have you considered discarding the identity information in the CSR, so the data inserted in the certificate is the validated information of the subscriber and not what coming in the CSR?

Yes, this is one of the considerations of an already ongoing project to re-evaluate the vetting process, where we are in progress of reviewing the subscriber and validated information flows.

Comment 5

[Christophe Bonjean]

The affected certificates have all been revoked as scheduled.

Comment 6

[Christophe Bonjean]

We are on track to deliver the actions as per the schedule. We propose to set the “Next Update” to 06/01/2025.

Comment 7

[Christophe Bonjean]

We completed the implementation of both the update to the vetting interface to explicitly require approval of the changes between the existing certificate and the renewal, and the lookup feature which enables the vetting agents to cross-check address information in an automated manner.

This concludes the identified remedial activities - unless there are any further questions we believe this issue can be closed.

Comment 8

[Ben Wilson]

Hi Christophe,
Even though this has not been formalized as a bug-closure requirement, could you please provide a closing summary?
Thanks,
Ben

A closing summary should briefly:

• describe the incident, its root cause(s), and remediation;
• summarize any ongoing commitments made in response to the incident; and
• attest that all Action Items have been completed.

Here is a markdown template if needed:

Incident Report Closure Summary

• Incident Description: [Two or three sentences summarizing the incident.]
• Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
• Remediation Description: [Two or three sentences summarizing the incident's remediation.]
• Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 9

[Christophe Bonjean]

Incident Report Closure Summary

• Incident Description:

12 certificates were issued with incorrect combinations of C, ST and/or L values due to changes by the customer at certificate renewal that were not identified during the vetting process.

• Incident Root Cause(s):

The root cause of this incident was human error and limitations of the vetting workflow.

Subject information was updated by the customer during certificate renewal and the (incorrect) changes to C, ST and/or L fields were submitted for vetting but mistakenly accepted by the vetting agents due to limitations in visually comparing and validating renewed order information.

• Remediation Description:

The remediation included two preventive measures, improving the vetting workflow:

• An explicit approval requirement for changed information during the renewal process.

• A lookup feature to enable cross-checking of address information in a more automated manner.

• Commitment Summary:

Subscriber and validated information flows are being reviewed as part of an ongoing project for evaluating the vetting processes, with the goal of identifying risk-prone information flows and increasing the use of automation to further reduce these risks within the vetting process.

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 10

[Ben Wilson]

I'll close this on or about Monday, 13-Jan-2025.