1870276 - GlobalSign: TLS OV Certificate containing unverified information

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Eva Van Steenberge]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

Steps to reproduce:

On 2023-12-14 at 06:10 UTC an internally requested certificate was issued containing unverified information in the Subject:OrganizationName, Subject:stateOrProvinceName, Subject:localityName, and Subject:countryName. It was revoked at discovery of the mis-issuance on 2023-12-14 at 06:38 UTC, after which it was internally escalated to Compliance at 2023-12-14 at 06:56 UTC.

We are investigating and will update with a full incident report by 2023-12-20.

Comment 1

[Christophe Bonjean]

Incident Report

Summary

On 2023-12-14 at 06:10 UTC an internally requested test certificate was issued containing unverified information in the Subject:OrganizationName, Subject:stateOrProvinceName, Subject:localityName, and Subject:countryName. It was revoked at discovery of the mis-issuance on 2023-12-14 at 06:38 UTC, after which it was internally escalated to compliance at 2023-12-14 at 06:56 UTC.

Impact

1 OV TLS certificate issued containing unverified subject organization information.

Timeline

All times are UTC.

2023-12-14:

• 5:40: Support agent performs testing in staging environment, but fails.
• 5:57: Support agent reaches out to vetting management to request approval to replicate issue in production.
• 6:01: To prevent issuance, validation specialist is requested not to process the case.
• 6:10: Validation specialist misunderstands the request and proceeds with issuance.
• 6:28: Vetting management becomes aware of case closure and certificate issuance.
• 6:33: Vetting management requests revocation of the certificate.
• 6:38: Certificate is revoked.
• 6:56: Vetting management internally escalates to the compliance team.

Root Cause Analysis

To investigate a time-sensitive customer certificate request issue, our support team attempted to request a test certificate in our staging environment. Due to planned maintenance (between 2023-12-13 and 2023-12-14) of the staging platform, the support team was unable to successfully complete the test.

Since the customer's issue initially occurred in the production environment, in an attempt to further investigate, the support agent explored the option of replicating the issue on our production environment. The agent reached out to the vetting team leader to confirm if the test certificate could be ordered. Vetting management (mis)understood that the certificate would only be ordered, but not issued (which would prevent any impact).

Vetting management approved this request and specified that the case should not be processed as it was a test case. This instruction was misunderstood, "not to be processed" was taken to mean "should not be vetted" instead of the intended meaning "do not proceed with vetting, to prevent issuing". The validation specialist proceeded with the case, the systematic two pair of eyes review was not applicable to this product since it's an organization-validated TLS certificate, and the test certificate was issued.

The root cause of this incident is therefore a combination of two layers of miscommunication.

Lessons Learned

What went well

• After the initial discovery, the certificate was revoked quickly, and the case was appropriately escalated to the compliance team for review.
• The stakeholders accurately documented the exceptional approval.

What didn't go well

• Vetting management misunderstood the scope of the request of the support agent.
• The validation specialist misunderstood the scope of approval of vetting management.
• The exception was not escalated to the compliance team prior to deviating from the standard vetting procedures.
• Testing was performed in our production environment, which should be limited to non-production environments.

Where we got lucky

• A single certificate was issued.

Action Items

Action Item	Kind	Due Date
Definition of formal escalation process and training of applicable departments, including vetting and support.	Prevent	2024-01-24
Update to Acceptable Use Policy to restrict testing in production environments	Prevent	2024-01-19
Investigation of Disciplinary actions	Prevent	2024-01-31

Appendix

Details of affected certificates

Fingerprint: 8a9c45e047b9a9cc8af51d6b8c05bee07c9c07aa51338a7692b1f3fee4cefd0f

Subject DN: C=JP, ST=Chiba, L=Chiba, O=GSKKTEST, CN=iis.gssup.work

https://crt.sh/?sha256=8a9c45e047b9a9cc8af51d6b8c05bee07c9c07aa51338a7692b1f3fee4cefd0f

Comment 2

[amir]

Please note: I don’t believe disciplinary actions have ever been a thing that any of the root programs have endorsed as a response to an incident.

And disciplinary actions do NOT prevent a reoccurrence of this in the future.

Comment 3

[Christophe Bonjean]

Thank you for the feedback Amir.

We agree that disciplinary actions in themselves do not holistically prevent re-occurrence of this incident, which is the reason we're also taking other actions. Our perspective is that disciplinary actions are imperative to the continuous process of ensuring the trustworthiness of all validation specialists to perform their duties satisfactorily. Considering that information verification still relies to a large extent on human processes and performance, we do believe further review of the appropriateness of disciplinary action is one of the necessary actions for this incident.

For the label, our point of view is that appropriate due diligence by the different actors in this incident would have prevented the certificate from being requested, vetted and issued and therefore we concluded to assign this the "prevent" label.

In terms of other actions, to further reduce the risk of human factors, as part of the escalation action item we deployed a technical and automated control to block certificates with "test" values in the Subject DN (excluding CN) to prohibit issuance without further due diligence and approval by our Compliance team.

Comment 4

[Christophe Bonjean]

There were no scheduled deliverables this week. We are on track to deliver the other actions as per the schedule.

Comment 5

[Christophe Bonjean]

There were no scheduled deliverables this week. We are on track to deliver the other actions as per the schedule.

Comment 6

[Mathew Hodson]

(In reply to Christophe Bonjean from comment #1)

The validation specialist proceeded with the case, the systematic two pair of eyes review was not applicable to this product since it's an organization-validated TLS certificate, and the test certificate was issued.

Based on this, a possible action item would be to add a second review to this product so that one person's mistake doesn't lead to mississuance. Was that evaluated? I don't see it mentioned in the report.

Comment 7

[Christophe Bonjean]

(In reply to Mathew Hodson from comment #6)

Based on this, a possible action item would be to add a second review to this product so that one person's mistake doesn't lead to mississuance. Was that evaluated? I don't see it mentioned in the report.

During our analysis of remediating actions, we compile the list of actions based on the effectiveness and appropriateness in relation to the root causes of the incident. We did evaluate the option of applying a second review for organization validated TLS on a product level, but determined that it would not effectively and proportionally address the underlying root causes of this particular incident. According to our analysis, the proposed measures are more effective and proportional to preventing a similar incident in the future.

Comment 8

[Christophe Bonjean]

We completed the following remaining action items:

• The Acceptable Use Policy has been updated with restrictions for testing in production environments.
• A formal escalation process is established, with training of the applicable departments.
• Disciplinary actions have been internally investigated.

This concludes the identified remedial activities - unless there are any further questions we believe this issue can be closed.

Comment 9

[Ben Wilson]

I will close this tomorrow, Wed. 24-Jan-2024, unless there are reasons to keep it open.