1744518 - GlobalSign: EV certificates with serialNumber Government Entity and businessCategory Private Organization

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Paul Brown]

GlobalSign has issued two EV SSL certificates with serialNumber "Government Entity" and businessCategory "Private Organization". We are currently investigating and will post a full incident report by Friday 10 December

https://crt.sh/?q=3a7894cbba70eb06bddae0f5ae9a0e21feb646b384d7aab94024e7b59ecbd2a3
https://crt.sh/?q=f33a69f82e348ceb876062fa677690b021f4342b10838ccfd6314e08bdb7f12b

Comment 1

[Arvid Vermote]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We received a certificate problem report via our [e-mail] email address at 17:17 UTC on 03/12/2021. This report created the corresponding tickets in our internal SOC, which escalated to the Compliance team, who started the investigation for the certificate problem report at 21:37 UTC on 03/12/2021.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY - Time in UTC	Description
03/12/2021 17:17	Certificate problem report received
03/12/2021 21:37	Investigation started.
03/12/2021 21:42	Confirmed the two certificates in report required revocation. Replacement process initiated.
03/12/2021 21:45	Started review of historically issued certificates and outstanding requests. Initiated root cause analysis.
03/12/2021 22:17	Root causes confirmed and change request created.
03/12/2021 22:32	Completed review of historically issued certificates. No additional certificates identified. Completed review of outstanding requests. No outstanding requests affected by this issue.
03/12/2021 22:37	Responded to reporter.
03/12/2021 22:45	Vetting and compliance put on high alert to monitor incoming requests, monitoring reports set up.
08/12/2021 14:58	Certificates revoked.

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Vetting management was put on high-alert with monitoring reports set-up and shared with Compliance for review on 03/12/2021 at 22:45. At the same time, we halted issuance on the affected profile, pending updates to the profile. We confirmed there were no additional historically issued certificates or outstanding requests affected by this issue (confirmed on 03/12/2021 at 22:32). Additional monitoring reports were set up on 03/12/2021 at 22:45. These measures will remain in place until all remedial actions have been completed.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

https://crt.sh/?q=3a7894cbba70eb06bddae0f5ae9a0e21feb646b384d7aab94024e7b59ecbd2a3
https://crt.sh/?q=f33a69f82e348ceb876062fa677690b021f4342b10838ccfd6314e08bdb7f12b

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?q=3a7894cbba70eb06bddae0f5ae9a0e21feb646b384d7aab94024e7b59ecbd2a3
https://crt.sh/?q=f33a69f82e348ceb876062fa677690b021f4342b10838ccfd6314e08bdb7f12b

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

One certificate was requested through an approved organization level profile (case 1), the other certificate was requested through our retail channel (case 2).

Case 1: Organization level profile

The validation rules explained in related ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1714968#c10 failed to work correctly, as two relevant fields (Registration Number and Business Category) were not populated on the case itself, but rather were populated on the organization level profile related to the case. Therefore the conditions of the validation rule on the case were not met, which meant that the validation agents involved were able to close the case successfully without the rule being triggered.

Case 2: Retail channel

The validation rule was triggered, but only at a later stage in the validation process. Since the information was reviewed by a 2nd validation agent, the validation agent assumed that the information was successfully validated and the certificate could be issued.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Actions will be taken on three levels:

• Improving training and performance of the validation agents to reduce the likelihood of human error
• Strenghtening the validation logic as part of the information validation flow
• Extending linting criteria with a custom lint for blocking the prohibited serialNumber and Business Category combination

End Date (DD/MM/YYYY)	Status	Description
07/12/2021	Done	Validation agents that performed the second review of the related cases have been removed from their 2nd vetting role.
20/12/2021	Ongoing	The validation rule will be extended to also cover the organization level profile fields and trigger for organization level profile requests.
27/12/2021	Ongoing	The phases in which the validation rule should trigger will be reviewed, the rule will be updated and tested.
10/01/2022	Ongoing	Specific re-training on the topic will be organized for all related validation agents.
14/01/2022	Ongoing	We will deploy a custom lint to block serialNumber=Government Entity where the Business Category does not equal "Government Entity". This lint has been deployed to our post-linting and will be deployed to staging before going to our production environment.

Comment 2

[Eva Van Steenberge]

We have extended the validation rule to also cover the organization level profile fields and trigger for organization level profile requests on 16/12/2021. We are on track to deliver the other actions as per the schedule. We continue to review new requests prior to issuance in order to help ensure no wrong values are included in new certificates.

Comment 3

[Eva Van Steenberge]

We have updated the phase in which the validation rule is triggered on 24/12/2021, so the validation happens prior to the due diligence checks of the second review. We are on track to deliver the other actions as per the schedule. We continue to monitor new requests in order to help ensure no wrong values are included in new certificates.

Comment 4

[Eva Van Steenberge]

There were no scheduled deliverable this week. We are on track to deliver the other actions as per the schedule. We continue to monitor new requests in order to help ensure no wrong values are included in new certificates.

Comment 5

[Eva Van Steenberge]

There were no scheduled deliverable this week. We are on track to deliver the other actions as per the schedule. We continue to monitor new requests in order to help ensure no wrong values are included in new certificates.

Comment 6

[Eva Van Steenberge]

Following individual re-training sessions with the related validation agents during the month of December, all vetting agents have been subject to re-training on this specific topic as of 10/01/2022.

On 13/01/2022 we have deployed a pre-issuance custom lint to block serialNumber=Government Entity where the Business Category does not equal "Government Entity".

This concludes the identified remedial activities - unless there are any further questions we believe this issue can be closed.

Comment 7

[Ben Wilson]

I anticipate closing this on Wed. 19-Jan-2022.