1782391 - GlobalSign: EV certificate with wildcard domain in common name and SAN

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Comment 1

[Christophe Bonjean]

GlobalSign issued 1 EV TLS certificate with a wildcard domain in the subject common name and SAN fields. We are currently investigating and will post a full incident report by August 3, 2022.

Comment 2

[Christophe Bonjean]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We received a system notification on 30/07/2022 at 15:00 UTC via email to our compliance team. This email was acknowledged at 15:37 UTC on the same day, and the investigation started on 18:50 UTC, confirming that the certificate was mis-issued.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY - Time in UTC	Description
30/07/2022 15:00	Certificate warning message received.
30/07/2022 18:50	Investigation started by compliance team.
30/07/2022 18:51	Confirmed mis-issuance. Replacement process initiated.
30/07/2022 19:07	Started review of historically issued certificates and outstanding requests. Initiated root cause analysis.
30/07/2022 19:21	Completed review of historically issued certificates. No additional certificates identified. Completed review of outstanding requests. No outstanding requests affected by this issue.
30/07/2022 19:23	Vetting and compliance put on high alert to monitor incoming requests, monitoring reports set up.
01/08/2022 01:36	Confirmed root cause.
01/08/2022 11:33	Applied code fix in production system.
02/08/2022 05:47	Updated linter in staging system.
02/08/2022 21:18	Certificate revoked.
03/08/2022 07:42	Updated linter in production system.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We confirmed there were no additional historically issued certificates or outstanding requests affected by this issue (confirmed on 30/07/2022 19:21 UTC). Vetting management was put on high-alert with monitoring reports set-up and shared with Compliance for review on 30/07/2022 at 19:23 UTC. These measures will remain in place until all remedial actions have been completed.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

https://crt.sh/?id=7231582044

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See #4.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Certificates can be ordered individually or via an organization level profile. This particular certificate was ordered through the organization level flow (MSSL).

Upon receipt of a certificate order, our normal procedure is to parse the CSR and process the fields against a set of validation rules. Prohibiting wildcards in EV TLS certificates is one of the validation rules. In this case, due to a code bug in our MSSL ordering procedure, the system failed to trim the Common Name before processing the validation rules and therefore failed to flag that the Common Name value was a wildcard due to a whitespace at the beginning of the field. The whitespace was trimmed before issuing the certificate, but after processing this particular validation rule.

Monitoring and alerting of wildcards in EV certificates has been enabled within our vetting application as an additional measure since the requirement became effective. This mechanism successfully reported prior occurrences of wildcards and the validation rule blocked the issuance.

The certificate was not blocked in the issuance path as the pre-linter was using zlint version 3.3.0, which did not yet include a lint to prohibit wildcards in EV TLS certificates. Version 3.3.1 was originally planned for deployment together with a number of custom, internally-developed lints. Due to delays in the development process of the custom lints, consequently the release of version 3.3.1 was also delayed.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The impacted certificate has been revoked and we have updated the affected functionality in the certificate processing component to ensure wildcards in EV TLS certificates are blocked, even if the provided domain name is prefixed with spaces.

The latest version of zlint has been rolled out to production, which includes a lint for this issue and blocks the certificates in the issuance path. For more regular and timely deployment of linter releases, the planning and deployment process has been revised.

Comment 3

[Ben Wilson]

Besides domain names prefixed with spaces, did GlobalSign evaluate other potential situations where the wildcard character might survive trimming and find its way into a certificate? In other words, shouldn't pre-issuance processes simply prevent a wildcard character from appearing anywhere in the CN or SAN of an EV certificate?

Comment 4

[Christophe Bonjean]

The CN and SAN fields are processed by two layers in the issuance path.

The first layer is a check applied to all TLS certificates, which ensures that a wildcard does not appear in a domain name, except at the beginning. The second layer is EV TLS specific and filters out all domains prefixed with a wildcard.

This combination of checks results in prohibiting the wildcard, either as a prefix or within the domain name, for all EV TLS certificates.

To further reduce reliance on a combination of checks, we will add a single "EV TLS certificate does not contain wildcards" in the next platform release, expected by August 29th.

Comment 5

[Christophe Bonjean]

The latest platform release has been deployed. This concludes the identified remedial activities, unless there are any further questions we believe this issue can be closed.

Comment 6

[Ben Wilson]

I'll close this on or about Friday 2-Sept-2022 unless new issues are raised.