1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
We received a system notification on 30/07/2022 at 15:00 UTC via email to our compliance team. This email was acknowledged at 15:37 UTC on the same day, and the investigation started on 18:50 UTC, confirming that the certificate was mis-issued.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
|DD/MM/YYYY - Time in UTC
||Certificate warning message received.
||Investigation started by compliance team.
||Confirmed mis-issuance. Replacement process initiated.
||Started review of historically issued certificates and outstanding requests. Initiated root cause analysis.
||Completed review of historically issued certificates. No additional certificates identified. Completed review of outstanding requests. No outstanding requests affected by this issue.
||Vetting and compliance put on high alert to monitor incoming requests, monitoring reports set up.
||Confirmed root cause.
||Applied code fix in production system.
||Updated linter in staging system.
||Updated linter in production system.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
We confirmed there were no additional historically issued certificates or outstanding requests affected by this issue (confirmed on 30/07/2022 19:21 UTC). Vetting management was put on high-alert with monitoring reports set-up and shared with Compliance for review on 30/07/2022 at 19:23 UTC. These measures will remain in place until all remedial actions have been completed.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Certificates can be ordered individually or via an organization level profile. This particular certificate was ordered through the organization level flow (MSSL).
Upon receipt of a certificate order, our normal procedure is to parse the CSR and process the fields against a set of validation rules. Prohibiting wildcards in EV TLS certificates is one of the validation rules. In this case, due to a code bug in our MSSL ordering procedure, the system failed to trim the Common Name before processing the validation rules and therefore failed to flag that the Common Name value was a wildcard due to a whitespace at the beginning of the field. The whitespace was trimmed before issuing the certificate, but after processing this particular validation rule.
Monitoring and alerting of wildcards in EV certificates has been enabled within our vetting application as an additional measure since the requirement became effective. This mechanism successfully reported prior occurrences of wildcards and the validation rule blocked the issuance.
The certificate was not blocked in the issuance path as the pre-linter was using zlint version 3.3.0, which did not yet include a lint to prohibit wildcards in EV TLS certificates. Version 3.3.1 was originally planned for deployment together with a number of custom, internally-developed lints. Due to delays in the development process of the custom lints, consequently the release of version 3.3.1 was also delayed.
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
The impacted certificate has been revoked and we have updated the affected functionality in the certificate processing component to ensure wildcards in EV TLS certificates are blocked, even if the provided domain name is prefixed with spaces.
The latest version of zlint has been rolled out to production, which includes a lint for this issue and blocks the certificates in the issuance path. For more regular and timely deployment of linter releases, the planning and deployment process has been revised.