1678720 - SSL.com: Wildcard DV certificate issued with a non-validated domain name

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Thomas Zermeno]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36

Steps to reproduce:

This is a preliminary incident report. Our investigation into this matter is ongoing.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

A validation Specialist (VS) found an DCV irregularity immediately after processing a wildcard server certificate request which had a typographical mistake in the domain. The VS reported the issue to the Security Auditors via an internal ticket, in accordance with our Incident Management Policy.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-11-18T19:42+00:00 Certificate mis-issuance
2020-11-18T19:43+00:00 Certificate revocation
2020-11-18T20:22+00:00 Event ticket opened by the VS to report the issue.
2020-11-18T21:11+00:00 Event ticket updated by the VS with more details on the issue.
2020-11-18T22:10+00:00 Event ticket was reviewed by a Security Auditor with a suggestion to declare an incident. Investigation initiated. Temporary measures to remediate the issue were communicated to the validation specialists.
2020-11-19T16:37+00:00 Event ticket was reviewed by a second Security Auditor and more details were requested from the VS and the engineers. Investigation continues. Review thus far found no similar cases.
2020-11-19T20:50+00:00 Internal meeting to discuss preliminary findings. Investigation continues to confirm and detail the preliminary findings and propose technical remediation measures.
2020-11-20T16:45+00:00 The issue is now re-produced by the engineers. Technical remediation measures are being implemented.
2020-11-20T18:40+00:00 Event ticket escalated to an incident.
2020-11-20: Filed initial Bugzilla report.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The problematic certificate was revoked immediately after issuance. Immediate remediation has been put in place; validation specialists have been alerted to stop using the administrative tool which made the mis-issuance possible (see section 6) until further notice (pending completion of our investigation and application of permanent remediation). Permanent technical measures are being prepared to prevent similar cases from arising in the future.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

This issue affects one certificate, issued 2020-11-18T19:42+00:00.

Our initial investigation finds that a particular and uncommon combination of access privileges and manual actions is required to replicate this issue, and our review thus far has found no similar issuances.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Impacted certificate:
SHA256 Fingerprint: 12:BA:D2:E1:73:A7:29:E2:8E:23:DE:3E:3A:2B:85:95:B5:26:0D:63:49:46:80:2E:DA:F6:DD:42:7A:4A:29:FD
SHA1 Fingerprint: B9:DD:72:6E:F0:44:E0:0C:4E:45:81:83:66:47:60:7F:0D:67:4F:BC
Pre-certificate:
https://crt.sh/?id=3665226166

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our initial review indicates that, under a strictly limited set of circumstances, safeguards in our RA system failed to prevent issuance of a certificate with a non-validated DN. In particular, the issue was caused after a manual re-processing of a wildcard DV certificate request which had a typographical mistake in the domain name and for which the initial domain validation failed. This uncommon case required manual re-processing using a tool which is available only to privileged users (senior validation specialists) and which does not enforce full re-validation checks to all domain entries (CN, SAN) of the certificate request. Because of that, a certificate mis-issuance was made possible when the validation specialist performed a partial correction of the typographical mistake.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Immediate actions are described in steps 2-4 of this report and include the immediate revocation of the affected certificate, the temporary measures which have already been applied and the technical measures to avoid re-occurrence.

Our investigation is ongoing and an analysis is also being conducted to reveal any underlying weaknesses and, according to the results, decide any additional measures and improvements in our systems and processes, so that such occurrences are not repeated in the future.

A full incident report shall be filed here when our investigation is complete.

Comment 1

[Thomas Zermeno]

This is an update to report current progress on this issue.

Our investigation is ongoing to confirm that no other occurrences exist, and technical measures have already been introduced to remediate the issue.

A full description shall be made here when the investigation is complete. Status updates to this bug will be made weekly until the final report is presented.

Comment 2

[Thomas Zermeno]

This is an update to report current progress on this issue.

Our investigation has finished and confirmed that no other occurrences exist.

Review of the entire incident is underway by our Security Auditors. A final report will be posted next week.

Comment 3

[Thomas Zermeno]

Our investigation into this matter has been completed, and this is our final report regarding this incident.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

A validation Specialist (VS) found an irregularity in our domain control validation (DCV) system immediately after processing a wildcard server certificate request which had a typographical mistake in the domain. The VS reported the issue to the Security Auditors via an internal ticket, in accordance with our Incident Management Policy.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-11-18T19:42+00:00 Certificate mis-issuance and immediate flagging as such by VS.
2020-11-18T19:43+00:00 Certificate revocation.
2020-11-18T20:22+00:00 Event ticket opened by the VS to report the issue.
2020-11-18T21:11+00:00 Event ticket updated by the VS with more details on the issue.
2020-11-18T22:10+00:00 Event ticket reviewed by a Security Auditor with a suggestion to declare an incident. Investigation initiated. Temporary measures to remediate the issue communicated to the validation specialists.
2020-11-19T16:37+00:00 Event ticket reviewed by a second Security Auditor and more details requested from VS and engineers. Investigation continues. Review thus far found no similar cases.
2020-11-19T20:50+00:00 Internal meeting to discuss preliminary findings. Investigation continues to confirm and detail the preliminary findings and propose technical remediation measures.
2020-11-20T16:45+00:00 Issue now reproduced by engineers. Implementation of technical remediation measures initiated.
2020-11-20T18:40+00:00 Event ticket escalated to an incident per our Incident Management Policy.
2020-11-20T20:49+00:00 Initial Bugzilla report filed.
2020-11-23T20:36+00:00 Security Auditors officially define the scope of the certificates population to be reviewed (target population); scope is extended to include all multi-domain server certificates, including DV, OV and EV (for reasons described in step 6 below).
2020-11-24T16:55+00:00 Engineers officially report that technical measures which remediate the issue have been deployed to the production system.
2020-11-25T19:05+00:00 Target population extracted and submitted for review by Security Auditors.
2020-11-25Τ22:38+00:00 Incident management team reviews progress and decides to proceed with review of the entire target population.
2020-12-03T19:26T00:00 Completion of review of all certificates in target population. Review confirms that no other occurrences exist.
2020-12-04T19:31T00:00 Bugzilla report updated with above confirmation.
2020-12-07 through 2020-12-11: Analysis of incident to identify possible underlying issues, with participation of all subject matter experts (security auditors, validation specialists, development manager).
2020-12-08T21:58T00:00 First draft of analysis submitted for internal review.
2020-12-11T19:36T00:00 Internal review finalized and results documented in the internal incident ticket.
2020-12-11: Final Bugzilla report filed.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The problematic certificate was revoked immediately after issuance and temporary measures were immediately introduced to avoid re-occurrence. Permanent technical measures are now in place to prevent similar cases from arising in the future.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

This issue affects one certificate, issued 2020-11-18T19:42+00:00.

A particular and uncommon combination of access privileges and manual actions by senior validation specialists was required to replicate this issue. Our review found no other occurrences.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Impacted certificate:
SHA256 Fingerprint: 12:BA:D2:E1:73:A7:29:E2:8E:23:DE:3E:3A:2B:85:95:B5:26:0D:63:49:46:80:2E:DA:F6:DD:42:7A:4A:29:FD
SHA1 Fingerprint: B9:DD:72:6E:F0:44:E0:0C:4E:45:81:83:66:47:60:7F:0D:67:4F:BC
Pre-certificate:
https://crt.sh/?id=3665226166

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our review indicated that, under a strictly limited set of circumstances, safeguards in our RA system failed to prevent issuance of a certificate with a non-validated DN. In particular, the issue was caused after a manual re-processing of a wildcard DV certificate request which had a typographical mistake in the domain name and for which the initial domain validation failed. This uncommon case required manual re-processing using a tool which is available only to privileged users (senior validation specialists) and which does not enforce full re-validation checks to all domain entries (CN, SAN) of the certificate request. Because of that, a certificate mis-issuance was made possible when the validation specialist performed a partial correction of the typographical mistake.

Our investigation confirmed that 1) no other certificate issuance was affected by this issue; 2) this tool was used very rarely; and 3) that once the certificate was mis-issued, it was detected immediately and revoked within two minutes.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Immediate actions are described in steps 2-4 of this report and include the immediate revocation of the affected certificate and the temporary measures which were initially applied.

A review was initiated to identify whether this issue affected the issuance of any other existing certificates. The affected certificate was a wildcard DV certificate, but after examination of the event, our Security Auditors decided to extend the scope of the investigation and conduct a full review of all multi-domain server certificates, including DV, OV and EV. The reason was that the issue was not related to the use of a wildcard per se, but to the manual re-processing of multiple FQDN entries (domains, subdomains), which may entail a higher human error than other cases. The investigation confirmed that no other certificate issuance was affected by this issue.

In parallel, after reproducing the issue in our systems and identifying the cause, permanent technical measures were implemented and introduced to the production RA to avoid re-occurrence when utilizing this tool. This fix checks all FQDNs for valid DCV records before allowing the tool to call the CA to issue certificates. If a FQDN or wildcard Domain Name is present which does not map to a proper DCV record, the system blocks the request and logs the offending dNSName entries. The system also alerts the validation specialists notifying them of the offending entries. The above fix addresses the current incident by preventing a privileged validation specialist using this tool from unintentionally including a domain name without proper DCV records into a certificate.

The above fix went through our standard review processes before deploying to production. A review of the RA codebase confirmed that all calls to the CA to issue TLS certificates enforce the DCV rule and that the issue was related only to this offending tool.

In accordance with our Incident Management Policy, apart from fixing the problem and improving our systems, an additional internal analysis took place to identify how the weakness was introduced in the first place and to ensure that any underlying issues are also mitigated. Our analysis showed that the issue stems from an older decision to provide specific senior validation specialists with the ability to manually update certificate orders. Limiting the use to well-trained and mindful VS was, at the time of adoption, considered a sufficient compensating control to ensure that, in the rare usage of the tool, no problem would emerge. However, in this instance reliance on training, rather than technical controls, created the opportunity for this error to occur.

Our processes have been improved since introduction of this tool. For example, active involvement of subject matter experts, compliance personnel and engineers is required for changes under our Change Management Policy. This improves the identification of possible pitfalls and the selection of measures to address these. Monitoring and evaluating the effectiveness of our processes is part of our standard practice.

Comment 4

[Ben Wilson]

It appears that the incident/issue has been appropriately remediated and I will close this bug on or about Friday, 15-Jan-2021, unless there are additional questions or issues to discuss.

Comment 5

[Ryan Sleevi]

Sounds good. I just want to acknowledge and thank SSL.com for the level of detail in Comment #3, in particular, Step 7, which has meant I've not had to freak out about this bug :)