This is our final report on this issue.
1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
The issue was reported by a third party to SSL.com Support via email and then to our Security Auditors via an internal ticket, in accordance with our Incident Management Policy.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2020-11-12T15:29:52-06:00 Two-person validation completed to issue an EV TLS certificate to ansonnet.com.
2020-11-16T22:55:45+00:00 Certificate with S/N 10975D2ED70F692430714C4586E8BD78 issued to ansonnet.com.
2021-03-11 Removal of an auto-population feature of the organization validation system which increased the probability of a human mistake.
2021-05-24 Validation of the organization information of TLS certificates transferred to our new Validation Control System (this VCP was already in use for Code Signing certificates).
2021-07-06T18:12+00:00 Front desk received an email from a third party reporting possible mis-issuance of the above certificate with S/N 10975D2ED70F692430714C4586E8BD78.
2021-07-06T20:03+00:00 Event ticket opened by the Validation Specialists (VS) to report the issue.
2021-07-06T20:04+00:00 VS report the issue to Security Auditors, investigation initiated.
2021-07-06T20:21+00:00 Event ticket reviewed by a Security Auditor who requests more information and possible evidence to support the selected "O” field value. Ongoing investigation by VS to provide more information and update the ticket accordingly.
2021-07-06T21:12+00:00 A second Security Auditor engages to assist and discuss the preliminary findings with the VS. The Security Auditor reports internally a possible match with criterion no.8 of our CP/CPS “184.108.40.206 Reasons for Revoking a Subscriber Certificate” which requires revocation within 5 days (preferably 24h) if “SSL.com determines or is made aware that any of the information appearing in the Certificate is inaccurate;”
2021-07-06T21:41+00:00 An internal meeting takes with the participation of VS and the Security Auditor. The issue is confirmed and a list of immediate actions is decided. These include:
(a) reissuance of the certificate with the correct "O" field value
(b) contact the customer regarding revocation of the affected certificate
(c) revocation of the affected certificate before 2021-07-08T17:00+00:00
(d) update the reporter on the issue
(e) continue collection of information to fully document the issue.
2021-07-06T22:13+00:00 VS continue to update the ticket with additional information (action item e).
2021-07-07T01:20+00:00 Our front desk updates the reporter about the issue (action item d).
2021-07-07T04:36+00:00 Our front desk informs the subscriber about the certificate replacement (action item b).
2021-07-07T14:36+00:00 Security Auditors start drafting a preliminary incident report.
2021-07-07T16:57+00:00 Event ticket escalated to an incident.
2021-07-08Τ12:43+00:00 Security Auditors are notified that the customer has initiated certificate replacement (action item a). Revocation targeted for the same date (action item c).
2021-07-08Τ17:01:42+00:00 Certificate revoked.
2021-07-08 to 2021-07-09: Security Auditors update this preliminary report after gathering and reviewing all the necessary information.
2021-07-09T19:50:00+00:00: Filed initial Bugzilla report, with full report to follow pending completion of in-depth review.
2021-07-12: Initiated review of all our non-expired non-revoked EV and OV TLS certificates issued before the removal of the feature in question on 2021-03-11 to discover any similar cases.
2021-07-23: Completed review of the target population and found one similar case (see section 5 of this report). A detailed manual review of 100% of potentially impacted certificates was also initiated looking for any other possible discrepancies.
2021-08-13: The manual and cross-checked review of the target population was completed without finding any other similar cases (apart from the abovementioned two cases).
2021-08-16 to 2021-08-27: The entire incident was analyzed by the compliance department in coordination with the validation and engineering departments to identify the underlying weaknesses and determine any mitigation measures.
2021-08-30: Started drafting the final Bugzilla report.
2021-09-03: Filed final Bugzilla report (this document).
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
The affected certificates were both issued before 2021. In 2021, as part of our ongoing improvement process, we introduced specific technical measures and applied improved validation control systems which minimize the possibility of human mistake when validating organization information. In particular:
On 2021-03-11, we removed a productivity feature which was determined to be error prone when processing organization information for larger volumes of certificates (see section 6).
On 2021-05-24 we also transferred the validation of the organization information of TLS certificates under our new Validation Control Panel (VCP) which uses structured online checklists to guide VS through each step of the process.
Since then, we have witnessed a significant improvement in our capacity to perform human reviews of certification information. These measures would have prevented this issuance, and no similar issuances can be performed currently.
4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
2 certificates, issued between 2019-05-09 and 2020-11-06.
5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
Serial Number: 10975D2ED70F692430714C4586E8BD78
Serial Number: 42aea52595aa415921fcc5c2fbbee58c
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
This issue involved a productivity feature in our certificate systems which has been updated since 2021-03-11.
Prior to 2021-03-11 this feature allowed registrant information (including in this case information placed in the "O" field) to be pre-populated using CSR content before processing by Validation Specialists.
Prior to our feature update, a manual check by multiple personnel was required and expected to capture any discrepancies. In these instances, our manual review failed to note the discrepancy in "O" field information before issuance.
In addition to analyzing the underlying weaknesses which led to the issuance of the problematic certificates (see section 7), a separate analysis took place on why the issue was not discovered by our standard monitoring processes. We determined that the issue was not caught by our linting tools because the mis-issuance was an inaccuracy in the validated content, not a certificate content error. It was also not discovered during quarterly certificate reviews (QCR) due to the very low number of cases (two).
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Immediate actions are described in steps 2-4 of this report and include the immediate involvement of our internal auditing department, revocation and replacement of the affected certificates, and investigation of the issue.
Our initial review confirmed that the validation control system which has been introduced in 2021 would have prevented these issuances. However, this had not been applied to our systems at time of issuance of the certificates in question.
Based on the full review of the entire population of all our non-expired non-revoked EV and OV TLS certificates issued before the removal of the feature in question on 2021-03-11, only a very limited number of certificates (two) were affected by the issue.
In accordance with our Incident Management Policy, an analysis of the root cause for this issue was conducted to reveal any underlying weaknesses. The event management team determined that the source of the issue was an older consideration that such a feature (pre-populating the contents of the form that is used by the VS to process the identity information) would facilitate and expedite the validation process and any discrepancies would be spotted by the VS by manual analysis and correction. In practice, the correction of pre-populated values allowed a higher probability of human error than starting from a clean sheet and entering validated values one-by-one. This aspect was pointed out by our validation team and an update to deprecate the feature was delivered as part of our continuous improvement plan in March 2021.