How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
SSL.com is continuously monitoring all CA compliance-related bugs at Mozilla Bugzilla. On May 2nd 2019, Kathleen Wilson opened a bug for a finding she discovered while processing revoked intermediate certificate records. In particular, a CRL issued by one CA was missing.
This fact was treated as an internal incident. During the investigation, it was discovered that the CRL resided in the wrong URL. The problem was fixed by offering the CRL from that additional location.
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
- 2019/01/16 - The related CA (CA Certificate serial number 0xcee08f3e87a0c9a) was issued.
- 2019/01/16 - Post ceremony actions were initiated. The CRL is published to the wrong location. Part of the post-ceremony actions, including the checks and verification of proper CRL handling were delayed.
- 2019/03/05 - The serial number issue was reported. SSL.com was affected and the related CA was scheduled to be revoked.
- 2019/04/05 - The CA was revoked. Since the CA was superseded,no more actions were performed, apart from maintaining repositories, CRLs, and related info.
- 2019/05/02 - A CA compliance bug ticket is created at Mozilla’s Bugzilla and assigned to an employee. The alert for the creation of this bug did not reach our compliance or support email.
- 2019/05/06 - The compliance team was informed about the issue and alerted other related departments. An internal incident was declared and the investigation begun.
- 2019/05/06 - The problem was identified and resolved.
Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We have not stopped issuing certificates. The problem did not affect certificate issuance but rather CRL publishing of a revoked CA.
A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
No problematic certificates have been issued.
The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
No problematic certificates have been issued.
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
SSL.com uses an internal naming scheme for CAs that is based on the hierarchy, the key usage, the algorithm and the generation of the CA. A number of different scripts and processes rely on this scheme, which does however allow for customized names. One of these processes is CRL publishing.
During generation of this CA, a name customization took place in the issuing script in which the string “SSL” was decided to be removed from the Subject Common Name. Unfortunately, the CRL publishing script was not re-configured in order to accommodate this customization. This led to the problem of publishing the CRL to the location http://crls.ssl.com/SSL.com-Enterprise-Intermediate-EV-SSL-RSA-4096-R1.crl instead of http://crls.ssl.com/SSL.com-Enterprise-Intermediate-EV-RSA-4096-R1.crl.
The post-ceremony steps that would identify the issue were postponed and finally suspended due to the revocation of the CA (due to the serial number issue). This was the main reason why the problem was not detected.
List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
A number of different issues were identified in our processes which were corrected. In particular:
- The primary issue was the incorrect configuration of the CRL publishing process. In cooperation with our auditors, we updated our CA Certificate issuance procedure to strengthen our multi-person verification scheme for scripts and related configuration files to minimize the risk of such issues happening in the future. This procedure has already been adopted and used by SSL.com.
- The secondary issue was the reporting of the bug to and within SSL.com. We have already contacted the Mozilla Root Program managers in order to ensure possible future issues are reported to the compliance team email alias, which will minimize the risk of e-mail notifications being misplaced to spam folders.
- A contributing factor to this incident was miscommunication between the technical and compliance departments. This was identified at the P-384 curve / ecdsa-with-SHA256 certificates issue and has already been mitigated.
We believe that these measures will prevent similar incidents from being repeated in the future.