Incident Report – Mozilla Policy Violation
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
We were notified through this opened bug (1647084). Thank you for bringing this to our attention.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
02/20/20 - The request sent from MS for the ICA signing.
04/29/20 – The naming docs were finalized.
05/20/20 - The ICA was created.
05/21/20 - The ICA uploaded to CCADB. The PKI Ops personnel marked same as parent.
06/20/20 - Bug 1657084 was filed notifying DigiCert of the potential error in the CA markings in CCADB.
06/20/20 - Investigation began. Compliance team starts monthly comprehensive sweep of CCADB for review of anomalies.
06/21/20 – We acknowledged the error; CCADB entry was updated for the ICA.
- Whether your CA has stopped, or has not yet stopped, issuing certificateswith the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We are holding off on any new DigiCert SubCA signings until we complete our investigations and our remediation is in place.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
ICA (issued on 5/20/2020): https://crt.sh/?sha256=3458C7F787C30FE9AF3E58A7AB6877B1959AB7CC2C4581B070BDD38C39B84AF7
5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
See 4) above
6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The issue was a mistake made by our PKI Operator in assuming that this ICA will be hosted by us and marked the ICA “Same as Parent” during the CCADB upload process. This is the first time for our PKI Operator in handling a cross signing ceremony for an external entity. Because we use the same request documents to create both external and internal ICAs , there was no distinguishing way for the PKI Ops team to know this was externally operated. We also rarely create these, allowing TLS SubCAs only for browsers. The general policy combined with the lack of visible indicators that this was different from other signings, meant we followed the standard process for uploading a new ICA to CCADB, rather than the process for external ICA selection.
To resolve this issue, we are going to add the CCADB information to the signing request so it’s clear what the CCADB information should look like before signing the CA. We’re still working out how to better distinguish documentation in creating On Prem and hosted ICAs.
To also note, there was a Compliance review of the cert before release which confirmed the ICA was created as per the approved naming document. Compliance also signed off on the CCADB upload. Similar to PKI Ops, there wasn’t a clear indication that this was an external sub CA and required a different CCADB entry from DigiCert-hosted ICAs.
This audit information would have been corrected in the next annual audit, when the next Microsoft audit lists the ICAs operated by them. That does leave a gap in information between new ICA creation and audit coverage, which the new request documentation will aim to clarify for correct entry into CCADB.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
We are re-examining the ICA creation document to make sure we fix the gap with key information missing that would have marked ICA signings as not hosted by us. This will include information about what to upload to CCADB for on-prem CAs and will require signed off by Compliance and the PKI Ops team before ICA creation.
We plan to create a separate naming document that will only be used for SubCAs or Intermediates that will be operated on-prem by the customer. Until this is in place, we are halting any new subCA signings for DigiCert until this is well established.