1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list (https://groups.google.com/a/mozilla.org/g/dev-security-policy), a Bugzilla bug, or internal self-audit), and the time and date.
This Bugzilla bug.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
Nov 14, 2022 - Notified of the bug
Nov 14, 2022 - ATS and Digicert audited the CRL entries in CCADB. ATS removed URLs that never ended up being used because no certificates were issued prior to the final URLs being in place.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
In support of our Oct 8, 2022 launch of new ICAs we were engaged with DigiCert in configuring those ICAs during Sept 2022. We originally planned to not put the CRLs into CCADB until after both sides had given the green light that the setup was complete. However, due to the confusion shortly before the Oct 1, 2022 deadline to meet the browser requirements to disclose CRLs in CCADB we determined that we should err on the side of caution and put in the staged non-final URLs for the CRLs even though we did not intend to use them. We did this before further clarification was issued from the browsers about CRL disclosure. But we did not remove the unused URLs after this clarification was made.
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
To remediate this issue ATS removed URLs that never ended up being used because no certificates were issued prior to the final URLs being in place.
Now that there is clarity that we do not need CRL entries for ICAs that haven’t yet been used we will verify CRL entries in CCADB before we plan to start issuance. This will ensure that the CRL infrastructure is finalized prior to issuance.