- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
As detailed by Ryan Hurst, we were aware of the need to update these roots in CCADB and contacted Mozilla when CCADB tooling prevented us from doing so. Initial guidance was to add the certificates to https://bugzilla.mozilla.org/show_bug.cgi?id=1652581 and that a CCADB admin would update the records. This is effectively another variation of https://github.com/mozilla/pkipolicy/issues/186 which is pending resolution and in this case caused both Mozilla and GTS representatives confusion about the best path forward.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2020-07-07 GTS determines that the best mitigation to incident 1652581 is the reissuance of the Root Certificates
2020-07 (multiple dates) GTS communicates with the various root programs and our plan to re-issue and request guidance. All contacted Root programs agreed that reissuance was the right way to address the issue.
2020-08-13 - Re-issuance ceremony conducted.
2020-08-14 - Initial attempt to update CCADB was made and not successful.
2020-08-17 - Other GTS members with CCADB access attempted upload to confirm the issue.
2020-08-21 - GTS reached out to Mozilla root program representatives for guidance on how to proceed.
2020-08-28 - The re-issued certs were added to https://bugzilla.mozilla.org/show_bug.cgi?id=1652581.
2020-09-28 - An incident report was requested and this report is being provided.
- Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.
Original versions of the certificates that were 'not reported' to CCADB are already trust anchors. Issuance from those CAs is already disclosed. No need was identified to stop issuance of certificates.
- In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
See point 5 for a list of affected certificates.
- In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
The relevant crt.sh links are:
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
CCADB does not have support for re-issued roots. This manifested as a permission error when GTS tried to update CCADB. We relied on direct communications with CCADB administrators to identify a workaround to this issue. This is an area where improved community guidance would help.
- List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
We are working with Mozilla to resolve the issue to their satisfaction. Update policy and associated guidance should ensure that in the future others do not encounter this problem. The 2.7.1 Mozilla Root Policy Store update with resolution of https://github.com/mozilla/pkipolicy/issues/186 including coverage of re-issued certs would provide good guidance.