We've continued doing some analysis here and re-running reports over our body of issued certificates.
As such, I can expand on sections 6 and 7:
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
As explained above, the report in https://bugzilla.mozilla.org/show_bug.cgi?id=1518553#c11 was generated by Robin following his comment at https://bugzilla.mozilla.org/show_bug.cgi?id=1518553#c7:
"We will follow up with analysis of the prevalence of subscriber keys in issued certificates whose RSA moduli are not divisible by 8."
After discussing the history of bug #1518553 in more depth with Robin and Rob yesterday, I now understand that that report was generated from CT log data (using crt.sh) and NOT, as I had stated earlier in this bug, from our internal dataset. Robin had wanted to look at and share data about disallowed RSA moduli in the wider WebPKI ecosystem, not just Sectigo's own failings in this regard. Robin's chosen approach was also influenced by the fact that, at the time, querying our internal CA database for disallowed RSA moduli was a slower and more complex process than querying crt.sh.
My statement about the certificates being 'replaced' was true, but was not the cause of the problem. The two certificates mentioned here did not appear in any CT logs before early 2020, having been issued prior to Chrome's CT enforcement date. The report that Robin generated was based on an incorrect understanding that the crt.sh dataset would cover all Sectigo-issued certificates. While CT compliance was mandatory for newly-issued certificates at the time of the report generation, it was overlooked that some older certificates issued by Sectigo had not been submitted to CT logs by Sectigo and may not have been found and submitted to CT logs by GoogleBot or other third-parties.
We neglected to double-check the reporting and that it covered all Sectigo-issued certificates. Consequently, the requirement to 'scan your corpus of certificates' [https://wiki.mozilla.org/CA/Responding_To_An_Incident#Follow-Up_Actions] was missed, and this resulted in 12 Sectigo-issued certificates with RSA moduli not divisible by 8 remaining undetected until Jeremy's report reached us.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
From the list provided by Jeremy (Digicert) we found two (2) certificates that were not in Robin's original report, as detailed above.
We then performed a scan over the body of certificates in our own CA system, looking for any certificates we issued whose RSA moduli are not divisible by 8.
Upon completion of the report, filtering down to unexpired, unrevoked certificates that were not on the list Robin originally reported, we had an additional ten (10) certificates.
As with the previous two, rather than suggest they should have been part of the original report and remain valid, our team is already processing revocations.
They will be revoked on or before 2-Aug-2020 6am UTC.
The crt.sh links for these ten are:
As will be detailed in an (imminent) update to bug #1563579, changes have and are being made to the processes by which incidents are handled now and going forward. The change of most relevance to this bug is that we will be introducing a more formal process of peer review for each phase of each incident response, to reduce the likelihood that one person's incorrect understanding can remain undetected.
As for ensuring that "such issuance will not be repeated in the future", it's worth noting that the defect in Sectigo's CA system (that permitted RSA moduli not divisible by 8) was successfully addressed by our response to bug #1518553.