- How your CA first became aware of the problem.
Microsoft PKI Services has identified four (4) Intermediate CA’s that have been mis-issued because they have malformed Key Usage extensions. We became aware of this issue on 24 June 2021 01:10 PM (Pacific Time) when the team manually inspected the just created certificates during the live certificate generation process.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
Note: Times are listed in the Pacific time zone.
• 13 May 2021 03:59 PM – Bugzilla Bug reporting (8) Malformed ICA’s (missing certificate policy extensions) created (1711147 - Microsoft PKI Services: Malformed ICAs (missing certificate policy extensions) (mozilla.org))
• 22 June 2021 – Live templates created in our internal tools.
• 24 June 2021 01:03 PM – Four (4) Intermediate CA certificates issued from our Microsoft RSA Root Certificate Authority 2017. This Root certificate is in an Offline environment segregated from any networks.
• 24 June 2021 -1:08 PM – Team identified during manual inspection of the certificates that the Key Usage field was malformed.
• 24 June 2021 01:18 PM – Revoked four (4) mis-issued ICA certificates from 24 June 2021.
• 24 June 2021 01:45 PM – Identified issue in template and repaired template.
• 24 June 2021 1:57 PM – Issued four (4) correctly formed ICA certificates from the RSA root.
• 24 June 2021 05:17 PM – Published updated CRL to revoke 4 mis-issued ICA certificates from 24 June 2021. http://www.microsoft.com/pkiops/crl/microsoft rsa root certificate authority 2017.crl
- Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
We stopped issuance via our offline CA systems and processes, as this is where this issue arose. We have now identified the root cause as being related to the way in which our internal software tools configure templates in our internal tools. Once the issue was fixed on the RSA root we resumed offline issuance, nevertheless we have stopped the creation of new templates until all remediations are completed.
- In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
There are four (4) certificates that we have created with this issue. All four (4) have now been revoked. All four (4) certificates were created on 24 June 2021. All four (4) of these CAs were revoked within minutes of creation and had not yet been used to issue subscriber certificates. No other ICA certificates with this issue have been issued since.
All four (4) ICA certificates have been attached to this bug.
- In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
See attachments for each of the four (4) certificates.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Upon seeing the malformed key usage in the issued certificates and revoking them immediately, the team began the investigation into root cause.
The root cause of this mis-issuance was related to the new certificate templates that the team instituted on this Root CA. These templates are configured using our own internal software and WebUI. We decided to add these templates as a defense in depth measure to remediate another ICA mis-issuance bug (1711147 - Microsoft PKI Services: Malformed ICAs (missing certificate policy extensions) (mozilla.org) that Microsoft PKI Services currently has open. We installed and tested the templates first on our test roots (one RSA and one ECC) and then on our live root certificates (one RSA and one ECC).
Our internal software and UI for creating these templates requires each template to be individually configured for each CA. So, the team had a ceremony and followed it to create the template on each of the four CAs described above (test and live, RSA and ECC roots). The configuration and setup of these templates went well on 3 of the 4 CAs but had an issue specific to the Key Usage on the RSA live root. The UI for our internal software was used during the configuration process to ensure the templates were configured identically and the UI indicated they were all configured identically.
When the team realized that there was a problem with the certificates issued from the live RSA root, they were able to do some further digging into our internal software’s database to identify that the Key Usage blob on that CA was malformed, specifically instead of the encrypted base 64 value AwIBhg== it was listed as "1". The team is not sure how the malformation happened to the Key Usage field in the template, but suspect it was a mistake when typing in the configuration of the template.
Once they identified the problem with the template, they were able to fix the configuration and successfully issue four (4) live Intermediate CA’s with the updated template.
We also want to address a very similar bug (1711147 - Microsoft PKI Services: Malformed ICAs (missing certificate policy extensions) (mozilla.org)) that Microsoft PKI Services currently has open. This mis-issuance is a different root cause, as discussed above. We would like to point out that the improvements that we have added to this process because of the recent bug helped during this issue. The manual checks that were added helped to immediately identify that these certificates had a problem, and we were able to fail fast.
- List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
• All four (4) certificates were revoked within minutes of their creation and the CRL was published a few hours later (24 June 2021).
• Within hours of the issue the team identified the problem with the template that was created on the live RSA Root CA and fixed the issue. The team was able to successfully issue the live certificates once the template was corrected (24 June 2021).
• We will update our Template creation process to compensate for the shortcomings of our internal template tools as identified in this bug. We will add steps to further interrogate the database in our internal tools to check the configuration of the template before it is used. We will not configure any new templates until this process is updated. (21 July 2021).