Microsoft PKI Services: CA Certificates not published in DER Encoded Format
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: Dustin.Hollenback, Assigned: Dustin.Hollenback)
Details
(Whiteboard: [ca-compliance] [policy-failure])
DRAFT Incident Report
Summary
Microsoft PKI Services discovered that 8 certificates published to the AIA repository were PEM encoded instead of DER encoded, which did not comply with RFC 5280 Section 4.2.2.1. These certificates were published to the AIA URIs by 2023-07-07. Our AIA publishing tools did not detect that they were not DER encoded and MS PKI Services published them.
Impact
The only known impact was from one Microsoft service that implemented custom certificate chaining validation that expected DER formatted certificates in the AIA path. This is an issue with the format of the files in our file repository that AIA URIs point to.
We have not stopped issuance of any certificates because certificates are not malformed. We started the process to update the 8 CA certificates referenced in the AIA. This deployment will use a safe deployment practice which takes 10 days to perform a staged deployment in order to minimize the risk that the change might impact existing customers.
Timeline
All times are UTC.
2023-06-27 – 2023-07-07: Microsoft PKI Services published the 8 CA certificates to the AIA file repository (listed in Appendix) that are the subject of this Incident.
2024-02-09 23:28: Internal Incident (x7168) with MS PKI Services Subscriber that identified an Internal Customer having issues with the certificates published to the AIA file repository. This was opened as a low priority incident that was not fully investigated immediately.
2024-03-07 16:27: Internal Incident (x4270) with Microsoft PKI Services team started to investigate AIA links being encoded in DER format.
2024-03-07 ~21:30: Microsoft PKI Services confirms that the 8 files we published were not DER encoded.
Root Cause Analysis
The team will update this report by Saturday, 2024-03-16 with a completed Root Cause Analysis.
Lessons Learned
The team will update this report by Saturday, 2024-03-16 with a completed Lessons Learned.
What went well
- This issue was self-identified from within Microsoft Corporation.
What didn't go well
- Microsoft PKI Services did not have a formal process to verify the files are DER-encoded.
Where we got lucky
-
Microsoft PKI Services did not have a formal process to verify the files are DER-encoded, but got lucky because we just happened to upload DER-encoded files in the past.
-
While RFC 5280 requires the files be DER encoded, there was very limited impact for Subscribers. Microsoft PKI Services are only aware of one service that noticed the discrepancy, which was from a Microsoft team.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Replace 8 old PEM encoded files with new DER encoded files at same AIA location | Mitigate | 2024-03-20 |
| Update AIA publishing process to check for DER encoding | Mitigate | TBD |
Appendix
Details of affected certificates
The 8 CA Certificates that were posted with a PEM encoded Certificate:
https://crt.sh/?id=9609730840, https://crt.sh/?id=9633426963, https://crt.sh/?id=9633426669, https://crt.sh/?id=9633426882, https://crt.sh/?id=9633426800, https://crt.sh/?id=9633426659, https://crt.sh/?id=9633426711, https://crt.sh/?id=9633426878.
These are the AIA URIs for the above certificates:
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2003%20-%20xsign.crt
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2004%20-%20xsign.crt
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2007%20-%20xsign.crt
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2008%20-%20xsign.crt
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2003%20-%20xsign.crt
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2004%20-%20xsign.crt
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2007%20-%20xsign.crt
- https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2008%20-%20xsign.crt
Impacted Requirements
-
CA/B Forum Baseline Requirements:
Section 7.1.2 Certificate Content and Extensions
“all certificates that it issues MUST comply with one of the following certificate profiles, which incorporate, and are derived from RFC 5280. Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.” -
RFC 5280:
Section 4.2.2.1. Authority Information Access
...
“Where the information is available via HTTP or FTP, accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in [RFC2585] or a collection of certificates in a BER or DER encoded "certs-only" CMS message as specified in [RFC2797].”
Based on Incident Reporting Template v. 2.0
|
Assignee | |
Updated•9 months ago
|
|
||
Updated•9 months ago
|
A couple of the cert that were linked in the report were issued by Digicert. I might be missing something but does this also make this a digicert incident?
|
||
Comment 2•9 months ago
|
||
Yes - Microsoft is both an independent CA and a sub-CA under Digicert. Although Microsoft issued the certificate, Digicert is the root owner. So the bug is both Digicert (root) and Microsoft (issuer). Microsoft posted the bug and details as the issue was wihtin their work flow instead of Digicert's.
I may be wrong, but from my understanding a separate bug is required for Digicert in this case too. The remediation items, timelines, etc may be different.
Maybe someone else more familiar with the incident reporting can confirm or deny this.
|
|
||
Comment 4•9 months ago
|
||
No - as Microsoft is a CA in the root program and only using Digicert for ubiquity, there wouldn't be a separate bug for Digicert. The bug would be super uninteresting as the root cause would be "Digicert issued a certificate to Microsoft".
|
||
Comment 5•9 months ago
|
||
I think the list of "affected certificates" is wrong. I would expect the affected certificates to be the certificates which contain a URL which pointed to a non-DER certificate, as those are the certificates which violated Section 4.2.2.1 of RFC 5280. Instead, Comment 0 lists the certificates which weren't DER-encoded, even though there's nothing wrong with the information in those certificates or how they were issued. So it's irrelevant that DigiCert issued some of them; the actually-affected certificates were all issued by Microsoft.
|
|
Assignee | |
Comment 6•9 months ago
|
||
Incident Report
Summary
Microsoft PKI Services discovered that 8 certificates published to the AIA repository were PEM encoded instead of DER encoded, which did not comply with RFC 5280 Section 4.2.2.1. These certificates were published to the AIA URIs by 2023-07-07. Our AIA publishing process did not detect that they were not DER encoded and MS PKI Services published them.
Impact
The only known impact was from one Microsoft service that implemented custom certificate chaining validation that expected DER formatted certificates in the AIA path. This is an issue with the format of the files in our file repository that AIA URIs point to.
We have not stopped issuance of any certificates because certificates are not malformed. We started the process to update the 8 CA certificate files referenced in the AIA. This deployment will use a safe deployment practice which takes 10 days to perform a staged deployment to minimize the risk that the change might impact existing customers.
Timeline
All times are UTC.
2023-06-27 – 2023-07-07: Microsoft PKI Services published the 8 CA certificates to the AIA file repository (listed in Appendix) that are the subject of this Incident.
2024-02-09 23:28: Internal Incident (x7168) with MS PKI Services Subscriber that identified an Internal Customer having issues with the certificates published to the AIA file repository. This was opened as a low priority incident that was not fully investigated immediately.
2024-03-07 16:27: Internal Incident (x4270) with Microsoft PKI Services team started to investigate AIA links being encoded in DER format.
2024-03-07 ~21:30: Microsoft PKI Services confirms that the 8 files we published were not DER encoded.
Root Cause Analysis
The root cause of this issue is Microsoft PKI Services missed this specific file encoding requirement from the RFC and did not have a validation step to ensure the files are DER encoded. It was a simple oversight that we will update in our publishing process to ensure we meet in the future.
Lessons Learned
The lesson we learned from this issue was reinforcement that we must automate every task no matter how frequently it is used. In addition to audits and self-assessments expected for any publicly trusted CA, we will strive to continue to mature every aspect of the service to perform more thorough reviews of design elements and processes to limit chances that these details are not missed.
What went well
- This issue was self-identified from within Microsoft Corporation.
What didn't go well
- Microsoft PKI Services did not have a formal process to verify the files are DER-encoded.
Where we got lucky
- Microsoft PKI Services did not have a formal process to verify the files are DER-encoded, but got lucky because we just happened to upload DER-encoded files in the past.
- While RFC 5280 requires the files be DER encoded, there was very limited impact for Subscribers. Microsoft PKI Services are only aware of one service in that 7+ month window that noticed the discrepancy, which was from a Microsoft team.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Replace 8 old PEM encoded files with new DER encoded files at same AIA location | Mitigate | 2024-03-20 |
| Update AIA publishing process to check for DER encoding | Mitigate | TBD |
Appendix
Details of affected certificates
All publicly trusted TLS certificates issued by the below CAs are impacted as they include these AIA URIs. These are the AIA URIs:
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2003%20-%20xsign.crt
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2004%20-%20xsign.crt
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2007%20-%20xsign.crt
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20ecc%20tls%20issuing%20ca%2008%20-%20xsign.crt
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2003%20-%20xsign.crt
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2004%20-%20xsign.crt
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2007%20-%20xsign.crt
https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2008%20-%20xsign.crt
Impacted Requirements
-
CA/B Forum Baseline Requirements:
Section 7.1.2 Certificate Content and Extensions
“all certificates that it issues MUST comply with one of the following certificate profiles, which incorporate, and are derived from RFC 5280. Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.” -
RFC 5280:
https://www.rfc-editor.org/rfc/rfc5280#section-4.2.2.1
Section 4.2.2.1. Authority Information Access
...
“Where the information is available via HTTP or FTP, accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in [RFC2585] or a collection of certificates in a BER or DER encoded "certs-only" CMS message as specified in [RFC2797].”
Based on Incident Reporting Template v. 2.0
|
||
Comment 7•8 months ago
|
||
Here is an update on this Incident.
We have completed one of the action items and are still working for a committed date on the other. We will update this bugzilla by end of day 2024-03-29 with a committed date on the other Action Item.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Replace 8 old PEM encoded files with new DER encoded files at same AIA location | Mitigate | 2024-03-20 - Completed |
| Update AIA publishing process to check for DER encoding | Mitigate | TBD |
|
|
||
Comment 8•8 months ago
|
||
Our team has completed all Repair Items related to this incident. With that work completed we respectfully request this incident be Resolved.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Replace 8 old PEM encoded files with new DER encoded files at same AIA location | Mitigate | 2024-03-20 - Completed |
| Update AIA publishing process to check for DER encoding | Mitigate | 2024-03-26 - Completed |
|
|
||
Comment 9•8 months ago
|
||
It appears that this incident has been addressed. Are there any objections to closing this on or about Friday, 5-Apr-2024?
|
|
||
Updated•8 months ago
|
|
|
||
Updated•6 months ago
|
Description
•