1962829 - Microsoft PKI Services: Policy document bug

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[u654666]

Preliminary Incident Report

Summary

• Incident description:
  Microsoft PKI Services made a change to a previous policy document that included a copy and paste typo that was missed until after the document had already been superseded, but still has active certificates related to that already superseded document. While reformatting the document to include various tables, a new detail was added that did not align with how we have been operating since inception. Specifically, CPS Version 3.2.4 incorrectly added that keyEncipherment is not present in Subscriber certificates even though it had always been present and continues to be present.

• Relevant policies:

  • Adherence to CPS
  • TLS Baseline Requirements Section 7.1.2.7.11 Subscriber Certificate Key Usage

• Source of incident disclosure:
  Certificate Problem Report from a third-party researcher.

Comment 1

[u654666]

Draft Incident Report

This draft incident report is an attempt to share as much information as possible while still working on collecting additional information for a full incident report. There are still missing details that we are actively working on identifying for the full incident report.

Summary

• CA Owner CCADB unique ID:
  A002577

• Incident description:
  This incident is related to a documentation problem with the CPS. There were no system or environmental changes made related to this bug.

  A previous version of the Public TLS CPS contained a typographical error that led to an incorrect statement regarding keyEncipherment in Subscriber certificates. Per the CA/B Forum Baseline Requirements (Section 7.1.2.7.11), keyEncipherment may be set for Subscriber certificates with RSA public keys but must not be set for Subscriber certificates with ECC public keys. However, during a document revision that introduced new tables, an inadvertent mistake stated that keyEncipherment is not present in Subscriber certificates with RSA public keys.

  This documentation mistake incorrectly stated that keyEncipherment was absent from Subscriber certificates with RSA public keys, despite it always having been present. A subsequent revision introduced an Appendix for Certificate Profiles, clarifying that keyEncipherment may be set, but it failed to explicitly state that keyEncipherment must not be set for Subscriber certificates with ECC public keys.

• Timeline summary:

  • Non-compliance start date:
    2024-07-01

  • Non-compliance identified date:
    2025-04-25

  • Non-compliance end date:
    2025-04-21

  • Relevant policies:
    TLS Baseline Requirements Section 7.1.2.7.11

  • Source of incident disclosure:
    A third-party researcher submitted a Certificate Problem Report directly to Microsoft PKI Services.

Impact

• Total number of certificates:
  This is a bug with CPS 3.2.4 introduced by a manual human typographical mistake during the document review change and review process.

• Total number of "remaining valid" certificates:
  This is a bug with CPS 3.2.4 introduced by a manual human typographical mistake during the document review change and review process.

• Affected certificate types:
  This is a bug with CPS 3.2.4 introduced by a manual human typographical mistake during the document review change and review process.

• Incident heuristic:
  This is a bug with CPS 3.2.4 introduced by a manual human typographical mistake during the document review change and review process.

• Was issuance stopped in response to this incident, and why or why not?:
  No. This is a bug with CPS 3.2.4 introduced by a manual human typographical mistake during the document review change and review process.

• Analysis:
  TBD

• Additional considerations:
  TBD

Timeline

• 2024-07-01 Public TLS CPS 3.2.4 published with new tables that included a typo in Section 7.1.2.7.11 stating keyEncipherment was not present in Subscriber certificates with RSA public keys even though this was being set at the time and continues to be set
• 2025-04-21 Public TLS CPS 3.3.0 published that replaced multiple tables with new Appendix B Certificate Profiles section where keyEncipherment may be set, but did not distinguish between ECC and RSA public keys
• 2025-04-25 Third-party researcher emailed a Certificate Problem Report to Microsoft PKI Services identifying mismatches between Subscriber certificates and CPS document language
• 2025-04-29 Public TLS CPS 3.3.1 published that retained language in Appendix B that keyEncipherment may be set, but did not distinguish between ECC and RSA public keys.

Related Incidents

TBD

Bug	Date	Description
[Related Bug ID](Related Bug URL)	Date Related Bug was opened	A description of how the subject Bug is related to the Bug referenced.

Root Cause Analysis

TBD

Contributing Factor 1: Policy documents are mostly written and reviewed by humans

• Description:
  Policy document management mostly relies on humans and when significant changes are made, a single detail is harder to identify. In this case, there was an effort to significantly improve the CPS to be more specific about certificate properties. This increased the document from 79 pages to 122 pages.

• Timeline:
  TBD

• Detection:
  TBD

• Interaction with other factors:
  TBD

• Root Cause Analysis methodology used:
  TBD

Lessons Learned

• What went well:
  TBD

• What didn’t go well:
  During a change that implemented a significant amount of text and new tables, multiple reviewers were unable to identify a mismatch between a setting defined in the CA environment and the value within the policy document.

• Where we got lucky:
  An updated version of the CPS, version 3.3.0, was a major rewrite of the document that used a different verification process for certificate properties that was previously used. When the tables in Section 7 were replaced with Certificate Profile details in Appendix B, keyEncipherment was correctly listed as may be set.

• Additional:
  TBD

Action Items

TBD

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Example	Prevent	Root Cause # 1	Criteria	2025-01-19	Value

Appendix

Relevant CPS Policy Documents:

• https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_CPS_v3.2.4.pdf
• https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_public_tls_CPS_v3.3.0.pdf
• https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_public_tls_CPS_v3.3.1.pdf

Comment 2

[Aaron Gable]

Despite this clearly being a bug in the policy document management process, similar historical incidents (e.g. Bug 1921573 "Let's Encrypt: No Meaningful Subject Distinguished Name" and Bug 1890896 "Entrust: CPS typographical (text placement) error") required the revocation of all affected certificates issued during the time that the document was incorrect. Does Microsoft plan to revoke all certificates issued with RSA public keys and the keyEncipherment EKU between 2024-07-01 and 2025-04-21?

Comment 3

[u654666]

(In reply to Aaron Gable from comment #2)

Despite this clearly being a bug in the policy document management process, similar historical incidents (e.g. Bug 1921573 "Let's Encrypt: No Meaningful Subject Distinguished Name" and Bug 1890896 "Entrust: CPS typographical (text placement) error") required the revocation of all affected certificates issued during the time that the document was incorrect. Does Microsoft plan to revoke all certificates issued with RSA public keys and the keyEncipherment EKU between 2024-07-01 and 2025-04-21?

Quick update. Microsoft PKI Services is still working on a response and will respond tomorrow.

Comment 4

[u654666]

(In reply to Aaron Gable from comment #2)

Despite this clearly being a bug in the policy document management process, similar historical incidents (e.g. Bug 1921573 "Let's Encrypt: No Meaningful Subject Distinguished Name" and Bug 1890896 "Entrust: CPS typographical (text placement) error") required the revocation of all affected certificates issued during the time that the document was incorrect. Does Microsoft plan to revoke all certificates issued with RSA public keys and the keyEncipherment EKU between 2024-07-01 and 2025-04-21?

Microsoft PKI Services investigated the revocation expectations further and created the following bug for failure to revoke certificates within 5 days:
https://bugzilla.mozilla.org/show_bug.cgi?id=1965612

Additional details regarding revocation will be provided in the new bug.

Comment 5

[u654666]

Full Incident Report

Summary

• CA Owner CCADB unique ID:
  A002577

• Incident description:
  This incident is related to a documentation problem with the CPS. There were no system or environmental changes made related to this bug.

  A previous version of the Public TLS CPS contained a typographical error that led to an incorrect statement regarding keyEncipherment in Subscriber certificates. Per the CA/B Forum Baseline Requirements (Section 7.1.2.7.11), keyEncipherment may be set for Subscriber certificates with RSA public keys but must not be set for Subscriber certificates with ECC public keys. However, during a document revision that introduced new tables, an inadvertent mistake stated that keyEncipherment is not present in Subscriber certificates with RSA public keys.

  This documentation mistakenly stated that keyEncipherment was absent from Subscriber certificates with RSA public keys, despite it always having been present. A subsequent revision introduced an Appendix for Certificate Profiles, clarifying that keyEncipherment may be set which mitigated the issue in the previous document. Worth noting is that the new update did not explicitly state that keyEncipherment must not be set for Subscriber certificates with ECC public keys which is not a compliance issue, but is a future change that will be made to avoid confusion.

• Timeline summary:

  • Non-compliance start date:
    2024-07-01

  • Non-compliance identified date:
    2025-04-25

  • Non-compliance end date:
    2025-04-21

  • Relevant policies:

    • TLS Baseline Requirements Section 7.1.2.7.11
    • TLS Baseline Requirements Section 4.9.1.1

  • Source of incident disclosure:
    A third-party researcher submitted a Certificate Problem Report directly to Microsoft PKI Services.

Impact

• Total number of certificates:
  103,469,764

• Total number of "remaining valid" certificates:
  ~77,890,368

• Affected certificate types:
  Organization Validated TLS Subscriber Certificates

• Incident heuristic:
  This incident includes all OV Subscriber certificates with RSA keys issued between 2024-07-01 and 2025-04-21.

• Was issuance stopped in response to this incident, and why or why not?:
  No. The problematic detail within the CPS had already been replaced in an updated version of the CPS before Microsoft PKI Services was aware of the incident.

• Analysis:
  Failure to revoke in time will be analyzed and tracked in the following bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1965612

• Additional considerations:
  Most Subscribers of the certificates issued by the CA require support for TLS 1.2, which requires keyEncipherment to be set as per RFC 5246: "keyEncipherment bit MUST be set if the key usage extension is present)." While this does not excuse the typographical mistake, it helps re-enforce that that this was a typo for a setting that was never planned to be changed.

Timeline

• 2024-07-01 Public TLS CPS 3.2.4 published with new tables that included a typo in Section 7.1.2.7.11 stating keyEncipherment was not present in Subscriber certificates with RSA public keys even though this was being set at the time and continues to be set
• 2025-04-21 Public TLS CPS 3.3.0 published that replaced multiple tables with new Appendix B Certificate Profiles section where keyEncipherment may be set, but did not distinguish between ECC and RSA public keys
• 2025-04-25 Third-party researcher emailed a Certificate Problem Report to Microsoft PKI Services identifying mismatches between Subscriber certificates and CPS document language
• 2025-04-29 Public TLS CPS 3.3.1 published that retained language in Appendix B that keyEncipherment may be set, but did not distinguish between ECC and RSA public keys.

Related Incidents

Bug	Date	Description
1962830	2025-04-25	The other bug was reported in the same Certificate Problem Report as this bug. The other bug also involves the CPS and certificate properties, but has a different root cause and resolution.
1965612	2025-05-08	Failure to revoke certificates within 5 days that were impacted by this bug

Root Cause Analysis

Contributing Factor 1: Policy documents are mostly written and reviewed by humans

• Description:
  Policy document management mostly relies on humans and when significant changes are made, a single detail is harder to identify. In this case, there was an effort to significantly improve the CPS to be more specific about certificate properties. This increased the document from 79 pages to 122 pages.

• Timeline:

  • 2024-07-01 Public TLS CPS 3.2.4 published with new tables that included a typo in Section 7.1.2.7.11 stating keyEncipherment was not present in Subscriber certificates with RSA public keys even though this was being set at the time and continues to be set
  • 2025-04-21 Public TLS CPS 3.3.0 published that replaced multiple tables with new Appendix B Certificate Profiles section where keyEncipherment may be set, but did not distinguish between ECC and RSA public keys
  • 2025-04-21 Public TLS CPS 3.3.0 published that replaced multiple tables with new Appendix B Certificate Profiles section where keyEncipherment may be set, but did not distinguish between ECC and RSA public keys

• Detection:
  Human review limitations are a known issue with document management.

• Interaction with other factors:
  N/A

• Root Cause Analysis methodology used:
  5-Whys

Lessons Learned

• What went well:
  Microsoft PKI Services was able to incorporate many other improvements to the CPS. Reviewers were able to catch and correct other mistakes before the CPS was published.

• What didn’t go well:
  During a change that implemented a significant amount of text and new tables, multiple reviewers were unable to identify a mismatch between a single setting defined in the CA environment and the value within the policy document.

• Where we got lucky:
  An updated version of the CPS, version 3.3.0, was a major rewrite of the document that used a different verification process for certificate properties that was previously used. When the tables in Section 7 were replaced with Certificate Profile details in Appendix B, keyEncipherment was correctly listed as may be set. This new document was published and mitigated the problem before Microsoft PKI Services was aware of the incident.

• Additional:
  None

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Increase number of CPS reviewers	Prevent	Root Cause # 1	Update document review process to include more reviewers from operational teams	TBD	In-Progress
Document version management improvements	Prevent	Root Cause # 1	Implement Git-based version control to replace Word documents managed in SharePoint	TBD	In-Progress
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause # 1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT be set when using an ECC public key	TBD	In-Progress

Appendix

Relevant CPS Policy Documents:

• https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_CPS_v3.2.4.pdf
• https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_public_tls_CPS_v3.3.0.pdf
• https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_public_tls_CPS_v3.3.1.pdf

Comment 6

[Wayne]

(In reply to Dustin Hollenback from comment #4)

(In reply to Aaron Gable from comment #2)

Despite this clearly being a bug in the policy document management process, similar historical incidents (e.g. Bug 1921573 "Let's Encrypt: No Meaningful Subject Distinguished Name" and Bug 1890896 "Entrust: CPS typographical (text placement) error") required the revocation of all affected certificates issued during the time that the document was incorrect. Does Microsoft plan to revoke all certificates issued with RSA public keys and the keyEncipherment EKU between 2024-07-01 and 2025-04-21?

Microsoft PKI Services investigated the revocation expectations further and created the following bug for failure to revoke certificates within 5 days:
https://bugzilla.mozilla.org/show_bug.cgi?id=1965612

Additional details regarding revocation will be provided in the new bug.

There seems to have been a question missed:

Does Microsoft plan to revoke all certificates issued with RSA public keys and the keyEncipherment EKU between 2024-07-01 and 2025-04-21?

Additionally did any further review of these policy documents occur after the certificate problem report? The issues I noticed were glancing at the document for a minute or two, and I'd wager I didn't find every issue comprehensively.

There is also the issue of a new process being made for 3.3.0, but no comparison made to what was in the previous document. This should have highlighted these issues internally. Are there any attempts to have document review between updated policy documents going forward? A move to git is mentioned, but nothing on the review process other than increasing number of reviewers when this seems to be a basic process step missed.

Comment 7

[u654666]

(In reply to Wayne from comment #6)

(In reply to Dustin Hollenback from comment #4)

(In reply to Aaron Gable from comment #2)

Despite this clearly being a bug in the policy document management process, similar historical incidents (e.g. Bug 1921573 "Let's Encrypt: No Meaningful Subject Distinguished Name" and Bug 1890896 "Entrust: CPS typographical (text placement) error") required the revocation of all affected certificates issued during the time that the document was incorrect. Does Microsoft plan to revoke all certificates issued with RSA public keys and the keyEncipherment EKU between 2024-07-01 and 2025-04-21?

Microsoft PKI Services investigated the revocation expectations further and created the following bug for failure to revoke certificates within 5 days:
https://bugzilla.mozilla.org/show_bug.cgi?id=1965612

Additional details regarding revocation will be provided in the new bug.

There seems to have been a question missed:

Does Microsoft plan to revoke all certificates issued with RSA public keys and the keyEncipherment EKU between 2024-07-01 and 2025-04-21?

We mistakenly implied that we were still working on the revocation question by opening 1965612, but did not clearly state that we have not yet finalized a revocation plan. Microsoft PKI Services acknowledges the requirements outlined in section 4.9.1.1 #12 of the Baseline Requirements, which mandate revocation of certificates that did not comply with the CPS at the time of issuance. We recognize that the five-day revocation window has already passed, and at this time, we have not finalized a plan for how to proceed with revocation.

We are currently evaluating next steps, taking into account the scale of the issue — over 103 million certificates issued that are impacted by this incident, with more than 77 million still active — and the potential impact on subscribers. While revocation is a requirement, we are being deliberate in our approach to ensure that any action taken is responsible and well-informed.

We are committed to transparency and will share further details about the revocation details, including our rationale and any decisions made, in the following bug as the investigation progresses: https://bugzilla.mozilla.org/show_bug.cgi?id=1965612

Additionally did any further review of these policy documents occur after the certificate problem report? The issues I noticed were glancing at the document for a minute or two, and I'd wager I didn't find every issue comprehensively.

Yes. Following the report, we conducted a focused review of the Certificate Profiles section in the CPS Appendix. While this review did not identify compliance issues, we did find an opportunity to clarify that the keyEncipherment EKU must not be used for ECC certificates, while remaining permissible for RSA certificates. This clarification will be included in a future CPS update and had been recorded as an action item.

There is also the issue of a new process being made for 3.3.0, but no comparison made to what was in the previous document. This should have highlighted these issues internally. Are there any attempts to have document review between updated policy documents going forward? A move to git is mentioned, but nothing on the review process other than increasing number of reviewers when this seems to be a basic process step missed.

Our standard document review process includes the use of Word with Track Changes enabled, allowing reviewers to clearly see all modifications. This process was followed for all versions of the CPS.

That said, version 3.3.0 introduced a significant structural change: the consolidation of certificate profile data into a new tabular format in Appendix B. This was intended to improve clarity and usability by presenting all relevant information in one place, rather than requiring readers to cross-reference multiple sections. Due to this shift from multiple locations in Section 7 to Appendix B, a direct line-by-line comparison with the previous version was not always feasible.

To ensure accuracy, the new appendix was developed by reviewing each field against actual certificate configurations and the Baseline Requirements. The focus was on ensuring the new representation was correct and complete, rather than identifying legacy issues in the prior format.

We recognize the importance of strengthening our document review processes, especially when structural changes are introduced. In addition to our existing cross-functional review—which includes individuals from across Microsoft beyond the Microsoft PKI Services CA team, we are expanding participation to include more Trusted Role personnel who are closely involved in operational processes.

We are also transitioning to Git-based document management. While this change is primarily intended to align with practices used by the CA/Browser Forum, it also brings benefits such as:

• Improved accountability through pull request-based reviews.
• Clearer assignment of responsibility for specific changes.
• Better traceability of feedback and approvals over time.

We believe these changes will help us catch issues earlier and improve the overall quality and transparency of our documentation process.

If there are additional practices or tools the community has found effective in similar situations, we would welcome that input. Our goal is to continuously improve in alignment with community expectations.

Comment 8

[chrome-root-program]

Due to the number of certificates impacted by this incident, we feel it’s worthwhile to reiterate Chrome’s stance on public incident reporting, as detailed in our policy. That is, we evaluate all incidents on a case-by-case basis and consistently point to the factors significant to our program when evaluating the response, which include (but are not limited to):

• a demonstration of understanding of the root causes of an incident,
• a substantive commitment and timeline to changes that clearly and persuasively address the root cause,
• past history by the Chrome Root Program Participant in its incident handling and its follow through on commitments, and,
• the severity of the security impact of the incident.

If and when revocation is delayed, a subsequent incident report needs to be opened (as Microsoft PKI Services has done in 1965612). As said in the past, we will again emphasize that, alone, delayed revocation of certificates is not cause for enforcement action, and that we always consider the wider context and the factors significant to the Chrome Root Program before taking such actions. We will continue to evaluate this incident as more information becomes available and we are keen to see a substantiative commitment by Microsoft PKI Services to changes that clearly and persuasively address the root cause of this incident, consistent with our view of all incidents disclosed to Bugzilla.

Questions:

(1) The “Related Incidents” section should be improved, as it does not accomplish the intended goal as described on CCADB.org. This has been clarified a few times (e.g., 1959733 and 1951415) since the updated CCADB Incident Reporting Guidelines took effect. We consider this review helpful for CA Owners and the broader community to better understand how the ecosystem has responded to prior incidents with the same or similar root cause(s).

Can you help us understand how Microsoft PKI Services is following reports filed to Bugzilla and applying lessons learned to its own policies and practices?

(2) Was Microsoft PKI Services familiar with the incidents described in Comment 2? We’re trying to understand why the revocation expectations described in Comment 4 were not readily apparent to Microsoft PKI Services when responding to this incident.

(3) The “Action Items” table in Comment 5 describes increasing the number of human policy document reviewers, changing the technical approach for managing policy documents (from SharePoint to Git), and an update to the Microsoft CP to clarify the intended use of keyEncipherment. The “Root Cause Analysis” section describes a single contributing factor - “Policy documents are mostly written and reviewed by humans.”

How do the changes meaningfully reduce likelihood of recurrence or contribute to the detection of other issues that might already exist, as also described in Comment 6? Comment 7 seems to suggest that the change in profile presentation format across policy versions made it difficult to detect the preexisting policy issue. Any future change of profile formatting seems to risk re-introducing similar issues as described in the transitions from CPS Version 3.2.3 -> 3.2.4, and Version 3.2.4 -> 3.3.0 (i.e., creating circumstances that could be confusing for humans, and where unintended changes are not readily apparent via a Git diff).

Did Microsoft consider any other technical controls as part of its Action Items? If so, can you share why they weren't ultimately adopted?

(4) Can you describe the motivation for changing the certificate profile presentation format when comparing CPS Versions 3.2.3 and 3.2.4 (i.e., where this issue was introduced)? In general, it seems the change was intended to offer better alignment with TLS BR 2.0, which landed approximately 15 months earlier (April 11, 2023).

(5) Has Microsoft imagined, through automation, translating its CP and CPS into custom lints to be incorporated in an already in-use linting solution? Is it possible such a solution might have detected this issue prior to CPS 3.2.4 being implemented? From 1674561 we understand ZLint was once in use - but it seems all open-source linters can be extended to support CA-specific lints.

(6) Looking at a sample TLS server authentication certificate, we see the following extensions:

• Microsoft Certificate Template (1.3.6.1.4.1.311.21.7)
• Microsoft Application Policies (1.3.6.1.4.1.311.21.10)

Can you help us understand use of these extensions considering TLS BR Section 7.1.2.11.5 (“Other Extensions”) - Bullet 1 (“MUST apply in the context of the public Internet, unless: the extension OID falls within an OID arc for which the Applicant demonstrates ownership, or, the Applicant can otherwise demonstrate the right to assert the data in a public context.”)?

(7) We randomly extracted 1,000,000 distinct non-wildcard dnsNames contained in time-valid and unrevoked certificates validating to the Microsoft Roots included in the Chrome Root Store (i.e,. 1 and 2). We then attempted to connect to those domains using OpenSSL (s_client -connect).

Of those 1,000,000 domains, only 2.9% connected successfully. This rate appears to be extremely low for publicly-trusted TLS certificates. About all but 700 served certificates issued by one of the Azure Issuing CAs. Additional internal tooling indicates only 8% of time-valid certificates issued by Microsoft have ever been observed in use in the wild by our monitoring infrastructure - rather than being observable at the time of our most recent s_client evaluation.

Sampling some the ~30,000 dnsNames that did connect successfully, we see names like:

• https://bst-b01b6441-b0e2-48b9-91eb-4f1bab2f70b4.bastion.azure.com/
• https://my-apim-useast.data.azure-api.net/
• https://all-test-all-gen-apim-uks-001.management.azure-api.net/
• https://playapi.azure-api.net/
• https://gw-api-coris.azure-api.net/
• https://k8bridge.eus.dogfood.fusion.azure-test.net/
• https://waws-prod-bay-155.publish.azurewebsites.windows.net/
• https://jit-soak-0-7edbd6ebd72a33f3.azure-api.net/
• https://apim-a-aoai-1d-ms-llm-manual-01.azure-api.net
• https://apieversaasdev.azure-api.net/

General themes observed include a significant number of names that contain “azure-api.net” (18,461) or “apim-” (6,142). Some names appear to describe the location of the corresponding server (e.g., ”eastus”, “westsurope”, “uksouth”, “australiaeast”, etc.). About 4,000 include “bastion”.

Could you help us understand or better interpret these observations, both the low observed use and whether the use cases observed (including API front-ends and bastion hosts) are consistent with those of a publicly-trusted CA (versus being a better fit for an enterprise CA)?

(8) The Azure ICAs issuing certificates related to this incident also appear cross-certified by DigiCert (e.g., 3 and 4). Due to the existence of these cross-certificates, can you describe the degree of oversight DigiCert maintains over Microsoft’s policies and practices that could have contributed to detecting this issue?

Comment 9

[Microsoft PKI Services]

Response to Comment 8

---

(1) Related Incidents

“Can you help us understand how Microsoft PKI Services is following reports filed to Bugzilla and applying lessons learned...?”

Microsoft has a formal process through which we manage changes related to CAB Forum and Trusted Root program requirements changes. We have formal processes to track our own Bugzilla bugs, but ad hoc process for tracking and learning from Bugzilla bugs from other CA's. We have taken an action to add a formal process to review and learn from all Bugzilla bugs to our repair items.

---

(2) Revocation Expectations

“Why were revocation expectations not readily apparent when responding to this incident?”

Considering that the certificates complied with all the baseline requirements, and that the issue was related to a typo in the CPS document, we were anticipating that a revocation would not be required. However, as stated in Comment 4, after assessing the requirements more closely, we opened a new bug to track that issue.

---

(3) Action Items and Recurrence Risk

“How do the changes reduce recurrence or help detect other issues? Did Microsoft consider any other technical controls as part of its Action Items? If so, can you share why they weren't ultimately adopted?”

We are actively exploring ideas to strengthen our process, including extending our linting to ensure consistency between our CPS and the certificates (see response to question 5 in this Comment). We welcome any additional ideas that would help.

---

(4) Profile Format Change Motivation

“Can you describe the motivation for changing the certificate profile presentation format when comparing CPS Versions 3.2.3 and 3.2.4 (i.e., where this issue was introduced)?”

Yes, the change was intended to improve alignment with TLS BR 2.0 and ensure our documentation reflects evolving baseline requirements.

---

(5) Custom Linting and Automation

“Has Microsoft imagined, through automation, translating its CP and CPS into custom lints to be incorporated in an already in-use linting solution? Is it possible such a solution might have detected this issue prior to CPS 3.2.4 being implemented? From 1674561 we understand ZLint was once in use - but it seems all open-source linters can be extended to support CA-specific lints.”

We use a combination of automated linting tools, including ZLint and custom linting to support certificate compliance against Baseline Requirements. We have work already started to ensure consistency between our CPS and our linting rules. Further, we have a repair item to move our CPS to markdown, which will pave the path for us to automate this consistency check in the future.

---

(6) Use of Microsoft-Specific Extensions

“Can you explain the use of these extensions under TLS BR 7.1.2.11.5?”

The OIDs referenced are part of Microsoft’s private arc, as noted in IANA reference: Private Enterprise Numbers (PENs). The named extensions are in the CP/CPS, and we will take an action item upon the next update to include the OIDs in our policies.

---

(7) Use Case Interpretation

“Are observed use cases consistent with a publicly-trusted CA?”

The services mentioned here are the Azure Bastion service (bastion.azure.com), the Azure API Management service (azure-api.net), and the Azure App Service (azurewebsites.net).
Regarding specific services, many Microsoft customers choose to use the Azure Virtual Network feature to secure their communication to Azure services. For services such as Azure API Management and Azure Bastion, it is common for customers to use Azure Virtual Network to control access to their API endpoints, thereby limiting the availability of these endpoints to the Internet at large. These services employ publicly trusted certificates as they generally serve not just a single enterprise but large multi-national corporations and governments with various organizations and clients.
For instance, a service deployed across multiple Azure regions, each with its own availability zones, may necessitate hundreds of certificates for the same subject name, with each instance possessing a unique private key. Additionally, if these certificates are renewed every 90 days, the number of time-valid versions of a certificate quickly escalates to thousands.

---

(8) DigiCert Oversight

“The Azure ICAs issuing certificates related to this incident also appear cross-certified by DigiCert (e.g., 3 and 4). Due to the existence of these cross-certificates, can you describe the degree of oversight DigiCert maintains over Microsoft’s policies and practices that could have contributed to detecting this issue?”

Microsoft is an independent Root CA. We are responsible for our own controls, detection, and compliance with all baseline requirements. We are accountable for our own policy updates, audits, and operations. Our full incident report in Comment 5 addresses the improvements we are making to our detection procedures. However, as part of our contractual obligations, Microsoft shares updates to our policies and audit reports with DigiCert.

Comment 10

[DigiCert]

Microsoft is an independent root CA that is trusted by all modern agents, including both Chrome and Mozilla. The cross-sign is primarily used for interoperability with important legacy platforms (e.g., iOS 8). Microsoft is accountable for its own CPS updates, its own audits, and its own operations. As a cross-signing authority, DigiCert requires, via contract, compliance with both the CA/B Forum requirements and root program requirements. We regularly communicate with all operators of cross-signed CAs and review annual WebTrust audits to verify compliance.

Comment 11

[Microsoft PKI Services]

Response to Comment 9

Per our response in Comment 9, here are the action items we are committing to:

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalize Bugzilla bug learnings process	Prevent	Root Cause #1	Formalize process to review and incorporate learnings from Bugzilla bugs from other CAs into our practices.	7/15/2025	In-Progress
Update next CP/CPS with Microsoft OID	Mitigate	Root Cause #1	Include extension and OID information from our Microsoft private Arc as identified in Comment 8 in our next public TLS CP/CPS update.	6/13/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Mitigate	Root Cause #1	Perform a comprehensive review of linting rules against certificate profiles defined in our CP/CPS to identify and address discrepancies.	6/13/2025	In-Progress
Convert CPS into Markdown	Prevent	Root Cause #1	Transition CPS documentation to Markdown format to improve maintainability and consistency.	9/1/2025	New
Ensure consistency between our CP/CPS and our linting mechanism	Prevent	Root Cause #1	Implement fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.	TBD	New

Comment 12

[Microsoft PKI Services]

Update to Comment 5

We have updated the due dates for the action items mentioned in Comment 5.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Increase number of CPS reviewers	Prevent	Root Cause # 1	Update document review process to include more reviewers from operational teams	6/13/2025	In-Progress
Document version management improvements	Prevent	Root Cause # 1	Implement Git-based version control to replace Word documents managed in SharePoint	6/13/2025	In-Progress
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause # 1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT be set when using ECC	6/13/2025	In-Progress

Comment 13

[chrome-root-program]

[Posting in response to Comment 9.]

Thank you for the additional detail. A few follow-up comments and questions are listed below.

(1) Comment 9 includes: “We have formal processes to track our own Bugzilla bugs, but ad hoc process for tracking and learning from Bugzilla bugs from other CA's.”

Can you help us understand what this means in practice? For example, can you describe the frequency and scope of the historical “ad hoc” reviews? Understanding the current state helps inform the impact of the Action Items proposed in response to this incident report.

(2) The corresponding action in Comment 11 lists: “Formalize process to review and incorporate learnings from Bugzilla bugs from other CAs into our practices.” as the Evaluation Criteria.

CCADB.org describes “Evaluation Criteria” as “how the CA Owner will measure the effectiveness of the Action Item in addressing the Root Cause. Include how the public can also measure this impact, if applicable.”. The incorporation of evaluation criteria into the IRG template was intended to help promote measurable and observable improvements in response to incidents. Even when criteria is not publicly measurable, it should be valuable for Microsoft PKI Services’ team members and its auditors (and other CAs monitoring and learning from this disclosure), fostering shared expectations and as part of an ongoing commitment to continuous improvement.

The “Evaluation Criteria” should be updated during the next planned update to the Action Items table.

(3) Related to the Action Item described above, can you share your initial thoughts on what the “formalized process” will look like? This represents an opportunity for CA Owners to share best practices as part of the ecosystem’s continuous improvement.

There have been some useful descriptions of some CA Owner practices re: their Bugilla review process, including here, which describes a twice-a-week review meeting.

(4) Our framing of the question “Why were revocation expectations not readily apparent when responding to this incident?” was in attempt to understand Microsoft’s familiarity with the prior incidents described in Comment 2. Given the description of the “ad hoc” incident review process in response to Comment 8 Question 1, we will assume the answer to our question “Was Microsoft PKI Services familiar with the incidents described in Comment 2? was “No.” Please correct this if you feel otherwise.

(5) We asked “How do the changes meaningfully reduce likelihood of recurrence or contribute to the detection of other issues that might already exist?” but we do not see a direct response. This question was posed to help consider the efficacy of the Action Items and to help consider alternatives.

Similarly, we do not see an answer to “Did Microsoft consider any other technical controls as part of its Action Items? If so, can you share why they weren't ultimately adopted?”

Can you please address these points in a subsequent update?

(6) Related to actions Microsoft PKI Services is taking in response to this incident, has it considered combining its CP and CPS into a single document (i.e., a combined CP/CPS) as a method for promoting simplicity and reducing the corresponding maintenance activities of using both a CP and CPS? From our view, the management of two similar (i.e., overlapping and in many cases, redundant) policy documents adds complexity and surface area for risk without much value.

(7) Our questions about the “Microsoft Certificate Template” and “Microsoft Application Policies” extensions were intended to improve our understanding re: what these OIDs are, and how they are expected to be relied upon by relying parties and user agents. Can you please share more information re: the second part (i.e., how they are expected to be relied upon)?

(8) Related to the above and given the BR reference in our question (from Section 7.1.2.11.5), can you help us better understand how the applicants can “demonstrate the right to assert the data in a public context” when asserting these extensions? Or, even more generally, how these extensions meet the 1.a or 1.b criteria detailed in Section 7.1.2.11.5?

(9) Thank you for the pointers re: Microsoft Services described in Comment 8. We were instead hoping to hear from you re: why these use cases are a better fit for the Web PKI rather than being supported by an enterprise PKI.

If availability to these endpoints is limited to a specific community or user population, is that indicative of a “public” use case, especially considering the framing of the TLS BRs which includes statements like “These Requirements only address Certificates intended to be used for authenticating servers accessible through the Internet.” (Section 1.1)?”

Our interest in answers to this question and the next largely stems from our monitoring tools’ limited observability of time-valid leafs issued by the Microsoft PKI Services CA hierarchies included in the Chrome Root Store, which is incidentally over 200 times less than the average of all CA Owners included in the Chrome Root Store.

(10) Beyond the interpreted challenges with root store management due to cross-organizational trust, can you share what other challenges would exist if these use cases were served from a non-publicly trusted PKI hierarchy?

Comment 14

[chrome-root-program]

[Posting in response to Comment 10.]

Thank you for your participation in this discussion!

Much of the framing for our questions is from the following statement included in the Chrome Root Program Policy:

“Chrome Root Program Participants MUST satisfy the requirements defined in this policy, including taking responsibility for ensuring the continued compliance of all corresponding subordinate CAs and delegated third parties participating in the PKI.”

(1) DigiCert noted that “Microsoft is an independent root CA that is trusted by all modern agents, including both Chrome and Mozilla”.

We’ll note that in practice, due to path building logic in Chrome, paths from Microsoft-issued leafs (example) validate to a DigiCert root, despite a path also existing to a Microsoft root. Without speaking for other browsers’ validation logic, this seems to be true in Safari, Firefox, and Edge.

Could you clarify the relevance of Microsoft's independent trust status to DigiCert's role and responsibilities concerning the cross-certificates it has issued to Microsoft's ICAs given the policy snippet above and how paths are built in practice?

(2) Does DigiCert alter its standard oversight for Microsoft's cross-certified ICAs because Microsoft PKI Services itself is a publicly-trusted Root CA Owner, or does it treat them like any other externally-operated CA it cross-signs?

(3) We’d like to better understand how DigiCert treats the externally-operated CAs it cross-signs given the statement it: “review[s] annual WebTrust audits to verify compliance”.

Beyond reviewing annual audits, what other mechanisms, if any, does DigiCert employ to ensure ongoing compliance of externally-operated, cross-signed CAs with CA/B Forum and root program requirements throughout the year? For example, does DigiCert require official notification from externally-operated CAs in advance of planned policy changes (example) and if so, are those changes reviewed, etc.?

Comment 15

[DigiCert]

(1) DigiCert noted that “Microsoft is an independent root CA that is trusted by all modern agents, including both Chrome and Mozilla”.
We’ll note that in practice, due to path building logic in Chrome, paths from Microsoft-issued leafs (example) validate to a DigiCert root, despite a path also existing to a Microsoft root. Without speaking for other browsers’ validation logic, this seems to be true in Safari, Firefox, and Edge.
Could you clarify the relevance of Microsoft's independent trust status to DigiCert's role and responsibilities concerning the cross-certificates it has issued to Microsoft's ICAs given the policy snippet above and how paths are built in practice?

We don’t have metrics on the pathbuilding used in the browsers or the percentage use of DigiCert vs. Microsoft roots. Because they are on prem CAs with Microsoft, the certificates can be served from the MS root independent of Digicert or using the Digicert chain without notice to Digicert and without our ability to detect how the certs are chaining. The independent status is that they went through the root embedment process, upload their own audits, and participate in the CABF as an independent CA. We maintain oversight, but Microsoft operates both their own root and the Sub CAs within the same ecosystem as approved during root inclusion.

(2) Does DigiCert alter its standard oversight for Microsoft's cross-certified ICAs because Microsoft PKI Services itself is a publicly-trusted Root CA Owner, or does it treat them like any other externally-operated CA it cross-signs?

It may be relevant to understand that DigiCert’s policy only permits 4 companies to obtain cross-signs or sub CAs from DigiCert: Apple, Microsoft, Google, and Mozilla (DigiCert has already shut down all other external subCAs that operated under our roots).
Two of those companies operate signed CAs. We treat both Apple and Microsoft similarly with similar processes. Microsoft can (and should) upload its own files to CCADB.

(3) We’d like to better understand how DigiCert treats the externally-operated CAs it cross-signs given the statement it: “review[s] annual WebTrust audits to verify compliance”.
 
Beyond reviewing annual audits, what other mechanisms, if any, does DigiCert employ to ensure ongoing compliance of externally-operated, cross-signed CAs with CA/B Forum and root program requirements throughout the year? For example, does DigiCert require official notification from externally-operated CAs in advance of planned policy changes (example) and if so, are those changes reviewed, etc.?

We meet with these companies every two to four weeks (depending on schedules). We regularly discuss CABF changes and updates. We coordinate with them on how changes in both root and CABF policy will impact their root operations. In addition to reviewing their annual audits, we require both companies to provide us with a copy of their annual root program assessment, notify us of compliance issues, and notify us before policy changes are effective so that we can review their documents to ensure compliance with the BRs. The certificates are not logged to our database or run through our compliance tools as the certificates are issued outside of our infrastructure. To mitigate the risk of non-compliance, we require that these companies use pkilint as a pre-issuance linter, and keep the linter up to date.

Microsoft’s CPS and certificates were issued in accordance with the BRs but not in accordance with the Microsoft CPS. Although we monitor third party issuance of problem certificates, we currently do not have an automated way to ensure that a third party CPS that is more narrow than the BRs is always in sync with the sub CA’s practices.

We are currently exploring how some of the internal changes we are making at DigiCert in terms of mandatory machine-readable profiles for both DigiCert and Sub CAs, combined with automated profile generation and enforcement, could be applied in general. We would appreciate community ideas on how to apply adequate controls over a SubCA that is also embedded in root programs.

Comment 16

[Microsoft PKI Services]

Weekly Status Update

---

We are actively progressing through all repair items identified in the incident report. All action items are currently in progress, with the exception of the implementation phase for identified linting gaps, which will begin following completion of the gap identification. We remain on track to meet the expected due dates outlined in the full report.

In addition we are committing to an additional action item (see last item) and updating the evaluation criteria based on Comment 13 - Question 2 (see first item):

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalize Bugzilla bug learnings process	Prevent	Root Cause #1	Effectiveness will be measured by adherence to a regular review schedule, tracking of each Bugzilla issue reviewed, review outcomes, and integration of those outcomes into our engineering rhythm in the form of work items.	7/15/2025	In-Progress
Update next CP/CPS with Microsoft OID	Mitigate	Root Cause #1	Include extension and OID information from our Microsoft private Arc as identified in Comment 8 in our next public TLS CP/CPS update.	6/13/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Mitigate	Root Cause #1	Perform a comprehensive review of linting rules against certificate profiles defined in our CP/CPS to identify and address discrepancies.	6/13/2025	In-Progress
Convert CPS into Markdown	Prevent	Root Cause #1	Transition CPS documentation to Markdown format to improve maintainability and consistency.	9/1/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Prevent	Root Cause #1	Implement fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.	TBD	In-Progress
Increase number of CPS reviewers	Prevent	Root Cause #1	Update document review process to include more reviewers from operational teams	6/13/2025	In-Progress
Document version management improvements	Prevent	Root Cause #1	Implement Git-based version control to replace Word documents managed in SharePoint.	6/13/2025	In-Progress
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause #1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT be set when using ECC.	6/13/2025	In-Progress
Evaluate CPS-to-lint translation to codify enforceable pre-issuance checks	Prevent	Root Cause #1	Effectiveness will be measured by the successful mapping CPS control to a lint rule along with an implementation plan for broader CPS rule coverage.	6/30/2025	New

For updates or questions specifically related to the revocation of impacted certificates, please refer to Bug 1965612.

In addition we would like to correct the non-compliance start date that was presented in the Full incident report under Comment 5. The correct start date is:
Non-compliance start date: 2024-07-21

Comment 17

[Microsoft PKI Services]

Response to Comment 13

---

(1) Historical Bugzilla Review Practices

“Comment 9 includes: “We have formal processes to track our own Bugzilla bugs, but ad hoc process for tracking and learning from Bugzilla bugs from other CA's.”
Can you help us understand what this means in practice? For example, can you describe the frequency and scope of the historical “ad hoc” reviews? Understanding the current state helps inform the impact of the Action Items proposed in response to this incident report.?”

Prior to this incident, Microsoft PKI Services did not have a formal, recurring process to review incident reports from other CAs published in Bugzilla. Instead, incident reviews occurred on an ad hoc basis, typically in response to:
• High-profile incidents flagged by members of our compliance or governance teams,
• Industry discussions in CA/Browser Forum working groups,
• Incidents with similar characteristics to our own infrastructure or practices, or
• Preparation for audits, root program reviews, or CP/CPS updates.

Though we have a robust, recurring process to review, analyze and manage changes to industry/compliance requirements; and also have a robust incident management process for our own bugs, we had not included a formal review of Bugzilla bugs at large into these processes. As part of the corrective actions from this incident (see Action Item #13 in the full report), we will include review of new Bugzilla bugs to our standing industry/compliance change control meetings which meet weekly.

---

(2) Evaluation Criteria for Bugzilla Review

“The corresponding action in Comment 11 lists: “Formalize process to review and incorporate learnings from Bugzilla bugs from other CAs into our practices.” as the Evaluation Criteria.
CCADB.org describes “Evaluation Criteria” as “how the CA Owner will measure the effectiveness of the Action Item in addressing the Root Cause. Include how the public can also measure this impact, if applicable.”. The incorporation of evaluation criteria into the IRG template was intended to help promote measurable and observable improvements in response to incidents. Even when criteria is not publicly measurable, it should be valuable for Microsoft PKI Services’ team members and its auditors (and other CAs monitoring and learning from this disclosure), fostering shared expectations and as part of an ongoing commitment to continuous improvement.
The “Evaluation Criteria” should be updated during the next planned update to the Action Items table.”

We have updated the Evaluation Criteria in Comment 16 Action Items table to define clear metrics for success. For the Bugzilla bugs at large, we will include criteria such as: adherence to a regular review schedule, tracking of each Bugzilla issue reviewed, review outcomes, and integration of those outcomes into our engineering rhythm in the form of work items. These metrics will be internally tracked and available to auditors.

---

(3) Formalized Bugzilla Review Process

“Related to the Action Item described above, can you share your initial thoughts on what the “formalized process” will look like? This represents an opportunity for CA Owners to share best practices as part of the ecosystem’s continuous improvement.
There have been some useful descriptions of some CA Owner practices re: their Bugilla review process, including here, which describes a twice-a-week review meeting.?”

We are in the process of formalizing a structured review process for external Bugzilla incident reports. As outlined in the previous comment, we will be adding a review of Bugzilla bugs at large to our weekly industry/compliance change control processes.

---

(4) Familiarity with Prior Incidents

“Our framing of the question “Why were revocation expectations not readily apparent when responding to this incident?” was in attempt to understand Microsoft’s familiarity with the prior incidents described in Comment 2. Given the description of the “ad hoc” incident review process in response to Comment 8 Question 1, we will assume the answer to our question “Was Microsoft PKI Services familiar with the incidents described in Comment 2? was “No.” Please correct this if you feel otherwise.”

Microsoft PKI Services was aware of the prior incidents referenced in Comment 2. Our delay in revocation was not due to lack of familiarity, but rather due to the technical constraints related to the exceptional scale involved in this case. Immediate revocation of tens of millions of certificates would have resulted in unmanageable CRL sizes which negatively affect client-side validation.

---

(5) Efficacy of Action Items and Consideration of Alternatives

“We asked “How do the changes meaningfully reduce likelihood of recurrence or contribute to the detection of other issues that might already exist?” but we do not see a direct response. This question was posed to help consider the efficacy of the Action Items and to help consider alternatives.
Similarly, we do not see an answer to “Did Microsoft consider any other technical controls as part of its Action Items? If so, can you share why they weren't ultimately adopted?”
Can you please address these points in a subsequent update?”

In regards to your first question, our action items are focused on both preventing recurrence and improving early detection of discrepancies between policy and practice. Examples of the action items we are committing to:
• Formalizing the Bugzilla review process enables systematic learning from ecosystem incidents.
• Migrating the CPS to markdown and Git supports structured version control and effective diffing, making policy changes more transparent and auditable.
Together, these steps reduce the likelihood of undetected misalignment and strengthen our controls against future non-compliance.

For the second question, we are exploring CPS-to-lint translation, which would allow us to codify CPS requirements as enforceable pre-issuance checks. While we have not yet committed to a specific implementation timeline, it is under evaluation alongside other technical enhancements.

---

(6) Combined CP/CPS Document

“Related to actions Microsoft PKI Services is taking in response to this incident, has it considered combining its CP and CPS into a single document (i.e., a combined CP/CPS) as a method for promoting simplicity and reducing the corresponding maintenance activities of using both a CP and CPS? From our view, the management of two similar (i.e., overlapping and in many cases, redundant) policy documents adds complexity and surface area for risk without much value.”

Thank you for the suggestion. Microsoft PKI Services has considered the benefits of a combined CP/CPS structure but have decided against it because our Certificate Policy (CP) encompasses a broader range of certificate types beyond public TLS (such as code signing); with separate CPS’s for each use case (public TLS being one).

---

(7) & (8) Use of Microsoft-Specific Extensions

“Our questions about the “Microsoft Certificate Template” and “Microsoft Application Policies” extensions were intended to improve our understanding re: what these OIDs are, and how they are expected to be relied upon by relying parties and user agents. Can you please share more information re: the second part (i.e., how they are expected to be relied upon)?
(8) Related to the above and given the BR reference in our question (from Section 7.1.2.11.5), can you help us better understand how the applicants can “demonstrate the right to assert the data in a public context” when asserting these extensions? Or, even more generally, how these extensions meet the 1.a or 1.b criteria detailed in Section 7.1.2.11.5?”

Thank you for raising this. In response to question 7 and 8 regarding Microsoft-specific extensions in question—Certificate Template (1.3.6.1.4.1.311.21.7) and Application Policies (1.3.6.1.4.1.311.21.10)—are asserted under Microsoft’s private enterprise arc (1.3.6.1.4.1.311), for which Microsoft is the IANA-registered owner.
All publicly trusted TLS certificates issued by Microsoft PKI Services are provided to internal Microsoft entities, and all certificate requests are authenticated via Microsoft identity systems. As such, Microsoft, acting both as CA and Applicant, demonstrates ownership of the OID arc used in these extensions. This aligns with BR 7.1.2.11.5 clause 1.a, which permits extension OIDs when the Applicant demonstrates ownership of the arc.
We recognize that these extensions are not intended for interpretation by external relying parties. Their inclusion supports Microsoft service compatibility and legacy infrastructure behavior, particularly for certificate lifecycle management and internal enforcement policies.

---

(9) Justification for Use of Web PKI

“Thank you for the pointers re: Microsoft Services described in Comment 8. We were instead hoping to hear from you re: why these use cases are a better fit for the Web PKI rather than being supported by an enterprise PKI.
If availability to these endpoints is limited to a specific community or user population, is that indicative of a “public” use case, especially considering the framing of the TLS BRs which includes statements like “These Requirements only address Certificates intended to be used for authenticating servers accessible through the Internet.” (Section 1.1)?”
Our interest in answers to this question and the next largely stems from our monitoring tools’ limited observability of time-valid leafs issued by the Microsoft PKI Services CA hierarchies included in the Chrome Root Store, which is incidentally over 200 times less than the average of all CA Owners included in the Chrome Root Store.”

Thank you for the follow up. While some Microsoft services use access controls like virtual networks or private endpoints, many such as Azure API Management, Bastion, and App Services, are designed for multi-tenant, public-facing scenarios. These services often support external clients, partners, and unmanaged systems, making publicly trusted certificates essential for interoperability.
Web PKI enables seamless trust across browsers, operating systems, and third-party environments without custom configuration. Given the global reach and external use cases of these services, we believe they meet the intent of BR 1.1 regarding public Internet accessibility.
That said, we continue to evaluate where internal PKI might be appropriate for non-public workloads as part of our long-term PKI strategy.

---

(10) Challenges with Using Private PKI

“Beyond the interpreted challenges with root store management due to cross-organizational trust, can you share what other challenges would exist if these use cases were served from a non-publicly trusted PKI hierarchy?”

Using a non-publicly trusted PKI would create challenges beyond just root store management. While distributing trust anchors is one concern, these services are accessed by external organizations and unmanaged clients, where Microsoft does not control the connecting environments.
Even with anchor distribution, a private PKI would introduce significant operational friction, breaking compatibility with browsers, APIs, and client software that expect publicly trusted certificates. This would increase support complexity and reduce adoption of these services in customer-facing or cross-organization scenarios. Public trust enables these services to operate at scale, with minimal configuration, and remains necessary for many of the workloads we support today.

Comment 18

[Zacharias]

Public trust enables these services to operate at scale, with minimal configuration, and remains necessary for many of the workloads we support today.

Yes, this is correct. There are many benefits provided by public trust, such as being able to trust that mississued certificates are revoked in a timely fashion. Hey, wait a minute!! Does Microsoft PKI understand that the benefits for subscribers of public trust aren’t free and without commitments?

Regarding the CRL bloat issue, I’m far from an expert in webPKI but I’m wondering if there is a point where it’s more appropriate to revoke an intermediaries instead of leaf certificates even if this would come with collateral damage. I really don’t think that ”we have mississued so many certificates that we cannot revoke them” acceptable, perhaps others have deeper insight? Is this covered in the new requirement regarding mass-revocation from Mozilla? Maybe it’s time for a similar requirement, but instead to demonstrate how a CA is monitoring bugzilla and learning from incidents happening for other CAs?

I welcome the requirement for ha

Comment 19

[Microsoft PKI Services]

Response to Comment 18

Public Trust and Revocation Expectations

Yes, this is correct. There are many benefits provided by public trust, such as being able to trust that mississued certificates are revoked in a timely fashion. Hey, wait a minute!! Does Microsoft PKI understand that the benefits for subscribers of public trust aren’t free and without commitments?

Regarding the CRL bloat issue, I’m far from an expert in webPKI but I’m wondering if there is a point where it’s more appropriate to revoke an intermediaries instead of leaf certificates even if this would come with collateral damage. I really don’t think that ”we have mississued so many certificates that we cannot revoke them” acceptable, perhaps others have deeper insight? Is this covered in the new requirement regarding mass-revocation from Mozilla? Maybe it’s time for a similar requirement, but instead to demonstrate how a CA is monitoring bugzilla and learning from incidents happening for other CAs?

I welcome the requirement for ha

We agree that public trust comes with meaningful responsibilities, including timely and effective revocation.

Please see the Full incident report from Bug 1965612 for to review our action items including revocation of impacted certificates and how we will improve our ability to respond in the future.

On the point about learnings from other incidents: we agree. Our incident report includes a formal action item to improve our Bugzilla review process for external incidents. This effort is already underway and is intended to ensure that insights from the broader Web PKI community are proactively incorporated into our practices.

Comment 20

[Microsoft PKI Services]

Update on Action Items

We are actively progressing through all repair items identified in the incident report. All action items are currently in progress, with the exception of the implementation phase for identified linting gaps, which will begin following completion of the gap identification. We remain on track to meet the expected due dates outlined in the full report.

Comment 21

[Microsoft PKI Services]

Weekly Status Update

---

We are actively progressing through all repair items identified in the incident report. All action items are currently in progress, with the exception of the implementation phase for identified linting gaps, which will begin following completion of the gap identification. We remain on track to meet the expected due dates outlined in the full report.

Comment 22

[Microsoft PKI Services]

Report Closure Summary

---

• Incident description:
  Microsoft PKI Services made a configuration change to remove the OCSP URI from certificates issued by 4 publicly trusted TLS Issuing CAs. This was related to some internal CA testing after OCSP became optional within the WebPKI industry. The Subscriber certificates without the OCSP URI did not comply with the active CPS at the time. The CPS version 3.2.4 stated in Section 7.1.2.7.7 that id-ad-ocsp was PRESENT in certificates. Subscriber certificates were issued before a new CPS version 3.3.0 was published that allowed id-ad-ocsp to be optional.

  • CPS 3.2.4 was effective until 2025-04-21. Section 7.1.2.7.7 has id-ad-ocsp 1.3.6.1.5.5.7.48.1 uniformResourceIdentifier PRESENT.
  • CPS 3.3.0 was effective starting 2025-04-21. Section 7.1.2.7.7 points to Appendix B, where the Organization Validated TLS Subscriber Certificate profile has extension:authorityInformationAccess not marked critical, contains the HTTP URL of an Issuing CA's certificate and MAY contain the HTTP URL of Issuing CA's OCSP responder.
    Certificates impacted by this bug were revoked within the 24-hour timeline per baseline requirements.

• Incident Root Cause(s):
  The project plan lacked a specific tracking item to ensure the CPS change was made before the CA change. Additionally, the change process did not include a specific callout to review the CPS prior to implementation. This oversight resulted in a race condition where the CA change occurred in parallel with the CPS change, but prior to the updated CPS being published.

• Remediation description:
  Microsoft PKI Services has implemented the following remediation steps:

  1. Updated the change management process to include a mandatory review of the CPS before implementing any CA changes.
  2. Added specific tracking items in project plans to ensure CPS updates are completed prior to CA modifications.

• Commitment summary:
  Microsoft PKI Services is committed to ongoing improvements in the alignment of operational and policy changes. As part of our continuous improvement efforts, we have made the following ongoing commitments:

  • We are investing in improved documentation version control and traceability via Git-based policy management, allowing for greater transparency and historical validation of sequencing. Action Items listed in Bug 1962829
  • We will integrate policy alignment checks into our internal tooling used for change tracking, ensuring that required documents are flagged for review prior to execution of configuration changes. Action Items listed in Bug 1962829
  • Require recurring compliance training for all stakeholders involved in CA changes, including specific modules on sequencing, documentation, and CP/CPS policy alignment. Action Items listed in Bug 1965612

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 23

[Wayne]

(In reply to Microsoft PKI Services from comment #22)

Report Closure Summary

---

• Incident description:
  Microsoft PKI Services made a configuration change to remove the OCSP URI from certificates issued by 4 publicly trusted TLS Issuing CAs. This was related to some internal CA testing after OCSP became optional within the WebPKI industry. The Subscriber certificates without the OCSP URI did not comply with the active CPS at the time. The CPS version 3.2.4 stated in Section 7.1.2.7.7 that id-ad-ocsp was PRESENT in certificates. Subscriber certificates were issued before a new CPS version 3.3.0 was published that allowed id-ad-ocsp to be optional.

  • CPS 3.2.4 was effective until 2025-04-21. Section 7.1.2.7.7 has id-ad-ocsp 1.3.6.1.5.5.7.48.1 uniformResourceIdentifier PRESENT.
  • CPS 3.3.0 was effective starting 2025-04-21. Section 7.1.2.7.7 points to Appendix B, where the Organization Validated TLS Subscriber Certificate profile has extension:authorityInformationAccess not marked critical, contains the HTTP URL of an Issuing CA's certificate and MAY contain the HTTP URL of Issuing CA's OCSP responder.
    Certificates impacted by this bug were revoked within the 24-hour timeline per baseline requirements.

• Incident Root Cause(s):
  The project plan lacked a specific tracking item to ensure the CPS change was made before the CA change. Additionally, the change process did not include a specific callout to review the CPS prior to implementation. This oversight resulted in a race condition where the CA change occurred in parallel with the CPS change, but prior to the updated CPS being published.

• Remediation description:
  Microsoft PKI Services has implemented the following remediation steps:

  1. Updated the change management process to include a mandatory review of the CPS before implementing any CA changes.
  2. Added specific tracking items in project plans to ensure CPS updates are completed prior to CA modifications.

• Commitment summary:
  Microsoft PKI Services is committed to ongoing improvements in the alignment of operational and policy changes. As part of our continuous improvement efforts, we have made the following ongoing commitments:

  • We are investing in improved documentation version control and traceability via Git-based policy management, allowing for greater transparency and historical validation of sequencing. Action Items listed in Bug 1962829
  • We will integrate policy alignment checks into our internal tooling used for change tracking, ensuring that required documents are flagged for review prior to execution of configuration changes. Action Items listed in Bug 1962829
  • Require recurring compliance training for all stakeholders involved in CA changes, including specific modules on sequencing, documentation, and CP/CPS policy alignment. Action Items listed in Bug 1965612

All Action Items disclosed in this report have been completed as described, and we request its closure.

Was this intended to be posted on a different incident?

Comment 24

[Microsoft PKI Services]

(In reply to Wayne from comment #23)

(In reply to Microsoft PKI Services from comment #22)

Report Closure Summary

---

• Incident description:
  Microsoft PKI Services made a configuration change to remove the OCSP URI from certificates issued by 4 publicly trusted TLS Issuing CAs. This was related to some internal CA testing after OCSP became optional within the WebPKI industry. The Subscriber certificates without the OCSP URI did not comply with the active CPS at the time. The CPS version 3.2.4 stated in Section 7.1.2.7.7 that id-ad-ocsp was PRESENT in certificates. Subscriber certificates were issued before a new CPS version 3.3.0 was published that allowed id-ad-ocsp to be optional.

  • CPS 3.2.4 was effective until 2025-04-21. Section 7.1.2.7.7 has id-ad-ocsp 1.3.6.1.5.5.7.48.1 uniformResourceIdentifier PRESENT.
  • CPS 3.3.0 was effective starting 2025-04-21. Section 7.1.2.7.7 points to Appendix B, where the Organization Validated TLS Subscriber Certificate profile has extension:authorityInformationAccess not marked critical, contains the HTTP URL of an Issuing CA's certificate and MAY contain the HTTP URL of Issuing CA's OCSP responder.
    Certificates impacted by this bug were revoked within the 24-hour timeline per baseline requirements.

• Incident Root Cause(s):
  The project plan lacked a specific tracking item to ensure the CPS change was made before the CA change. Additionally, the change process did not include a specific callout to review the CPS prior to implementation. This oversight resulted in a race condition where the CA change occurred in parallel with the CPS change, but prior to the updated CPS being published.

• Remediation description:
  Microsoft PKI Services has implemented the following remediation steps:

  1. Updated the change management process to include a mandatory review of the CPS before implementing any CA changes.
  2. Added specific tracking items in project plans to ensure CPS updates are completed prior to CA modifications.

• Commitment summary:
  Microsoft PKI Services is committed to ongoing improvements in the alignment of operational and policy changes. As part of our continuous improvement efforts, we have made the following ongoing commitments:

  • We are investing in improved documentation version control and traceability via Git-based policy management, allowing for greater transparency and historical validation of sequencing. Action Items listed in Bug 1962829
  • We will integrate policy alignment checks into our internal tooling used for change tracking, ensuring that required documents are flagged for review prior to execution of configuration changes. Action Items listed in Bug 1962829
  • Require recurring compliance training for all stakeholders involved in CA changes, including specific modules on sequencing, documentation, and CP/CPS policy alignment. Action Items listed in Bug 1965612

All Action Items disclosed in this report have been completed as described, and we request its closure.

Was this intended to be posted on a different incident?

Thank you for bringing this to our attention. Please disregard this comment in the bug. This was intended for https://bugzilla.mozilla.org/show_bug.cgi?id=1962830 and will be posted their shortly.

Comment 25

[Microsoft PKI Services]

Weekly Status Update

---

We’re actively making progress on several of the repair items identified in the incident report. For the action items highlighted in bold, we’ve updated the due dates accordingly.
Regarding the CPS-related updates (action items #2 & 8), we’ve encountered a number of questions during our internal governance review, which has extended the timeline for publishing the latest CPS.
As for our internal analysis comparing linting rules and certificate profiles (action item #3), we’re in the final stages and expect to complete it by next week.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalize Bugzilla bug learnings process	Prevent	Root Cause #1	Effectiveness will be measured by adherence to a regular review schedule, tracking of each Bugzilla issue reviewed, review outcomes, and integration of those outcomes into our engineering rhythm in the form of work items.	7/15/2025	In-Progress
Update next CP/CPS with Microsoft OID	Mitigate	Root Cause #1	Include extension and OID information from our Microsoft private Arc as identified in Comment 8 in our next public TLS CP/CPS update.	6/27/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Mitigate	Root Cause #1	Perform a comprehensive review of linting rules against certificate profiles defined in our CP/CPS to identify and address discrepancies.	6/20/2025	In-Progress
Convert CPS into Markdown	Prevent	Root Cause #1	Transition CPS documentation to Markdown format to improve maintainability and consistency.	9/1/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Prevent	Root Cause #1	Implement fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.	TBD	Not Started
Increase number of CPS reviewers	Prevent	Root Cause #1	Update document review process to include more reviewers from operational teams	6/13/2025	Done
Document version management improvements	Prevent	Root Cause #1	Implement Git-based version control to replace Word documents managed in SharePoint.	6/13/2025	Done
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause #1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT be set when using ECC.	6/27/2025	In-Progress
Evaluate CPS-to-lint translation to codify enforceable pre-issuance checks	Prevent	Root Cause #1	Effectiveness will be measured by the successful mapping CPS control to a lint rule along with an implementation plan for broader CPS rule coverage.	6/30/2025	In-Progess

Comment 26

[Microsoft PKI Services]

Weekly Status Update

---

We’re actively making progress on several of the repair items identified in the incident report. Regarding the CPS-related updates (action item #2/ #8), we plan to incorporate several other important updates, which has extended the timeline for publishing the latest CPS.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalize Bugzilla bug learnings process	Prevent	Root Cause #1	Effectiveness will be measured by adherence to a regular review schedule, tracking of each Bugzilla issue reviewed, review outcomes, and integration of those outcomes into our engineering rhythm in the form of work items.	7/15/2025	In-Progress
Update next CP/CPS with Microsoft OID	Mitigate	Root Cause #1	Include extension and OID information from our Microsoft private Arc as identified in Comment 8 in our next public TLS CP/CPS update.	7/3/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Mitigate	Root Cause #1	Perform a comprehensive review of linting rules against certificate profiles defined in our CP/CPS to identify and address discrepancies.	6/20/2025	Done
Convert CPS into Markdown	Prevent	Root Cause #1	Transition CPS documentation to Markdown format to improve maintainability and consistency.	9/1/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Prevent	Root Cause #1	Implement fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.	9/30/2025	Not Started
Increase number of CPS reviewers	Prevent	Root Cause #1	Update document review process to include more reviewers from operational teams	6/13/2025	Done
Document version management improvements	Prevent	Root Cause #1	Implement Git-based version control to replace Word documents managed in SharePoint.	6/13/2025	Done
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause #1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT	6/30/2025	In-Progress
Evaluate CPS-to-lint translation to codify enforceable pre-issuance checks	Prevent	Root Cause #1	Effectiveness will be measured by the successful mapping CPS control to a lint rule along with an implementation plan for broader CPS rule coverage.	6/30/2025	In-Progress

Comment 27

[Microsoft PKI Services]

Weekly Status Update

---

We’re actively making progress on several of the repair items identified in the incident report. Please see action item updates.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalize Bugzilla bug learnings process	Prevent	Root Cause #1	Effectiveness will be measured by adherence to a regular review schedule, tracking of each Bugzilla issue reviewed, review outcomes, and integration of those outcomes into our engineering rhythm in the form of work items.	7/15/2025	In-Progress
Update next CP/CPS with Microsoft OID	Mitigate	Root Cause #1	Include extension and OID information from our Microsoft private Arc as identified in Comment 8 in our next public TLS CP/CPS update.	7/3/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Mitigate	Root Cause #1	Perform a comprehensive review of linting rules against certificate profiles defined in our CP/CPS to identify and address discrepancies.	6/20/2025	Done
Convert CPS into Markdown	Prevent	Root Cause #1	Transition CPS documentation to Markdown format to improve maintainability and consistency.	9/1/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Prevent	Root Cause #1	Implement fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.	9/30/2025	In-Progress
Increase number of CPS reviewers	Prevent	Root Cause #1	Update document review process to include more reviewers from operational teams	6/13/2025	Done
Document version management improvements	Prevent	Root Cause #1	Implement Git-based version control to replace Word documents managed in SharePoint.	6/13/2025	Done
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause #1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT	7/3/2025	In-Progress
Evaluate CPS-to-lint translation to codify enforceable pre-issuance checks	Prevent	Root Cause #1	Effectiveness will be measured by the successful mapping CPS control to a lint rule along with an implementation plan for broader CPS rule coverage.	6/30/2025	Done

Comment 28

[Microsoft PKI Services]

Weekly Status Update

We’re actively making progress on several of the repair items identified in the incident report. Main changes are to the status of action items #2 and #8. Our new CPS has been published.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalize Bugzilla bug learnings process	Prevent	Root Cause #1	Effectiveness will be measured by adherence to a regular review schedule, tracking of each Bugzilla issue reviewed, review outcomes, and integration of those outcomes into our engineering rhythm in the form of work items.	7/15/2025	In-Progress
Update next CP/CPS with Microsoft OID	Mitigate	Root Cause #1	Include extension and OID information from our Microsoft private Arc as identified in Comment 8 in our next public TLS CP/CPS update.	7/3/2025	Done
Ensure consistency between our CP/CPS and our linting mechanism	Mitigate	Root Cause #1	Perform a comprehensive review of linting rules against certificate profiles defined in our CP/CPS to identify and address discrepancies.	6/20/2025	Done
Convert CPS into Markdown	Prevent	Root Cause #1	Transition CPS documentation to Markdown format to improve maintainability and consistency.	9/1/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Prevent	Root Cause #1	Implement fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.	9/30/2025	In-Progress
Increase number of CPS reviewers	Prevent	Root Cause #1	Update document review process to include more reviewers from operational teams	6/13/2025	Done
Document version management improvements	Prevent	Root Cause #1	Implement Git-based version control to replace Word documents managed in SharePoint.	6/13/2025	Done
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause #1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT be set when using ECC.	7/3/2025	Done
Evaluate CPS-to-lint translation to codify enforceable pre-issuance checks	Prevent	Root Cause #1	Effectiveness will be measured by the successful mapping CPS control to a lint rule along with an implementation plan for broader CPS rule coverage.	6/30/2025	Done

Comment 29

[Microsoft PKI Services]

Weekly Status Update

---

We are actively progressing through all repair items identified in the incident report. We remain on track to meet the expected due dates outlined in the full report.

Comment 30

[Microsoft PKI Services]

Weekly Status Update

---

We would like to notify that we finished Action Item #1 on 7/15/2025. Additionally, we would like to request a change to the cadence of our action item updates for this bug. The remaining action items are targeted for September, and as such, we propose to provide our next update on Friday, August 29th, unless action items are completed sooner.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalize Bugzilla bug learnings process	Prevent	Root Cause #1	Effectiveness will be measured by adherence to a regular review schedule, tracking of each Bugzilla issue reviewed, review outcomes, and integration of those outcomes into our engineering rhythm in the form of work items.	7/15/2025	Done
Update next CP/CPS with Microsoft OID	Mitigate	Root Cause #1	Include extension and OID information from our Microsoft private Arc as identified in Comment 8 in our next public TLS CP/CPS update.	7/3/2025	Done
Ensure consistency between our CP/CPS and our linting mechanism	Mitigate	Root Cause #1	Perform a comprehensive review of linting rules against certificate profiles defined in our CP/CPS to identify and address discrepancies.	6/20/2025	Done
Convert CPS into Markdown	Prevent	Root Cause #1	Transition CPS documentation to Markdown format to improve maintainability and consistency.	9/1/2025	In-Progress
Ensure consistency between our CP/CPS and our linting mechanism	Prevent	Root Cause #1	Implement fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.	9/30/2025	In-Progress
Increase number of CPS reviewers	Prevent	Root Cause #1	Update document review process to include more reviewers from operational teams	6/13/2025	Done
Document version management improvements	Prevent	Root Cause #1	Implement Git-based version control to replace Word documents managed in SharePoint.	6/13/2025	Done
Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC)	Mitigate	Root Cause #1	Modify Appendix B Certificate Profile for OV Subscriber Certificates to explicitly state that keyEncipherment MUST NOT be set when using ECC.	7/3/2025	Done
Evaluate CPS-to-lint translation to codify enforceable pre-issuance checks	Prevent	Root Cause #1	Effectiveness will be measured by the successful mapping CPS control to a lint rule along with an implementation plan for broader CPS rule coverage.	6/30/2025	Done

Please let us know if we can set the Next Update = 2025-08-29.

Comment 31

[Microsoft PKI Services]

Weekly Status Update:

---

We’re actively making progress on the remaining repair items identified in the incident report. No significant changes to report at this time. Additionally, we would like to request a change to the cadence of our action item updates for this bug. The remaining action items are targeted for September, and as such, we propose to provide our next update on Friday, August 29th, unless action items are completed sooner.