2059818 - Microsoft PKI Services: Third-Party Code Signing CPS missed annual update and four Issuing CAs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Qais Al Hajri]

Preliminary Incident Report

Summary

• Incident description:

  Microsoft PKI Services missed updates to its third-party code signing Certification Practice Statement (Microsoft_PKI_ThirdParty_CPS). Microsoft PKI Services became aware of this on 2026-07-30 at 02:06 UTC. Two omissions are reported:

  1. Missed annual republication. The CPS was last published as v1.0.5 on 2025-07-21. It was not reviewed and republished by its annual deadline of 2026-07-21, and no incremented version or dated changelog entry was published.

  2. Four Issuing CAs absent from the CPS. Four subordinate CAs subordinate to Microsoft ID Verified Code Signing PCA 2021 were created on 2026-03-26 and were never added to the CPS:

  Subject CN	SHA-256 Thumbprint	Not Before (UTC)	Not After (UTC)
  Microsoft ID Verified CS EOC CA 03	BFF5C1A54B421E0CC12372F221FA56E6F1B0305A9C071A2AF1BF45893D2A4623	2026-03-26 18:11:28	2031-03-26 18:11:28
  Microsoft ID Verified CS AOC CA 04	15CEF5F63CA6D1F022B293E92C61C0059C49BB92EECDBCDE3A125D3C6356D94F	2026-03-26 18:11:29	2031-03-26 18:11:29
  Microsoft ID Verified CS EOC CA 04	BC309555FA42C563B8EEBFB7117695C6BAE89E4BA19C03CB721979A563A60929	2026-03-26 18:11:31	2031-03-26 18:11:31
  Microsoft ID Verified CS AOC CA 03	ACAA07D57D4274B290D86C156AA0D6C5633E3CD92D43751E6A0F797C5B2DFF4D	2026-03-26 18:11:32	2031-03-26 18:11:32

  All four remain valid. The root of this hierarchy, Microsoft Identity Verification Root Certificate Authority 2020 (SHA-256 5367F20C7ADE0E2BCA790915056D086B720C33C1FA2A2661ACF787E3292E1270), is disclosed in the published CPS.

  Impact and containment. No end-entity certificate is known to have been mis-issued as a result of either omission; no certificate content, validation, or key management deficiency has been identified in connection with them. The two omissions are contained: the scope is fixed and bounded by the four Issuing CAs listed above and by the CPS publication date. Issuance was not stopped, because neither omission indicates a defect in certificate issuance itself. An updated CPS containing all four Issuing CAs has been drafted and is in approval.

  A Full Incident Report, including the complete timeline, root cause analysis, lessons learned, and action items, will follow within 14 days.

• Relevant policies:

  • CA/Browser Forum Code Signing Baseline Requirements (CSBR) v3.11.0, Section 2.3 (Time or frequency of publication): "The CA SHALL develop, implement, enforce, and annually update a Certificate Policy and/or Certification Practice Statement that describes in detail how the CA implements the latest version of these Requirements. The CA SHALL indicate conformance with this requirement by incrementing the version number and adding a dated changelog entry, even if no other changes are made to the document." — Not met for the annual period ending 2026-07-21.
  • CSBR v3.11.0, Section 2.2 (Publication of certification information): requires the CA to "periodically update its policies and practices, including its Certificate Policy and/or Certification Practice Statement," and requires that the CP/CPS "specify the CA's (and applicable Root CA's) entire root certificate hierarchy."
  • Microsoft_PKI_ThirdParty_CPS v1.0.5, Section 2.3: "The CA SHALL annually review their CP and CPS… Updates SHALL be published annually, in accordance with Section 1.5, and the document version number SHALL be incremented to account for the annual review and potential content revisions." — Not met. Section 1.1 of the same document enumerates the Root and Subordinate CAs the CPS governs; the four Issuing CAs above were omitted from that enumeration.
  • Microsoft Trusted Root Program Requirements: CA policy documentation currency and accuracy.

• Source of incident disclosure: Self Reported

Comment 1

[Qais Al Hajri]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A002577

• Incident description: Microsoft PKI Services missed updates to its third-party code signing Certification Practice Statement (Microsoft_PKI_ThirdParty_CPS). Microsoft PKI Services became became aware of this on 2026-07-30 at 02:06 UTC. Two omissions are reported:

  1. Missed annual review. The CPS was last published as v1.0.5 on 2025-07-21 and was not updated by its annual review deadline of 2026-07-21.
  2. Four Issuing CAs missing from the CPS. Four subordinate CAs under Microsoft ID Verified Code Signing PCA 2021 were created on 2026-03-26 and never added to the CPS.

  The two have separate causes. The missing Issuing CAs came from a gap in the CA creation workflow in March. The missed annual review came from a monitoring gap in July. We report them together because they affect the same document.

  No end-entity certificate was mis-issued and no revocation is required. Both defects concern the accuracy and currency of a published document.

• Timeline summary:

  • Non-compliance start date: 2026-03-26 18:11 UTC (Issuing CAs created but not added to the CPS). The annual review non-compliance started separately at 2026-07-22 00:00 UTC.
  • Non-compliance identified date: 2026-07-30 02:06 UTC
  • Non-compliance end date: 2026-08-05 (publication of v1.0.6)

• Relevant policies:

  • CSBR v3.11.0 §2.3 (Time or frequency of publication): "The CA SHALL develop, implement, enforce, and annually update a Certificate Policy and/or Certification Practice Statement… The CA SHALL indicate conformance with this requirement by incrementing the version number and adding a dated changelog entry, even if no other changes are made to the document."
  • CSBR v3.11.0 §2.2 (Publication of certification information): requires the CP/CPS to be periodically updated and to "specify the CA's (and applicable Root CA's) entire root certificate hierarchy."
  • Microsoft_PKI_ThirdParty_CPS v1.0.5 §2.3: requires annual review, annual publication, and an incremented version number. §1.1 lists the Root and Subordinate CAs the CPS governs; the four Issuing CAs were missing from that list.
  • Microsoft Trusted Root Program Requirements: CA policy documentation currency and accuracy.
  • CCADB Policy v2.1 §4 (Policy Disclosures): requires CA Owners to "accurately describe the policies and practices of their CA(s)" in a CP and corresponding CPS, "sufficiently detailed to assess the operations of the CA(s)." A §1.1 list that omitted four live Issuing CAs did not meet that expectation.

• Source of incident disclosure: Self Reported

Impact

• Total number of certificates: 4 subordinate CA certificates were not listed in the governing CPS. 0 end-entity certificates were mis-issued.
• Total number of "remaining valid" certificates: 4 (see Appendix).
• Affected certificate types: Third-party code signing subordinate CAs under Microsoft ID Verified Code Signing PCA 2021.
• Incident heuristic: Every subordinate CA certificate issued by CN=Microsoft ID Verified Code Signing PCA 2021, O=Microsoft Corporation, C=US on 2026-03-26 that was not listed in v1.0.5. All four are in the Appendix.
• Was issuance stopped in response to this incident, and why or why not?: No. The four subordinate CAs were created under the approved ceremony, technical profile and operational controls already described in the CPS, and were in scope of the WebTrust for CA – Code Signing audit. Only the published document was wrong.
• Analysis: N/A. No revocation is required or delayed.
• Additional considerations:
  • Relying parties and root programs use the published CPS to determine which CAs operate under which practices. For about four months it did not list four live Issuing CAs, and for nine days it was past its annual deadline. We are not aware of anyone being affected, but the point of the CPS is that nobody should have to ask us.
  • The Root CA and its Time Stamping subordinate CAs are disclosed in CCADB. The four Issuing CAs are not. The Microsoft Trusted Root Program confirmed they have no other third-party code-signing consumers and are not subject to subordinate CA disclosure under CCADB Policy §3.2. This report therefore covers the CPS only, not CCADB certificate disclosure.
  • Two CA/Browser Forum ballots were adopted after v1.0.5 was published, and both were checked. CSC-31 (Maximum Validity Reduction, effective 2026-03-01) caps code signing validity at 460 days; CPS §6.3.2 conforms. CSC-32 (Mandatory Reserved Policy OID, effective 2026-09-15) is not yet in force; §1.2 and §7.1.6 already carry the reserved policy OIDs and we will confirm conformance before the effective date. CCADB Policy §4 also requires policy documents to state adherence to the CCADB Policy from 2026-09-15, which we will address before that date.

Timeline

All times UTC.

Date/Time	Event
2025-07-21	Microsoft_PKI_ThirdParty_CPS v1.0.5 published, starting the one-year review period.
2026-03-26 18:11:28–18:11:32	Four subordinate CAs created under Microsoft ID Verified Code Signing PCA 2021 (see Appendix). No CPS update started. Non-compliance with CSBR §2.2 begins.
2026-07-21	Annual review deadline. No review or update happens.
2026-07-22 00:00	Non-compliance with CSBR §2.3 begins.
2026-07-28	WebTrust for CA – Code Signing report issued. The four Issuing CAs are in the audited scope but absent from the published CPS. The discrepancy is not spotted.
2026-07-30 02:06	Internal compliance review finds the CPS is past its annual review deadline. Incident awareness established.
2026-07-30 16:35	Scope set: (1) missed annual review, (2) Issuing CAs missing from the CPS, (3) whether those CAs need CCADB disclosure.
2026-07-30 22:23	Certificate inventory for the four Issuing CAs compiled and independently checked against CA records.
2026-07-31 17:53	Preliminary Incident Report filed as Bugzilla 2059818, about 39.8 hours after awareness.
2026-08-05	v1.0.6 approved by the Microsoft PKI Policy Authority and published, listing all four Issuing CAs with an incremented version number and dated changelog entry. Non-compliance ends.

Related Incidents

Bug	Date	Description
1817023	2023-02-15	Microsoft PKI Services: Failure to Modify Policy Documents within 365 Days. Concerned the TLS CP/CPS. Stated root cause was that "policy document management was entirely a manual process managed by a primary individual and backup missed this year." Remediation was automated monitoring of policy-document age. See Contributing Factor 1.
2056934	2026-07-22	Actalis: failure to timely update CP/CPS for AgID SubCAs.
1935393	2024-12-05	Asseco DS / Certum: Failure to Update Policy Documents within 365 Days. Also found by internal review rather than by a control.
1769222	2022-05-13	SECOM: Failed an annual CPS update of Cybertrust Japan. One document missed within a broader hierarchy.
1956681	2025-03-27	Entrust: Cross-certified CA CP/CPS not updated in CCADB. Documentation not kept current for the full set of CAs in scope.

Recurrence. Bug 1817023 was our own incident on the same requirement, closed in 2023 on a commitment to automated monitoring. What we built was a scheduled script deployed to an environment that was later deprecated. The script went away with that environment and was never re-implemented elsewhere, and we had closed the bug on that basis. Our takeaway is that a remediation is not complete until the resulting control is owned, runs on managed infrastructure whose lifecycle is tracked, and covers every document in the class.

Root Cause Analysis

Contributing Factor 1: Annual review monitoring was a stop-gap script that no longer existed

• Description: After Bug 1817023 we committed to automated monitoring of policy-document age. What we built was a script on a schedule, deployed to an environment that was later deprecated. When that environment was retired the script was decommissioned with it and never re-implemented. Nothing tied the retirement of that environment to the compliance control it was carrying, so its loss was never noticed. We had closed Bug 1817023 anyway. This factor explains the missed annual review and the nine days it went unnoticed.
• Timeline: Committed in 2023 and implemented as a scheduled script; the hosting environment was subsequently deprecated and the script was never re-implemented, so no monitoring existed when the 2026-07-21 deadline passed.
• Detection: Not detected by any monitoring, because none was running. That the script no longer existed came to light during triage, when we asked directly what alerting existed and what it covered.
• Interaction with other factors: The CPS lifecycle was a manual, single-threaded workflow. Drafting, review, approval routing and publication were all done by hand, with nothing that re-raised an approaching deadline independently of the people doing the work. Automated monitoring was meant to be the backstop, and it had silently ceased to exist, so there was no effective second check.
• Root Cause Analysis methodology used: 5 Whys; Fishbone (Process / People / Tooling).

Contributing Factor 2: CA creation was not linked to policy-document updates

• Description: Creating an Issuing CA is a controlled, ceremonied event, but the workflow had no step requiring the governing CPS to be updated to list the new CA. The four Issuing CAs went into production correctly from a key management standpoint, with no corresponding document update. This factor explains the four undisclosed Issuing CAs, and it is sufficient on its own. Even if the annual review had happened on time, nothing would have surfaced the March 2026 CAs, so a reviewer working from the existing document would have had no reason to think §1.1 was incomplete.
• Timeline: Present since the CA creation workflow was defined; realised on 2026-03-26.
• Detection: Not detected at creation. Found on 2026-07-30 while investigating the missed annual review, and confirmed by comparing the WebTrust audit scope against the published CPS.
• Interaction with other factors: Independent of Contributing Factor 1 in origin, though it produced the same kind of defect.
• Root Cause Analysis methodology used: 5 Whys; Second-order thinking.

Lessons Learned

What went well:

• We engaged the right people on the day of discovery, including the Microsoft Trusted Root Program and engineering and compliance leadership, before taking any public position.
• Mitigation took five days. Publishing a revised CPS normally takes weeks because of the approvals involved.
• A second person independently verified the certificate inventory against CA records before approval.
• No certificate profile, validation or key management problem was involved, so there was no subscriber impact and nothing to revoke.

What didn't go well:

• Annual review monitoring was a script that had been decommissioned along with its hosting environment and never re-implemented, so nothing raised the deadline to anyone who could act on it and the lapse ran for nine days. (Action Item 2)
• We hit the same failure mode as Bug 1817023 because we closed that bug on a stop-gap instead of a durable control covering all policy documents. (Action Item 2)

Where we got lucky:

• Detection depended on someone looking, not on a control. An unrelated internal compliance review happened to check the CPS publication date nine days after the deadline. Nothing bounded how long this could have continued. (Action Item 2)
• The two omissions were only found together by coincidence. The new WebTrust report was being reviewed at the same time as the CPS, which put two documents describing the same hierarchy side by side. Had those reviews been weeks apart, the missing CAs would not have surfaced here and the four-month exposure would have continued. (Action Item 3)

Additional: N/A

Action Items

#	Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
1	Publish Microsoft_PKI_ThirdParty_CPS v1.0.6 listing all four Issuing CAs, after reconciling the full Issuing CA inventory against Section 1.1 and obtaining Policy Authority approval.	Mitigate	Contributing Factor 1, Contributing Factor 2	Publicly verifiable: the updated CPS is in the Microsoft PKI repository with an incremented version number, a dated changelog entry, and Section 1.1 listing all Issuing CAs with matching serial numbers and SHA-256 thumbprints.	2026-08-05	Done
2	Replace the decommissioned stop-gap script with annual-review-deadline monitoring covering every Certificate Policy and Certification Practice Statement. Alerts raised as tracked items in the owning service's incident queue at 180, 90, 60 and 30 days before the deadline, and a high-severity incident at 14 days. Coverage taken from the published repository rather than a maintained list. The service must monitor its own health so that a failure of the monitor is itself detectable.	Detect	Contributing Factor 1	Zero policy documents pass their annual review date without a published update, with coverage verified against the public repository rather than a maintained list. Publicly measurable via on-time publication dates.	2026-10-05	Not started
3	Update the CA creation process so the governing CPS must be updated and published before a newly created Issuing CA enters production.	Prevent	Contributing Factor 2	No Issuing CA enters production without appearing in the current published governing CPS, evidenced in ceremony records available to auditors. Publicly verifiable by comparing published CPS contents against ceremony dates.	2026-09-01	Not started

Appendix

All four subordinate CA certificates were issued by CN=Microsoft ID Verified Code Signing PCA 2021, O=Microsoft Corporation, C=US, chaining to Microsoft Identity Verification Root Certificate Authority 2020 (SHA-256 5367F20C7ADE0E2BCA790915056D086B720C33C1FA2A2661ACF787E3292E1270), which is listed in the published CPS. All four remain valid and none require revocation.

#	Subject CN	Serial Number	SHA-256 Thumbprint	Not Before (UTC)	Not After (UTC)
1	Microsoft ID Verified CS EOC CA 03	3300000015053E6E198E9329D9000000000015	BFF5C1A54B421E0CC12372F221FA56E6F1B0305A9C071A2AF1BF45893D2A4623	2026-03-26 18:11:28	2031-03-26 18:11:28
2	Microsoft ID Verified CS AOC CA 04	330000001631928D9202657B9A000000000016	15CEF5F63CA6D1F022B293E92C61C0059C49BB92EECDBCDE3A125D3C6356D94F	2026-03-26 18:11:29	2031-03-26 18:11:29
3	Microsoft ID Verified CS EOC CA 04	33000000172745090B8B87AFC6000000000017	BC309555FA42C563B8EEBFB7117695C6BAE89E4BA19C03CB721979A563A60929	2026-03-26 18:11:31	2031-03-26 18:11:31
4	Microsoft ID Verified CS AOC CA 03	33000000180DEB9155A98A0098000000000018	ACAA07D57D4274B290D86C156AA0D6C5633E3CD92D43751E6A0F797C5B2DFF4D	2026-03-26 18:11:32	2031-03-26 18:11:32

Policy documents: Microsoft_PKI_ThirdParty_CPS v1.0.5 (published 2025-07-21); v1.0.6 (published 2026-08-05), available at https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_ThirdParty_CPS_v1.0.6.pdf. Repository: https://www.microsoft.com/pkiops/docs/repository.htm

Comment 2

[Microsoft PKI Services]

Weekly Status Update

We have closed one remediation action item related to the reported incident. The following two action items remain open.

We will provide the next update on: September 2nd.

#	Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
1	Publish Microsoft_PKI_ThirdParty_CPS v1.0.6 listing all four Issuing CAs, after reconciling the full Issuing CA inventory against Section 1.1 and obtaining Policy Authority approval.	Mitigate	Contributing Factor 1, Contributing Factor 2	Publicly verifiable: the updated CPS is in the Microsoft PKI repository with an incremented version number, a dated changelog entry, and Section 1.1 listing all Issuing CAs with matching serial numbers and SHA-256 thumbprints.	2026-08-05	Done
2	Replace the decommissioned stop-gap script with annual-review-deadline monitoring covering every Certificate Policy and Certification Practice Statement. Alerts raised as tracked items in the owning service's incident queue at 180, 90, 60 and 30 days before the deadline, and a high-severity incident at 14 days. Coverage taken from the published repository rather than a maintained list. The service must monitor its own health so that a failure of the monitor is itself detectable.	Detect	Contributing Factor 1	Zero policy documents pass their annual review date without a published update, with coverage verified against the public repository rather than a maintained list. Publicly measurable via on-time publication dates.	2026-10-05	In Progress
3	Update the CA creation process so the governing CPS must be updated and published before a newly created Issuing CA enters production.	Prevent	Contributing Factor 2	No Issuing CA enters production without appearing in the current published governing CPS, evidenced in ceremony records available to auditors. Publicly verifiable by comparing published CPS contents against ceremony dates.	2026-09-01	In Progress

Comment 3

[Microsoft PKI Services]

We kindly request that the next update date be set to September 2nd, 2026.

Comment 4

[Microsoft PKI Services]

Status Update - 2/3 Action Items Completed

We have closed one more remediation action item related to the reported incident. Only one action item remains open (#2).

#	Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
1	Publish Microsoft_PKI_ThirdParty_CPS v1.0.6 listing all four Issuing CAs, after reconciling the full Issuing CA inventory against Section 1.1 and obtaining Policy Authority approval.	Mitigate	Contributing Factor 1, Contributing Factor 2	Publicly verifiable: the updated CPS is in the Microsoft PKI repository with an incremented version number, a dated changelog entry, and Section 1.1 listing all Issuing CAs with matching serial numbers and SHA-256 thumbprints.	2026-08-05	Done
2	Replace the decommissioned stop-gap script with annual-review-deadline monitoring covering every Certificate Policy and Certification Practice Statement. Alerts raised as tracked items in the owning service's incident queue at 180, 90, 60 and 30 days before the deadline, and a high-severity incident at 14 days. Coverage taken from the published repository rather than a maintained list. The service must monitor its own health so that a failure of the monitor is itself detectable.	Detect	Contributing Factor 1	Zero policy documents pass their annual review date without a published update, with coverage verified against the public repository rather than a maintained list. Publicly measurable via on-time publication dates.	2026-10-05	In Progress
3	Update the CA creation process so the governing CPS must be updated and published before a newly created Issuing CA enters production.	Prevent	Contributing Factor 2	No Issuing CA enters production without appearing in the current published governing CPS, evidenced in ceremony records available to auditors. Publicly verifiable by comparing published CPS contents against ceremony dates.	2026-09-01	Done

Comment 5

[Microsoft PKI Services]

We kindly request that the next update date be set to September 15th, 2026.

Comment 6

[Microsoft PKI Services]

We kindly request that the next update date be set to October 5th, 2026, to match the due date of our remaining action item.

Comment 7

[Microsoft PKI Services]

Report Closure Summary

• Incident description:

  Microsoft PKI Services did not update Microsoft_PKI_ThirdParty_CPS within its annual review period, and the published CPS did not list four Issuing CAs created under Microsoft ID Verified Code Signing PCA 2021.

• Incident Root Cause(s):

  1. The annual-review monitoring committed to after Bugzilla 1817023 was implemented as a stop-gap script in an environment that was later decommissioned. The monitoring was not re-implemented, and the loss of the control was not detected.
  2. The CA creation process did not require the governing CPS to be updated and published before a newly created Issuing CA entered production.

• Remediation description:

  #	Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
  1	Publish Microsoft_PKI_ThirdParty_CPS v1.0.6 listing all four Issuing CAs, after reconciling the full Issuing CA inventory against Section 1.1 and obtaining Policy Authority approval.	Mitigate	Contributing Factor 1, Contributing Factor 2	Publicly verifiable: the updated CPS is in the Microsoft PKI repository with an incremented version number, a dated changelog entry, and Section 1.1 listing all Issuing CAs with matching serial numbers and SHA-256 thumbprints.	2026-08-05	Done
  2	Replace the decommissioned stop-gap script with annual-review-deadline monitoring covering every Certificate Policy and Certification Practice Statement. Alerts raised as tracked items in the owning service's incident queue at 180, 90, 60 and 30 days before the deadline, and a high-severity incident at 14 days. Coverage taken from the published repository rather than a maintained list. The service must monitor its own health so that a failure of the monitor is itself detectable.	Detect	Contributing Factor 1	Zero policy documents pass their annual review date without a published update, with coverage verified against the public repository rather than a maintained list. Publicly measurable via on-time publication dates.	2026-10-05	Done
  3	Update the CA creation process so the governing CPS must be updated and published before a newly created Issuing CA enters production.	Prevent	Contributing Factor 2	No Issuing CA enters production without appearing in the current published governing CPS, evidenced in ceremony records available to auditors. Publicly verifiable by comparing published CPS contents against ceremony dates.	2026-09-01	Done

• Commitment summary:

  Microsoft PKI Services will maintain the production policy-document monitoring control, preserve the CA-creation and CPS-publication gates, and retain evidence of monitor operation, incident creation, ceremony verification, and policy publication for ongoing compliance and audit review.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 8

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-10-05.