Closed Bug 2021175 Opened 7 months ago Closed 6 months ago

Microsoft PKI Services: Failure to update action item status within 3 days

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: CentralPKI, Assigned: CentralPKI)

Details

(Whiteboard: [ca-compliance] [policy-failure])

Preliminary Incident Report

Summary

  • Incident description:
    Microsoft PKI Services failed to update open incident reports within 3 days of action items being changed, completed, or delayed. This specifically impacts Bug 1962829 and Bug 1965612 , where action item timeline changes and status updates were not posted within the required 3-day window. Throughout June 2025 and February 2026, in some cases when action item due dates were extended, updates were not posted to these bugs within the required 3-day timeframe. Specific instances will be included in the appendix of the full incident report.

  • Relevant policies:
    CCADB Incident Reporting Guidelines
    CCADB Policy – Section 6.1- Disclosing and Responding to Incidents

  • Source of incident disclosure:
    Self- Reported

Assignee: nobody → CentralPKI
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [policy-failure]

We're past 14 days with no final incident report.

Full Incident Report

Summary

  • CA Owner CCADB unique ID:
    A002577

  • Incident description:
    Microsoft PKI Services (MPS) failed to update open incident reports within 3 days of action items being changed, completed, or delayed. This specifically impacts Bug 1962829 and Bug 1965612, where action item timeline changes and status updates were not posted within the required 3-day window. From June 2025 to February 2026, in some cases when action item due dates were extended, updates were not posted to these bugs within the required timeframe. Specific instances are included in the timeline of this full incident report.

  • Timeline summary:

    • Non-compliance start date:
      2025-06-13
    • Non-compliance identified date:
      2026-03-04
    • Non-compliance end date:
      2026-03-04

  • Relevant policies:
    CCADB Incident Reporting Guidelines
    CCADB Policy – Section 6.1- Disclosing and Responding to Incidents

  • Source of incident disclosure:
    Self-Reported (Third-Party Comment led to an analysis that discovered this bug)

Impact

  • Total number of certificates: N/A
  • Total number of "remaining valid" certificates: N/A
  • Affected certificate types: N/A
  • Incident heuristic: N/A
  • Was issuance stopped in response to this incident, and why or why not?: N/A
  • Analysis: N/A
  • Additional considerations: N/A

Timeline

_ We have separated our timeline by bug for clarity._

Bugzilla 1962829

  • 2025-05-16 : Microsoft PKI Services (MPS) created multiple action items:
    • Action item #2 – Update the next CP/CPS with Microsoft OID and assigned the due date of 6/13/2025.
    • Action item #3 – Perform analysis to ensure consistency between our CP/CPS and our linting mechanism due 6/13/2025.
    • Action Item #4 - Convert CPS into Markdown – due date 9/1/2025
    • Action Item #5 – Ensure consistency between our PC/CPS and our linting mechanism due date TBD
    • Action Item #8 – Clarify uses when keyEncipherment may be set (RSA) versus must not be set (ECC) – due 6/13/2025
  • 2025-06-13: MSP made an extension to the action items below; update not posted within the required 3‑day window
    • Action item #2 – extended from 6/13/2025 to 6/27/2025
    • Action item #3 – extended from 6/13/2025 to 6/20/2025
    • Action Item #8 – extended from 6/13/2025 to 6/27/2025
  • 2025-06-20: MPS made second extension for the items below; update not posted within the required 3‑day window
    • Action item #2 – extended from 6/27/2025 to 7/3/2025
    • Action Item #8 – extended from 6/27/2025 to 6/30/2025
  • 2025-06-20: Due date for Action Item #5 was published (9/30/2025)
  • 2025-06-27: MPS made a third extension for the items below; update not posted within the required 3‑day window
    • Action Item #8: extended from 6/30/2025 to 7/3/2025
  • 2025-08-29: MSP made an extension to the action item below; update not posted within the required 3‑day window
    • Action Item #4 – extended from 9/1/2025 to 9/30/2025
  • 2025-09-19: MPS made extension to the item below; update not posted within the required 3‑day window
    • Action Item #5 – extended from 9/30/2025 to 12/31/2025
  • 2025-09-26: MPS made a second extension to the item below; update not posted within the required 3-day window
    • Action Item #4 - extended from 9/30/2025 to 10/15/2025
  • 2025-10-10: MPS made a third extension for the items below; update not posted within the required 3‑day window
    • Action Item #4 - extended from 10/15/2025 to 11/15/2025
  • 2025-11-14: MPS made a fourth extension for the items below; update not posted within the required 3‑day window
    • Action Item #4 - extended from 11/15/2025 to 12/9/2025
  • 2025-12-05: MPS made a second extension for the items below
    • Action Item #5 – extended from 12/31/2025 to 2/28/2026
  • 2026-02-27: MPS made a third extension for the item below; update not posted within the required 3‑day window
    • Action Item #5 - extended from 2/28/2026 to 3/13/2026
  • 2026-02-27: A third-party source provided a comment to the most recent extension. Upon further discussion it was determined that we had a non-compliance against CCADB IRG.
  • 2026-03-04: Bugzilla 2021175 was opened to track the non-compliance for both bugs.

Bugzilla 1965612

  • 2025-05-23: The full incident report for this bug was reported that included the following action items (as they pertain to this bug):
    • Action Item #1: Revoke impacted certificate – due 11/15/2025
    • Action Item #2: Migrate cert issuance to use partitioned CRLs – due 11/15/2025
    • Action Item #3: Standup cross-signed warm stand bys – due TBD
    • Action Item #6: Exercise and refine mass revocation playbook – due 7/27/2025
  • 2025-05-30: Due Date for Action Item #3 set to 9/30/2025
  • 2025-06-13: MSP made an extension to the action item below; update not posted within the required 3‑day window
    • Action Item #6 – extended from 7/27/2025 to 9/1/2025
  • 2025-08-08: MPS added the following action items:
    • Action Item #9: Complete migration of all customers to G2 ICAs with CRL partitioning – set due date to 2/16/2026
    • Action Item #10: Develop/Publish plan for regular ICA rotation – due TBD
  • 2025-08-15: Due Date for Action Item #10 set to 10/17/2025.
  • 2025-09-19: MSP made an extension to the action item below; update not posted within the required 3‑day window
    • Action Item #3 – extended from 9/30/2025 to 11/30/2025
  • 2025-10-10: MPS made an extension to the action item below; update not posted within the required 3-day window
    • Action Item #10 – extended from 10/17/2025 to 12/31/2025
  • 2025-11-14: MSP made an extension to the action item below; update not posted within the required 3‑day window
    • Action Item #1 – extended from 11/15/2025 to 3/6/2026
  • 2025-12-12: MSP made a second extension to the action item below; update not posted within the required 3‑day window
    • Action Item #10 – extended from 12/31/2025 to 2/1/2026
  • 2026-01-23: MPS made a third extension to the action item below; update not posted within the required 3-day window
    • Action Item #10 – extended from 2/01/2026 to 3/6/2026
  • 2026-02-27: Upon further investigation it was found that the issues found in Bugzilla 1986829 also applied to this bug.
  • 2026-03-04: Bugzilla 2021175 was opened to track the non-compliance for both bugs.

Related Incidents

Bug Date Description
1965612 2025-05-09 This bug is relevant as the delayed notification occurred in this bug.
1962829 2025-04-25 This bug is relevant as the delayed notification occurred in this bug.

Root Cause Analysis

Contributing Factor #: Reliance on manual oversight

  • Description:
    MPS’ Bugzilla governance process lacked sufficient controls to ensure that any extensions to repair actions are disclosed within 3 days. As a result, we had multiple instances of delayed disclosure of extensions throughout this period.
  • Timeline:
    • 2025-06-13: The first instance of this non-compliance began under Bugzilla 1962829
    • 2026-02-27: A third party relying party commented in Bugzilla 1962829 inquiring about our most recent extension.
    • 2026-03-04: MPS conducted an analysis/investigation of all active bugs for instances of this non-compliance occurred.
    • 2026-03-04: Bugzilla 2021175 was opened to track this non-compliance
  • Detection:
    This delay was identified by a third party within the comments of a different bug.
  • Interaction with other factors: N/A
  • Root Cause Analysis methodology used: 5 whys

Lessons Learned

  • What went well:
    • N/A

  • What didn’t go well:
    • There were multiple instances in which action item delay updates were not published within the required three‑day timeframe.

  • Where we got lucky:
    • The issue was limited to the delay in reporting action item updates.

  • Additional: N/A

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Update internal Bugzilla governance process to include checks for timely reporting of action item delays/completion/ Prevent Root Cause # 1 Effectiveness will be measured by demonstrating that future action item extensions or timeline changes are publicly disclosed within three days with proper justification, as evidenced by Bugzilla comment timestamps. TBD New

Appendix

The 'Related incidents' section of this report lists only two bugs that are the same bugs identified in the incident description. Can I understand from this that in the last 2 years, Microsoft PKI Services is the only CA that has failed to update incident reports in a timely manner relating to action items?

Weekly Status Update

No major changes to report. We will provide an ETA for the action item defined in the Full Incident Report in the upcoming status update.

(In reply to Wayne from comment #1)

We're past 14 days with no final incident report.

I will note that it has been more than 72 hours since Microsoft were made aware they are required to raise an incident report for 'No Final Incident Report after 14 days'. That is an additional incident for 'No Preliminary Incident Report after 72 hours'.

(In reply to Microsoft PKI Services from comment #2)

Root Cause Analysis

Contributing Factor #: Reliance on manual oversight
...

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Update internal Bugzilla governance process to include checks for timely reporting of action item delays/completion/ Prevent Root Cause # 1 Effectiveness will be measured by demonstrating that future action item extensions or timeline changes are publicly disclosed within three days with proper justification, as evidenced by Bugzilla comment timestamps. TBD New

Ironically there is no 'Root Cause #1' despite reliance on manual oversight. There was little point in rushing out this incident report before it was ready once you went past the 14 day deadline.

Response to Comment 3 - bsmith

The 'Related incidents' section of this report lists only two bugs that are the same bugs identified in the incident description. Can I understand from this that in the last 2 years, Microsoft PKI Services is the only CA that has failed to update incident reports in a timely manner relating to action items?

While there are other Bugzilla incidents that involve different disclosure or communication timing issues—such as delayed posting of a full incident report or delayed responses to comments, we did not find other Bugzilla incidents opened specifically for this issue, failure to provide timely public status updates after an action item delay was known, within the past two years. Below are a few examples of other communication issues:

Response to Comment 5 - Wayne

(In reply to Wayne from comment #1)

We're past 14 days with no final incident report.

I will note that it has been more than 72 hours since Microsoft were made aware they are required to raise an incident report for 'No Final Incident Report after 14 days'. That is an additional incident for 'No Preliminary Incident Report after 72 hours'.

We acknowledge that the Full Incident Report was 4 hours and 31 minutes over the 14 day timeline. Our original interpretation calculated the due date based on calendar days, which is why we did not open a bug sooner. Upon further investigation into how other CAs have handled these types of situations in the past, it became evident that we should interpret these requirements down to the minute. We have opened these bugs:

Weekly Status Update

We have updated the due date for Action Item #1 and have marked it complete. All action items associated to this bug are now complete and we will be posting our closure report soon.

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Update internal Bugzilla governance process to include checks for timely reporting of action item delays/completion/ Prevent Root Cause # 1 Effectiveness will be measured by demonstrating that future action item extensions or timeline changes are publicly disclosed within three days with proper justification, as evidenced by Bugzilla comment timestamps. 2026-03-21 Complete

Report Closure Summary

  • Incident description:
    Microsoft PKI Services (MPS) failed to update open incident reports within 3 days of action items being changed, completed, or delayed. This specifically impacts Bug 1962829 and Bug 1965612, where action item timeline changes and status updates were not posted within the required 3-day window. From June 2025 to February 2026, in some cases when action item due dates were extended, updates were not posted to these bugs within the required timeframe. Specific instances are included in the timeline of the full incident report.

  • Incident Root Cause(s):
    The delayed disclosures were caused by MPS' Bugzilla response process where action item ETA updates were not reviewed on a daily basis, resulting in updates not being consistently disclosed within the required timeframe.

  • Remediation description:
    MPS has updated its internal Bugzilla governance procedures to include daily monitoring of action item status, incorporated into the standard responsibilities of the 24/7 on-call engineer rotation. Detection of potential action item slippage initiates a process to publish an update within the required timeline.

  • Commitment summary:
    MPS will continue to follow its updated Bugzilla governance procedures to support timely public disclosure of action item timeline changes within open incident reports, consistent with CCADB Incident Reporting Guidelines. These procedural updates are intended to improve ongoing governance of incident reporting activities.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Whiteboard: [ca-compliance] [policy-failure] → [close on 2026-04-02] [ca-compliance] [policy-failure]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-04-02.

Flags: needinfo?(incident-reporting)
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Flags: needinfo?(incident-reporting)
Resolution: --- → FIXED
Whiteboard: [close on 2026-04-02] [ca-compliance] [policy-failure] → [ca-compliance] [policy-failure]
You need to log in before you can comment on or make changes to this bug.