Microsoft PKI Services: Vulnerability Management Exception Tracking
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: Dustin.Hollenback, Assigned: Dustin.Hollenback)
Details
(Whiteboard: [ca-compliance] [audit-finding])
Incident Report
Summary
During the audit period ending 2024-03-31, a qualified opinion was made by our auditors and includes a finding that there are deficiencies with how vulnerability mitigation plans and timelines are documented within the Microsoft PKI Services Vulnerability Management process.
Language from the qualified opinion:
“During the Period, upon the discovery of a Critical Vulnerability, any of the following did not occur within 96 hours:
- Remediation of the vulnerability;
- Creation and implementation of a plan to mitigate the vulnerability; or
- Documentation of the factual basis for Management’s determination that the vulnerability does not require remediation.
This caused WebTrust Principles and Criteria for Certification Authorities – Network Security – Version 1.0 to not be met.”
When there were exceptions to the 96-hour remediation timeline, the vulnerability mitigation plan and timelines were known, but not consistently documented and attached to the vulnerability tracker. Also, the vulnerability dashboard did not display the remediation plan when there was an exception.
Impact
When patches required more than 96-hours to remediate, Microsoft PKI Services could not produce evidence for each asset and vulnerability of a documented mitigation plan.
The current exception tracking process is very difficult for an outsider to follow because it relies heavily on multiple disparate tools/reports. Microsoft PKI Services completely agrees that there is a need to improve and consistently document the mitigation plan for exceptions to the 96-hour remediation timeline as well as improve tooling to make it easier to follow the status of each vulnerability.
Timeline
All times are UTC.
2024-06-28: Auditor provided draft audit reports that contained the qualified opinion
Root Cause Analysis
The tracking of exceptions to the 96-hour remediation timeline has used a process that was difficult to follow for someone other than the members who directly perform vulnerability management.
Vulnerability mitigation plans for specific exceptions were inconsistently tracked. They were known by the team responsible for vulnerability management, but were not consistently documented.
The dashboard for tracking vulnerabilities does not include a way to track exceptions and document plans inline.
Lessons Learned
What went well
Vulnerabilities have been remediated based on a well-known process within the team. Assets are scanned daily for vulnerabilities. We know the processes for vulnerability mitigation are sound, but require additional documentation and improved tooling, especially related to tracking mitigation plans for exceptions to the 96-hour patching timeline.
What didn't go well
The written process and the tracking of exceptions to the 96-hour remediation timeline had gaps. Because a mitigation plan was not specifically tracked with each vulnerability, it made it impossible to verify that the plan was being followed.
Where we got lucky
- N/A
Action Items
Microsoft PKI Services will investigate further over the next week to determine commitment timelines for the below Action Items.
| Action Item | Kind | Due Date |
|---|---|---|
| Document mitigation plans for 96h exceptions | Process | TBD |
| Expand vulnerability tracking dashboard to include additional fields, including mitigation plan | Process | TBD |
Appendix
Details of affected certificates
No certificates were impacted by this process issue.
|
||
Updated•5 months ago
|
|
Assignee | |
Comment 1•4 months ago
|
||
Updated Incident Report
Summary
During the audit period ending 2024-04-30, a qualified opinion was made by our auditors and includes a finding that there are deficiencies with how vulnerability mitigation plans and timelines are documented within the Microsoft PKI Services Vulnerability Management process.
Language from the qualified opinion:
“During the Period, upon the discovery of a Critical Vulnerability, any of the following did not occur within 96 hours:
- Remediation of the vulnerability;
- Creation and implementation of a plan to mitigate the vulnerability; or
- Documentation of the factual basis for Management’s determination that the vulnerability does not require remediation.
This caused WebTrust Principles and Criteria for Certification Authorities – Network Security – Version 1.0 to not be met.”
When there were exceptions to the 96-hour remediation timeline, the vulnerability mitigation plan and timelines were known, but not consistently documented and attached to the vulnerability tracker. Also, the vulnerability dashboard did not display the remediation plan when there was an exception.
Impact
When patches required more than 96-hours to remediate, Microsoft PKI Services could not produce evidence for each asset and vulnerability of a documented mitigation plan.
The current exception tracking process is very difficult for an outsider to follow because it relies heavily on multiple disparate tools/reports. Microsoft PKI Services completely agrees that there is a need to improve and consistently document the mitigation plan for exceptions to the 96-hour remediation timeline as well as improve tooling to make it easier to follow the status of each vulnerability.
Timeline
All times are UTC.
2024-06-28: Auditor provided draft audit reports that contained the qualified opinion
2024-07-03: Bugzilla bug opened
Root Cause Analysis
The tracking of exceptions to the 96-hour remediation timeline has used a process that was difficult to follow for someone other than the members who directly perform vulnerability management.
Vulnerability mitigation plans for specific exceptions were inconsistently tracked. They were known by the team responsible for vulnerability management, but were not consistently documented.
The dashboard for tracking vulnerabilities does not include a way to track exceptions and document plans inline.
Lessons Learned
What went well
Vulnerabilities have been remediated based on a well-known process within the team. Assets are scanned daily for vulnerabilities. We know the processes for vulnerability mitigation are sound, but require additional documentation and improved tooling, especially related to tracking mitigation plans for exceptions to the 96-hour patching timeline.
What didn't go well
The written process and the tracking of exceptions to the 96-hour remediation timeline had gaps. Because a mitigation plan was not specifically tracked with each vulnerability, it made it impossible to verify that the plan was being followed.
Where we got lucky
- N/A
Action Items
Microsoft PKI Services will investigate further over the next week to determine commitment timelines for the dashboard changes. We are still researching which dashboard tools to use for the best long-term solution. Some solutions are faster than others to implement, but may not be the best long-term tool.
| Action Item | Kind | Due Date |
|---|---|---|
| Document mitigation plans for 96h exceptions | Process | 2024-07-25 |
| Expand vulnerability tracking dashboard to include additional fields, including mitigation plan | Process | TBD |
Appendix
Details of affected certificates
No certificates were impacted by this process issue.
|
|
Assignee | |
Comment 2•4 months ago
|
||
The Microsoft PKI Services team was able to complete one Action Item and can commit to a Due Date for the remaining Action Item. We may complete this sooner, but have a few tasks related to data sources for the tracking dashboard that may take a few more weeks to finalize.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Document mitigation plans for 96h exceptions | Process | Completed |
| Expand vulnerability tracking dashboard to include additional fields, including mitigation plan | Process | 2024-08-12 |
With this update, would it be possible to set the Next Update field to 2024-08-13? Of course, if there are any comments from others before then, we would be address them more quickly.
|
|
||
Updated•4 months ago
|
|
|
Assignee | |
Comment 3•4 months ago
|
||
We were able to complete the final Action Item. If there are no additional questions, can this bug be closed? Thank you.
| Action Item | Kind | Due Date |
|---|---|---|
| Document mitigation plans for 96h exceptions | Process | Completed |
| Expand vulnerability tracking dashboard to include additional fields, including mitigation plan | Process | Completed |
|
|
||
Comment 4•4 months ago
|
||
I'll close this on Friday, 9-Aug-2024, unless discussions are needed.
|
|
||
Updated•3 months ago
|
Description
•