1990801 - Microsoft: improper disclosure of CRL

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

For the intermediate "Microsoft TLS G2 ECC CA OCSP 06" (Salesforce record ID 001TO00000VJZ54YAH), Microsoft has disclosed a "JSON Array of Partitioned CRLs" which is not a well-formed JSON array.

Comment 1

[Microsoft PKI Services]

Preliminary Incident Report

Summary

• Incident description: Microsoft PKI Services created twelve (12) new CAs recently and added them to CCADB. While updating the metadata about the CAs, on 2025-09-23 at ~12:15PM Pacific Time, one of the fields related to the partitioned CRL of the “Microsoft TLS G2 ECC CA OCSP 06” CA was incorrectly formatted. The partitioned CRL field was missing bracket symbols (“[“ and “]”) at the beginning and end of the CRL string to create a properly formed JSON array. We were notified on 2025-09-25 at ~6:15AM Pacific Time via our problem reporting Email by a user named “kidmin” of the issue.
  By 2025-09-2025 at ~8:13AM Pacific Time we had investigated and resolved the issue in CCADB specific to the formatting of the JSON array of the partitioned CRL for this CA.
• Relevant policies:
  • TLS Baseline Requirements Section 4.9.7 requires that the CA must make a CRL available within 24 hours of issuing its first Certificate (generate and publish a full or partitioned CRL)
  • CCADB Policy Section 6.2 within 7 days of CA issuing its first certificate, CA Owner MUST disclose a SON Array of Partitioned CRL URLs.
• Source of incident disclosure:
  • Problem reporting email from a user named “kidmin”
  • Submission of this Bugzilla from Andrew Ayer

Comment 2

[Wayne]

Might I recommend reading up on CCADB's Incident Reporting Guidelines:

Note: If notified of an incident by an external, third party reporter, please respect their privacy by only disclosing their name if affirmatively approved to do so (e.g., say “we received a report from a community member” instead of explicitly naming individuals). Fields marked with an asterisk (*) are not required in Preliminary Incident Reports if submitted by a third party reporter.

There is no need for anyone to know the username of someone sending in a certificate problem report.

Comment 3

[Microsoft PKI Services]

(In reply to Wayne from comment #2)

Might I recommend reading up on CCADB's Incident Reporting Guidelines:

Note: If notified of an incident by an external, third party reporter, please respect their privacy by only disclosing their name if affirmatively approved to do so (e.g., say “we received a report from a community member” instead of explicitly naming individuals). Fields marked with an asterisk (*) are not required in Preliminary Incident Reports if submitted by a third party reporter.

There is no need for anyone to know the username of someone sending in a certificate problem report.

Thank you for the reminder.

Comment 4

[Microsoft PKI Services]

Full Incident Report

Summary

-CA Owner CCADB unique ID: A002577

-Incident description: Microsoft PKI Services created twelve (12) new CAs recently and added them to CCADB. While updating the metadata about the CAs, on 2025-09-23 at ~12:15PM Pacific Time, one of the fields related to the partitioned CRL of the “Microsoft TLS G2 ECC CA OCSP 06” CA was incorrectly formatted. The partitioned CRL field was missing bracket symbols (“[“ and “]”) at the beginning and end of the CRL string to create a properly formed JSON array. We were notified of the issue on 2025-09-25 at ~6:15AM Pacific Time via our problem reporting Email by a user. By 2025-09-25 at ~8:13AM Pacific Time we had investigated and resolved the issue in CCADB specific to the formatting of the JSON array of the partitioned CRL for this CA.

-Timeline Summary:

• Non-compliance start date: 2025-09-23 ~12:15 PM PST
• Non-compliance identified date: 2025-09-25 ~06:15 AM PST
• Non-compliance end date: 2025-09-25 ~08:13 AM PST

-Relevant policies:

• CCADB Policy Section 6.2  Within 7 days of CA issuing its first certificate, CA Owner MUST disclose the URL of a full and complete Certificate Revocation List (CRL); or a JSON Array of Partitioned CRL URLs.

-Source of incident disclosure: Microsoft PKI Services was notified via Certificate Problem Reporting email on 2025-09-25 ~6:15 AM PST.

Impact

• Total number of certificates: 0
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: N/A
• Incident heuristic: N/A
• Was issuance stopped in response to this incident, and why or why not?: N/A
• Analysis: N/A
• Additional considerations: N/A

Timeline

-2025-09-23 at ~12:15PM Pacific Time – MS PKI Services Added 12 new CA certs to CCADB and updated Metadata (including CRL information) ** -2025-09-25 at ~6:15AM Pacific Time – MS PKI Services was notified via email Problem Report that there was an issue with 1 of the 12 Partitioned JSON Array’s that we updated in CCADB
-2025-09-25 at ~8:13AM Pacific Time – MS PKI Services investigated the Problem Report and resolved the JSON Array issue for the CA in question.

Related Incidents

Bug	Date	Description
1818833 - IdenTrust: Inaccurate CRL Details in CCADB	2023-03-20	Similar issue, in that CRLs were missing (versus this issue where they were incorrectly formatted).
1700809 - Microsoft PKI Services: Failure to disclose Unconstrained Intermediate within 7 Days	2021-03-24	MPS failed to update CCADB in 7 days for CA creation of Eight CAs. MPS began to investigate using the API to update CCADB. This feature may have helped in mitigating this bug.

Root Cause Analysis

-Contributing Factor #1: Validating the Posted JSON prior to updating CCADB
-Description: We did not do formal validation of the Posted JSON, before we posted in CCADB.
-Timeline: the team created a script to generate the list of partitioned CRLs with all of the correct formatting needed to post in CCADB, except the addition of opening and closing brackets (“[“, “]”). And we did not test that the entire string we were posting to CCADB was a valid JSON.
-Detection: This was the first time we had used this field in CCADB and we developed this new process. The new process dd not include verification steps using code or tooling that would be able to verify the JSON Array was well formed. Our new process did have human review that also failed to identify the missing brackets in one of the updated JSONs.
-Interaction with other factors: n/a
-Root Cause Analysis methodology used: 5 Whys

-Contributing Factor #2: Validating the Posted JSON after updating CCADB
-Description: We do not have monitoring tools that identify any errors in our partitioned CRL JSON updates to CCADB. We always use the “CA Task List” feature in CCADB to look for issues (for the metrics that CCADB highlights, for example “missing Full CRL” metric). However, the “CA Task List” does not highlight a failure mode like we had with this Bugzilla.
-Timeline: We were lucky that two separate folks identified our error and notified us, so we were able to repair it quickly.
-Detection: Similar analysis as in Root Cause #1.
-Interaction with other factors: n/a
-Root Cause Analysis methodology used: 5 Whys

Lessons Learned

-What went well: We successfully updated 11 of the 12 CA’s Partitioned CRL JSON Arrays that we updated.
-What didn’t go well: We had an issue with 1 of the 12 CA’s Partitioned CRL JSON Arrays that we updated.
-Where we got lucky: We were notified by the community (via CPR email) quickly with our error.
-Additional: n/a

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
All CAs that MPS creates contain at least a Full CRL and may also have partitioned CRLs. CCADB requires us to post one or the other, so we have made the decision to only post our Full CRLs for each CA. This will prevent this failure mode in the future. We only have 12 CAs that we have posted the JSON Array of the partitioned CRLs, so we will post their Full CRLs and remove the JSON Arrays.	Prevent	Root Cause # 1 and #2	We will continue to monitor the “CA Task List” feature in CCADB metrics, specifically the “missing Full CRL” metric, to ensure that all our CA’s have full CRLs listed that are correct and well formed.	2025-10-17	In Progress
MPS will add backlog item such that if we stop doing Full CRLs that we will have tools that perform a full validation of a partitioned CRL JSON Array prior to posting to CCADB	Prevent	Root Cause # 1 and #2	We will have a clear backlog item that is included in our quarterly planning process.	2025-10-17	In Progress
MPS will open a support ticket with CCADB to consider updating the partitioned CRL JSON Array field to check for a well-formed array before saving the field.	Mitigate	Root Cause # 1	We will initiate a support email/ticket with the CCADB support team.	2025-10-17	Value

Appendix

Comment 5

[Colin]

For the support ticket with CCADB action item, could this be better served as an enhancement request under the Common CA Database component here on Bugzilla?

Comment 6

[Microsoft PKI Services]

Response to Comment 5 - Colin

---

Thank you for the suggestion. We acknowledge your recommendation to file this as an enhancement request under the Common CA Database component and have proceeded with doing that. It can be found here: Bug# 1994886

Comment 7

[Microsoft PKI Services]

Weekly Status Update

---

All repair items have been completed. For action item #3 we followed the suggestion presented in Comment #5 and have opted to open an enhancement feature within Bugzilla.

Comment 8

[Microsoft PKI Services]

Report Closure Summary

• Incident description:
  Microsoft PKI Services created twelve (12) new CAs recently and added them to CCADB. While updating the metadata about the CAs, on 2025-09-23 at ~12:15PM Pacific Time, one of the fields related to the partitioned CRL of the “Microsoft TLS G2 ECC CA OCSP 06” CA was incorrectly formatted. The partitioned CRL field was missing bracket symbols (“[“ and “]”) at the beginning and end of the CRL string to create a properly formed JSON array. We were notified of the issue on 2025-09-25 at ~6:15AM Pacific Time via our problem reporting Email by a user. By 2025-09-25 at ~8:13AM Pacific Time we had investigated and resolved the issue in CCADB specific to the formatting of the JSON array of the partitioned CRL for this CA.

• Incident Root Cause(s):
  The issue stemmed from gaps in JSON validation before and after posting updates to CCADB. Initially, the team did not formally verify that the partitioned CRL JSON was well-formed, missing opening and closing brackets in the new process, which relied on human review rather than automated checks. After posting, there were no monitoring tools to detect malformed JSON, and CCADB’s “CA Task List” did not flag this failure mode. As a result, the error persisted until external parties identified it, highlighting the need for stronger pre-validation and post-update monitoring.

• Remediation description:
  Three remediation actions were fully implemented to address this issue: (1) posting only Full CRLs for all CAs and removing partitioned CRL JSON arrays to eliminate the failure mode, (2) adding a backlog item to ensure future processes include automated validation of partitioned CRL JSON arrays before posting to CCADB, and (3) filing a feature enhancement in Bugzilla to request CCADB validation checks on the partitioned CRL JSON field. All actions have been completed and tracked.

• Commitment summary:
  All identified action items were completed. Beyond these action items, we remain committed to continuous improvements in our tooling to minimize opportunities for human errors.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 9

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-11-03.

Comment 10

[Microsoft PKI Services]

Weekly Status Update

---

The closure report associated to this bug has been submitted. Please close if no other comments are provided.