Closed Bug 2031164 Opened 1 month ago Closed 2 days ago

Google Trust Services: Incomplete CRL Distribution Point URLs in CCADB for GTS Roots

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gts-external, Assigned: gts-external)

Details

(Whiteboard: [ca-compliance] [disclosure-failure])

Preliminary Incident Report

Summary

  • Incident description: The URLs disclosed in the Common CA Database (CCADB) for the Certificate Revocation Lists (CRLs) of Google Trust Services (GTS) Root CAs are not fully accurate.
  1. For GTS Roots R1-R4, the disclosed URLs are incomplete. They do not include the newer, shorter URLs (e.g., http://c.pki.goog/r/r1.crl) GTS now uses in the CRL Distribution Points of recently issued intermediate CA certificates, in addition to its older, longer URLs.
  2. For the GlobalSign Root CA R4 (GSR4), the disclosed URL (http://crl.pki.goog/root-r4.crl) is incorrect and appears to have never been used in any certificate issued by GSR4.

This means the CCADB entries do not reflect the "complete set of distinct HTTP URLs" present in the CRL Distribution Point extensions of all unexpired certificates issued by these CAs, as required by recent CCADB policy. The actual CRL files have always been available at all correct locations. Because revocation data was and is being served correctly to Relying Parties, no certificates were mis-issued and the incident can be considered contained.

  • Relevant policies:
  • CCADB Policy v2.1, Section 6.2: Effective 2026-03-20, requires disclosure of "the complete set of distinct HTTP URLs". The 14-day grace period for updating CCADB expired on 2026-04-03.
  • CCADB Policy v2.0, Section 6.2: Effective 2025-07-15, required the disclosed URL to "match exactly as they appear in the certificates issued". This policy was potentially violated for GSR4 since the disclosed URL was seemingly never used.
  • Source of incident disclosure: Internal discovery on 2026-04-10, following analysis of a similar incident reported by another CA.

GTS will publish a full incident report by 2026-04-24. We kindly request the nextUpdate field be set to 2026-04-24.

Flags: needinfo?(incident-reporting)
Assignee: nobody → gts-external
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [disclosure-failure] Next update 2026-04-24

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A004159
  • Incident description: Google Trust Services (GTS) was not fully compliant with the Common CA Database (CCADB) policy (https://www.ccadb.org/policy) regarding the disclosure of Certificate Revocation List (CRL) Distribution Point URLs. The URLs disclosed in CCADB for GTS Root CAs did not include the complete set of distinct HTTP URLs present in the crlDistributionPoints extensions of all unexpired certificates issued by these CAs. The shift from requiring "valid" URLs to a comprehensive listing of all CRL URLs present in unexpired certificates was not handled properly.
  • Timeline summary:
    • Non-compliance start date: 2025-07-29 for GSR4 (14 days after the CCADB Policy v2.0 effective date) and 2026-04-03 for GTS Roots R1-R4 (14 days after the CCADB Policy v2.1 effective date)
    • Non-compliance identified date: 2026-04-10
    • Non-compliance end date: 2026-04-13
  • Relevant policies:
    • CCADB Policy v2.0, Section 6.2: Effective 2025-07-15, requires the disclosed URL to "match exactly as they appear in the certificates issued".
    • CCADB Policy v2.1, Section 6.2: Effective 2026-03-20, requires disclosure of "the complete set of distinct HTTP URLs".
    • CCADB Policy v2.1, section 1: "CA Owners have an overarching responsibility to keep the information in the CCADB about themselves, their operations, and their certificates accurate, and to make updates in a timely fashion. Minimally, CA Owners with certificates included in a Root Store MUST ensure their information stored in the CCADB is kept up to date as changes occur. This responsibility includes the timely population of new data fields or values added to the CCADB. When a timeline is not defined for a requirement specified in this policy, updates MUST be submitted to the CCADB within 14 calendar days of an activity being completed."
  • Source of incident disclosure: Self reported

For GTS Roots R1-R4: shorter CRL URLs (e.g., http://c.pki.goog/r/r1.crl) included in GTS intermediate CA certificates issued from 2023 were not disclosed to CCADB.

For GlobalSign Root CA R4 (GSR4): the disclosed URL (http://crl.pki.goog/root-r4.crl) was not included in the crlDistributionPoints extension of any unexpired certificates issued by that CA.

GTS root CAs (including GSR4) only issue full CRLs (and no partitioned CRLs). The same full CRL of any given GTS root CA is available at multiple URLs. For example, the full CRL of GTS Root R1 is served at both http://crl.pki.goog/gtsr1/gtsr1.crl and http://c.pki.goog/r/r1.crl. The smaller version is a minor optimization to save space in certificates.

All CRL URLs disclosed to CCADB pointed to the right CRL file of the corresponding GTS root CA. All the relevant CRL files for each of the respective root CAs were served at the disclosed URLs. The issue was data incompleteness in CCADB.

Concretely, the following CRL URLs were disclosed to CCADB at the time of this incident:

While they should have been (this is now fixed):

GTS maintains automated checks to ensure that the expected CRL files are served at the expected URLs - including both the previously and newly disclosed set of URLs in CCADB - in the expected (DER-encoded) format, and that a set of URLs all point to the same canonical CRL file. This automation was implemented when setting up the shorter URLs.

Impact

  • Total number of certificates: N/A
  • Total number of "remaining valid" certificates: N/A
  • Affected certificate types: N/A
  • Incident heuristic: N/A
  • Was issuance stopped in response to this incident, and why or why not?: Issuance was not stopped as this was a CCADB data inaccuracy error that did not affect revocation information served at the disclosed URLs.
  • Analysis: The impact was limited to non-compliance with CCADB disclosure policies. There was no impact on the accessibility of revocation data. All CRL files remained available at all URLs. No certificates were mis-issued so revocation was not conducted.
  • Additional considerations: N/A

Timeline

All times are UTC.

2022-10-01 00:00 - CCADB CRL URL disclosure requirement becomes effective in Mozilla Root Store Policy 2.8 and Apple Root Store Policy - The requirement doesn't specify that the URL must match exactly one that appears in a certificate.
2023-12-13 09:00 - The first GTS intermediate CAs including shorter CRL URLs are issued.
2025-02-15 00:00 - CCADB CRL URL disclosure requirement becomes effective in Chrome Root Program policy 1.6 - The requirement doesn't specify that the URL must match exactly one that appears in a certificate.
2025-05-02 14:49 - CCADB Policy 2.0 draft policy is released for public feedback link.
2025-05-05 11:16 - GTS internal analysis ticket is filed to track the impact assessment of CCADB Policy 2.0.
2025-06-24 18:22 - GTS team member triages CCADB Policy 2.0 changes and notes the requirement to have "valid" CRL(s) for active intermediates. 2025-06-24 18:41 - GTS team member creates various sub-tickets to verify compliance with the identified CCADB Policy 2.0 changes, but does not create one to verify CRL URLs as they were deemed to be in compliance.
2025-07-15 00:00 - CCADB Policy v2.0 becomes effective.
2025-07-15 00:00 - Chrome Root Program Policy 1.7 becomes effective - It removes requirements redundant with CCADB Policy Version 2.0.
2025-07-29 00:00 - Start of non-compliance for GSR4 - 14 day deadline for keeping CCADB up to date is missed.
2025-12-20 00:13 - Preliminary Incident Report gets posted by GoDaddy to Bugzilla bug 2007216.
2025-12-20 00:36 - Internal shadow ticket for Bugzilla bug 2007216 gets created by GTS automation.
2026-01-14 18:25 - Bugzilla bug 2007216 gets triaged by a GTS team member incorrectly concluding that that GTS is compliant already and doesn't need to take any action. If the Bugzilla bug is still open and has recent updates, GTS keeps the internal shadow ticket open for some time, to monitor for updates.
2026-01-23 20:48 - CCADB Policy 2.1 draft policy is released for public feedback link.
2026-01-29 03:31 - Internal analysis ticket is filed to track the impact assessment of CCADB policy 2.1.
2026-02-19 17:17 - GTS team member triages CCADB Policy 2.1 changes and notes "We need to do an analysis so we know we are compliant" while reviewing the impact of the new "All Full CRL URIs for This Hierarchy" field.
2026-02-25 15:22 - Another GTS team member completes the secondary review and concludes: "My conclusions are similar to that of the primary review", but fails to create a sub-ticket to perform the CRL URL analysis.
2026-03-20 00:00 - CCADB Policy v2.1 becomes effective and CCADB adds support for multiple full CRL URLs.
2026-04-03 00:00 - Start of non-compliance for GTS Root R1-R4 - 14 day deadline for keeping CCADB up to date is missed.
2025-04-07 - CCADB access is granted to additional GTS team members, other than members of the Policy Authority. This was not done as a consequence of this incident. The timing is coincidental.
2026-04-09 16:37 - Bugzilla bug 2007216 gets accidentally re-triaged by another GTS team member because they did not realize it was triaged already. That second review identified that GTS may not be compliant and decided to file a ticket to verify.
2026-04-09 16:38 - GTS team member creates ticket to verify that CRL URLs of GTS R1-4 and GS R4 are disclosed to CCADB properly.
2026-04-10 05:33 - GTS team member concludes that the URLs of full CRL of GTS root CAs disclosed to CCADB are incomplete.
2026-04-10 05:44 - Incident management process is invoked. Facts and relevant requirements are collected.
2026-04-11 13:22 - The GTS Policy Authority determines this is a compliance incident.
2026-04-13 08:52 - The preliminary report is published to Bugzilla.
2026-04-13 17:12 - End of non-compliance - CCADB case updating CRL URLs for GTS R1-4 and GS R4 is submitted.
2026-04-13 17:15 - GTS team member finishes reviewing the CRL URLs of GTS intermediate CAs disclosed to CCADB and concludes that all URLs are correctly disclosed.
2026-04-20 10:07 - GTS team member adds a step to the post-ceremony checklist to verify that the CRL URL included in the CRLDP of newly issued intermediate CA certificates is disclosed in the CCADB entry of its issuer.

Related Incidents

Bug Date Description
2034360 2026-04-23 The full incident report isn't published yet. The preliminary report mentions an incomplete JSON Array of all Full CRL URLs
2007297 2025-12-21 URL discrepancy between CRLDP and CCADB is limited to URL "normalization" mismatches
2007219 2025-12-19 URL discrepancies between CRLDP and CCADB got introduced by updates in leaf certificate profiles
2007216 2025-12-19 CA produces both full and partitioned CRLs. Only the full CRL URLs were disclosed to CCADB, while the partitioned CRL URL are disclosed in the CRLDP
2007116 2025-12-19 Some of the CRL URLs reported to CCADB didn't match exactly as they appear in the certificates issued
2007105 2025-12-19 Disclosed CRL URLs used the https scheme, and for some URLs, a dedicated CRL subdomain was omitted
2007098 2025-12-19 URL discrepancy between CRLDP and CCADB is limited to an extra "/gs/" in the URL
2007089 2025-12-19 CA produces both full and partitioned CRLs. Only the full CRL URLs were disclosed to CCADB, while the partitioned CRL URL are disclosed in the CRLDP
2007066 2025-12-18 URL discrepancy between CRLDP and CCADB is limited to a "cdp." vs. "cdn." subdomain label

Root Cause Analysis

Contributing Factor #1: Informal Tracking and Assignment of CCADB Policy Change Analysis

  • Description: When new CCADB policies or versions were released, the internal process for analyzing the impact and required actions lacked formal, granular tracking. While analysis tickets were created, the specific tasks to verify compliance with each section or change within the policy update were not always individually assigned to, and tracked by, the most appropriate Subject Matter Experts (SMEs). This resulted in some requirements, like the changes to CRL URL disclosures in Section 6.2, not undergoing a sufficiently detailed review and verification by the engineers most familiar with the CRL infrastructure.
  • Timeline: This contributing factor existed since GTS started monitoring CCADB policy updates.
  • Detection: Internal review process identified the non-compliance.
  • Interaction with other factors: This lack of formal assignment and tracking meant that the lower priority afforded to CCADB changes (Factor #2) more easily led to items being missed.
  • Root Cause Analysis methodology used: 5-Whys

Contributing Factor #2: Concentrated Responsibility and High Burden on Policy Authority due to Restricted CCADB Access

  • Description: Access to CCADB was historically limited to members of the Policy Authority, following guidance from CCADB administrators to minimize logins. While intended to enhance security, this placed the entire burden of managing CCADB entries and interpreting the technical implications of CCADB policy changes solely on the Policy Authority. This concentration of responsibility on a small group, already managing significant policy and compliance workloads, created a bottleneck and sometimes limited the depth of technical cross-verification against the live certificate infrastructure.
  • Timeline: This contributing factor existed since GTS started monitoring CCADB policy updates.
  • Detection: Internal review process identified the non-compliance.
  • Interaction with other factors: The perception of CCADB changes as requiring less technical scrutiny meant that the informal processes (Factor #1) were not questioned or reinforced. The limited access also meant fewer opportunities for discrepancies to be incidentally spotted by engineers working on the systems.
  • Root Cause Analysis methodology used: 5-Whys

Contributing Factor #3: Changes in intermediate CAs require updating root CA entries in CCADB

  • Description: GTS procedures include steps to disclose all newly issued intermediate CAs to CCADB and populate all their fields, including their CRL URLs. These updates all happen on the intermediate CA entry in CCADB, and not on the entry of its issuer (the root CA). Having to update a separate CCADB entry when creating another one is counter-intuitive. This could be automated by CCADB by parsing the CRLDP of disclosed intermediate CA certificates.
  • Timeline: This contributing factor has always existed.
  • Detection: While reviewing Bugzilla bug 2007216.
  • Interaction with other factors: This lack of awareness was exacerbated by the fact that most team members didn't have access to CCADB (Factor #2).
  • Root Cause Analysis methodology used: 5-Whys

Lessons Learned

  • What went well:

    • The issue was identified internally.
    • Once identified, the GTS team member escalated the issue promptly according to GTS' incident management process.

  • What didn’t go well:

    • The small, but significant change from accurate CRLs to all CRLs was not recognized as a change that impacted GTS.

  • Where we got lucky:

    • All CRL URLs disclosed to CCADB pointed to the right CRL file of the corresponding GTS root CA, meaning that revocation checking mechanisms were not impaired.
    • Bugzilla bug 2007216 was accidentally re-triaged, which surfaced the non-compliance.

  • Additional:

N/A

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Grant CCADB access to GTS team members other than the Policy Authority Prevent Root Cause #2 Expand access to CCADB to more SMEs in GTS 2026-04-07 Complete
Fix disclosed CRL URLs of GTS root CAs Mitigate Root Cause #3 The complete set of distinct CRL URLs is disclosed to CCADB 2026-04-13 Complete
Add post-ceremony activity to update CCADB CRL URLs when issuing new intermediate CAs Prevent Root Cause #3 The GTS post-ceremony checklist includes the corresponding step 2026-04-20 Complete
Update the CCADB policy review process to use the same level of rigor as CABF ballot reviews to ensure that the changes are analyzed correctly Prevent Root Causes #1 and #2 Confirmation of consistency across all review types 2026-05-15 Ongoing
Update the CA incident review process to ensure the most appropriate SMEs are involved Detect Root Cause #2 Future reviews of CA incidents are assigned more appropriately 2026-05-15 Ongoing
Verify compliance with the current (v2.1) CCADB policy to ensure GTS didn't miss any other changes Mitigate Root Cause #2 GTS will review CCADB policy v2.1 as if it was a policy update to verify the updated triage procedure and the distribution of analysis among SMEs 2026-05-15 Ongoing

Appendix

We will monitor this thread for any comments or questions and kindly ask that the nextUpdate field be set to 2026-05-15.

Flags: needinfo?(incident-reporting)
Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [disclosure-failure] Next update 2026-04-24 → [ca-compliance] [disclosure-failure] Next update 2026-05-15
Flags: needinfo?(incident-reporting)

We have mistakenly re-added the needInfo flag. The flag should be removed from this bug for the time being.

Flags: needinfo?(incident-reporting)
Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Grant CCADB access to GTS team members other than the Policy Authority Prevent Root Cause #2 Expand access to CCADB to more SMEs in GTS 2026-04-07 Complete
Fix disclosed CRL URLs of GTS root CAs Mitigate Root Cause #3 The complete set of distinct CRL URLs is disclosed to CCADB 2026-04-13 Complete
Add post-ceremony activity to update CCADB CRL URLs when issuing new intermediate CAs Prevent Root Cause #3 The GTS post-ceremony checklist includes the corresponding step 2026-04-20 Complete
Update the CCADB policy review process to use the same level of rigor as CABF ballot reviews to ensure that the changes are analyzed correctly Prevent Root Causes #1 and #2 Confirmation of consistency across all review types 2026-05-15 Complete
Update the CA incident review process to ensure the most appropriate SMEs are involved Detect Root Cause #2 Future reviews of CA incidents are assigned more appropriately 2026-05-15 Complete
Verify compliance with the current (v2.1) CCADB policy to ensure GTS didn't miss any other changes Mitigate Root Cause #2 GTS will review CCADB policy v2.1 as if it was a policy update to verify the updated triage procedure and the distribution of analysis among SMEs 2026-05-15 Complete

We’ve expanded access to CCADB to more SMEs in GTS, confirmed that the complete set of distinct URLs is disclosed to CCADB, and updated our post-ceremony checklist to ensure we update CCADB CRL URLs when issuing new intermediate CAs. We’ve also updated our CCADB policy review process to match the level of rigor applied to reviewing CABF ballots to ensure that changes are analyzed correctly. We’ve adjusted the CA incident review process to leverage internal tooling to ensure appropriate SMEs are assigned to specific tasks to verify compliance with section or change within future CCADB policy or version updates. Additionally, CA Policy Authority and CA Engineers holistically reviewed each section of the current CCADB policy (v2.1) and confirmed compliance.

We will continue to monitor this thread for any comments or questions and request that the nextUpdate field be set to 2026-05-22.

Flags: needinfo?(incident-reporting)
Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [disclosure-failure] Next update 2026-05-15 → [ca-compliance] [disclosure-failure] Next update 2026-05-22

Report Closure Summary

Incident description:

Google Trust Services (GTS) failed to disclose the complete set of CRL Distribution Point URLs in CCADB as required by recent policy updates. Specifically, shorter URLs used in recent intermediate certificates were missing for GTS Roots R1-R4, and an incorrect URL was listed for GlobalSign Root CA R4.

Incident Root Cause(s):

  • Process Gaps in Policy Review: The internal process for tracking CCADB policy changes was too informal, failing to assign specific technical requirements to the appropriate subject matter experts for detailed verification.
  • Restricted Access and Bottlenecks: Access to CCADB was limited to a small Policy Authority group, creating a bottleneck that prevented engineers from cross-verifying entries against the live infrastructure.
  • Intermediate CA Update Requirements: Existing procedures focused on populating data for newly issued intermediate CAs, but did not account for the requirement to simultaneously update the CRL fields of the parent root CA entry.

Remediation description:

GTS has corrected the CCADB entries to include the complete set of distinct CRL URLs and granted CCADB access to additional Subject Matter Experts. Procedural updates include adding a post-ceremony checklist step for CCADB updates and increasing the rigor of CCADB policy reviews to match CA/Browser Forum ballot standards.

Commitment summary:

GTS will continue to monitor the incident thread for any further comments or questions.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Flags: needinfo?(incident-reporting)

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-05-29.

Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [disclosure-failure] Next update 2026-05-22 → [close on 2026-05-29] [ca-compliance] [disclosure-failure]
Status: ASSIGNED → RESOLVED
Closed: 2 days ago
Resolution: --- → FIXED
Whiteboard: [close on 2026-05-29] [ca-compliance] [disclosure-failure] → [ca-compliance] [disclosure-failure]
You need to log in before you can comment on or make changes to this bug.