Closed Bug 1991215 Opened 7 months ago Closed 5 months ago

IdenTrust: ICA with invalid CDP

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: roots, Assigned: roots)

Details

(Whiteboard: [ca-compliance] [ca-misissuance])

Preliminary Incident Report

Summary

On September 15, 2025, we issued a new enterprise subordinate CA certificate. However, the following day, we identified issues with the Certificate Revocation List (CRL). Upon investigation, we discovered that the CRL Distribution Point (CDP) specified in both the Authority Information Access (AIA) and CRL extensions was incorrect—the certificate pointed to itself as the issuer of the CRL.

To remediate this mis-issuance, the subordinate CA was revoked on September 17, 2025.

Subsequently, we were notified by CCADB that the revoked subordinate CA is not listed in the published CRL, indicating a potential compliance issue.

Relevant policies

Baseline Requirements Section 4.9.1.2 - Circumstances for Revoking a CA Certificate: (4) The Issuing CA obtains evidence that the Certificate was misused and Section 7.1.2 (Certificate Content and Extensions) - The values in certificate fields are expected to be functional

Source of Incident Disclosure

CCADB.

Impact

The issue was limited to the newly issued TLS subordinate CA, which has since been revoked. No end-entity certificates were issued from this CA, and therefore, no relying parties were affected.

A complete incident report will be submitted by October 10, 2025.

Assignee: nobody → roots
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [ca-misissuance]

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000036
  • Incident description:
    On September 15, 2025, we issued a new Subordinate CA certificate. The following day, we discovered that the location of the Certificate Revocation List (CRL) in CRL Distribution Point (CDP) and location of CA Issuers in the Authority Information Access (AIA) and CRL extensions were incorrect and not functional.
    To remediate this mis-issuance, the Subordinate CA was revoked on September 17, 2025.
  • Timeline summary:
    All times are UTC.
    • 2025-09-08 :22:23 UTC – Issued pre-production Subordinate CA
    • 2025-09-15: 19:42 UTC – Issued production Subordinate CA
    • 2025-09-16: 22:00 UTC – Disclosed production Subordinate CA in CCADB
    • 2025-09-16: 19:14 UTC – Identified issue with production Subordinate CA and corrected the certificate profile
    • 2025-09-17: 20:14 UTC – Issued the new Subordinate CA in production
    • 2025-09-18: 01:43 UTC – Revoked the misissued Subordinate CA
    • 2025-09-18: 23:30 UTC – Updated CCADB to reflect revocation and disclosed the new production Subordinate CA
    • 2025-09-24: 16:17 UTC – Received notification from CCADB that the revoked Subordinate CA is not present in the root CRL
    • 2025-09-26: 19:20 UTC – Disclosed preliminary incident report

Relevant policies

Baseline Requirements
Section 4.9.1.2 - Circumstances for Revoking a CA Certificate: (4) The Issuing CA obtains evidence that the Certificate was misused
Section 7.1.2 - Certificate Content and Extensions: - The values in certificate fields are expected to be functional

Source of Incident Disclosure

CCADB

Impact

The issue was limited to a newly issued TLS Subordinate CA, which has since been revoked. No end-entity certificates were issued from this CA, and therefore, no relying parties were affected.

  • Total number of certificates affected: 1
  • Total number of "remaining valid" certificate : 0
  • Affected certificate types: TLS Subordinate CA
  • Incident heuristic: crt.sh ID:21201367454
  • Was issuance stopped in response to this incident, and why or why not?:
    Yes
  • Analysis: See Root Cause Analysis below
  • Additional considerations: N/A

Timeline

Refer to the “Timeline Summary” section above

Related Incidents

There is no record of a prior incident that mirrors this specific issue.

Root Cause Analysis

During the creation of the certificate profile for the new Subordinate CA, an incorrect value string was inadvertently copied and pasted into the AIA (Authority Information Access) extension and CRL DP extension.

Contributing Factors
Lack of enforcement to the certificate profile approval Workflow:
The certificate profile was reviewed by only one individual and did not undergo the established approval process which prevented correction to the correct location prior to issuance.

Lessons Learned

A systematic enforcement mechanism is needed for the certificate profile approval workflow to ensure implementation is blocked until all required validations are successfully completed.

What went well

The discrepancy was identified before any end-entity certificates were issued.

What didn't go well

Failure to enforce the certificate profile approval workflow allowed the error to go undetected.

Where we got lucky

No end-entity certificates were issued from the affected Subordinate CA.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Correct the Subordinate CA certificate profile Corrective Root Cause Issue resolved 2025-09-17 Completed
Add systematic enforcement to approval workflow Preventive Root Cause Prevent recurrence 2025-10-31 Pending

Appendix

Details of affected certificates

ID=21201367454

Hi, the Full Incident Report deviates significantly from the template required by CCADB. You can find the templates at https://www.ccadb.org/cas/incident-report#incident-report-field-definitions-and-expectations

Specifically,
Timeline summary: - this should be a summary with the 3 listed sub sections, not the entire timeline. (Also putting all times are UTC and UTC on each line is redundant, but not against the template)
Timeline - this is where the timeline you posted actually goes.
Related incidents - there is at least 1 in the last 2 years that violated the same relevant policies.
Root Cause Analysis - this entire section is mal-formed, and missing information.
Lessons Learned - Additional section missing (even if marked N/A)

(In reply to Colin from comment #2)

Full Incident Report - Corrected

Summary

  • CA Owner CCADB unique ID: A000036
  • Incident description:
    On September 15, 2025, we issued a new enterprise subordinate CA certificate. However, the following day, we identified issues with the Certificate Revocation List (CRL). Upon investigation, we discovered that the CRL Distribution Point (CDP) specified in both the Authority Information Access (AIA) and CRL extensions was incorrect—the certificate pointed to itself as the issuer of the CRL.
  • Timeline summary:
    • Non-compliance start date: 2025-09-15
    • Non-compliance identified date: 2025-09-16
    • Non-compliance end date: 2025-09-17
  • Relevant policies:
    Baseline Requirements Section 4.9.1.2 - Circumstances for Revoking a CA Certificate: (4) The Issuing CA obtains evidence that the Certificate was misused and Section 7.1.2 (Certificate Content and Extensions) - The values in certificate fields are expected to be functional
  • Source of incident disclosure:
    CCADB

Impact

  • Total number of certificates: 1
  • Total number of "remaining valid" certificates: 0
  • Affected certificate types: TLS Subordinate CA
  • Incident heuristic: https://crt.sh/?id=21201367454
  • Was issuance stopped in response to this incident, and why or why not?: Yes

Timeline

All times are UTC.
2025-09-08: 22:23 – Issued pre-production Subordinate CA
2025-09-15: 19:42 – Issued production Subordinate CA
2025-09-16: 22:00 – Disclosed production Subordinate CA in CCADB
2025-09-16: 19:14 – Identified issue with production Subordinate CA and corrected the certificate profile
2025-09-17: 20:14 – Issued the new Subordinate CA in production
2025-09-18: 01:43 – Revoked the misissued Subordinate CA
2025-09-18: 23:30 – Updated CCADB to reflect revocation and disclosed the new production Subordinate CA
2025-09-24: 16:17 – Received notification from CCADB that the revoked Subordinate CA is not present in the root CRL
2025-09-26: 19:20 – Disclosed preliminary incident report

Related Incidents

Bug Date Description
[1969036] https://bugzilla.mozilla.org/show_bug.cgi?id=1969036 2025-05-28 Incorrect CDP.
[1793787] https://bugzilla.mozilla.org/show_bug.cgi?id=1793787 2022-10-05 Non-existent hostname in CDP and AIA URLs

Root Cause Analysis

Contributing Factor: Lack of enforcement to the certificate profile approval Workflow

  • Description:
    During the creation of the certificate profile for the new Subordinate CA, an incorrect value string was inadvertently copied and pasted into the AIA (Authority Information Access) extension and CRL DP extension.
    The certificate profile was reviewed by only one individual and did not undergo the established approval process which prevented correction to the correct location prior to issuance.

  • Timeline:
    2025-09-15: Issued production Subordinate CA with invalid CDP value
    2025-09-16: Identified and corrected the source of the issue
    2025-09-17: Issued new CA with correct CDP value

  • Detection: 2025-09-16: Identified and corrected the source of the issue

  • Interaction with other factors: The invalid string prevented the creation of the expected CDP

Lessons Learned

  • What went well:
    The discrepancy was identified before any end-entity certificates were issued.

  • What didn’t go well:
    Failure to enforce the certificate profile approval workflow allowed the error to go undetected.

  • Where we got lucky:
    No end-entity certificates were issued from the affected Subordinate CA.

  • Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Correct the Subordinate CA certificate profile Corrective Root Cause Issue resolved 2025-09-17 Complete
Correct the Subordinate CA certificate profile Preventive Root Cause Prevent recurrence 2025-10-31 Pending

Appendix

https://crt.sh/?id=21201367454

Shouldn't the action items address the root cause, as opposed to the symptoms? ie why the certificate approval workflow wasn't enforced?

The approval workflow was not enforced due to the absence of a defined enforcement mechanism. As a result, the implementation team mistakenly assumed that the provided certificate profile had already been reviewed by subject matter experts, when in fact it had not.

Updated action items:

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Correct the Subordinate CA certificate profile Corrective Root Cause Issue resolved 2025-09-17 Complete
Add systematic enforcement to approval workflow Preventive Root Cause Enforcement in place 2025-10-31 Pending

Thank you for this report. However, we’re seeing some concerning patterns with IdenTrust and we have some questions.

(Q1): In 1669594, IdenTrust committed to a "thorough review of subordinate CA certificate profiles prior to issuance, including quality review by our Risk Management Committee." In 1850807, IdenTrust committed to "strengthen our quarterly self-audit processes to ensure that any discrepancies between the CPS and the certificate profiles are corrected." Both of these controls could have contributed to preventing the failure in this incident from occuring. Why did these controls fail, and what makes the new action item to "add systematic enforcement" technically different and more likely to succeed than these past failed commitments?

(Q2): The root cause of this incident was identified by IdenTrust as a bypassed procedural control, which is functionally similar to the root cause of 1653680, where an expedited update process lacked SME review. What systemic changes has IdenTrust made since 2020 to inventory all compliance-critical workflows and ensure they are technically enforced, rather than applying point-fixes only to the process that failed in a specific incident?

(Q3): In the Timeline we see: “2025-09-24: 16:17 – Received notification from CCADB that the revoked Subordinate CA is not present in the root CRL” but no additional commentary. We’ve seen past IdenTrust incidents related to revocation services (e.g., 1709192, 1636544, etc.). What was the specific root cause for failing to publish this revocation, and how do your Action Items for this failure differ from previous commitments to improve the monitoring and reliability of your revocation infrastructure?

(Q4): Reviewing past IdenTrust incident reports, and somewhat reflective of the questions above, we’re seeing recurring, functionally similar failures in certificate profiling, procedural enforcement, and revocation systems. It seems, issues persist despite numerous incident reports and commitments to improve. Is IdenTrust considering or planning to conduct a holistic, top-down review of its compliance culture, quality assurance, and control environment to address these recurring systemic weaknesses, rather than continuing to address them on an incident-by-incident basis?

(In reply to chrome-root-program from comment #6)

Thank you for this report. However, we’re seeing some concerning patterns with IdenTrust and we have some questions.

(Q1): In 1669594, IdenTrust committed to a "thorough review of subordinate CA certificate profiles prior to issuance, including quality review by our Risk Management Committee." In 1850807, IdenTrust committed to "strengthen our quarterly self-audit processes to ensure that any discrepancies between the CPS and the certificate profiles are corrected." Both of these controls could have contributed to preventing the failure in this incident from occuring. Why did these controls fail, and what makes the new action item to "add systematic enforcement" technically different and more likely to succeed than these past failed commitments?
IdenTrust:
The control failure was attributed to a breakdown of the established certificate profile validation process. Specifically, the implementation team mistakenly assumed that the certificate profile had already been reviewed and approved by the designated subject matter experts (SMEs), as required by the established certificate profile approval process. This incorrect assumption led to a bypass of the intended quality control measures.
To address this gap and prevent recurrence, a new action item has been introduced that incorporates a systematic enforcement mechanism into the workflow. This enhancement includes templatizing all subordinate CA certificate profiles, gating function that technically restricts the progression of certificate profiles within version control system. This enhancement control will ensure that profiles cannot be delivered to the implementation team until all assigned SMEs have explicitly completed their review and provided formal approval. By embedding this control into the process, the risk of assumptions and other forms of error is significantly reduced, thereby strengthening the integrity of the approval workflow.

(Q2): The root cause of this incident was identified by IdenTrust as a bypassed procedural control, which is functionally similar to the root cause of 1653680, where an expedited update process lacked SME review. What systemic changes has IdenTrust made since 2020 to inventory all compliance-critical workflows and ensure they are technically enforced, rather than applying point-fixes only to the process that failed in a specific incident?
IdenTrust:
Since 2020, IdenTrust has undertaken a comprehensive initiative to strengthen compliance posture by implementing systematic, technically enforced safeguards across all CA/B forum requirements.
Comprehensive Mapping of CA/B Forum Baseline Requirements
IdenTrust has completed a full mapping of the CA/Browser Forum Baseline Requirements, Extended Validation Guidelines, and applicable RFCs to its internal policies, procedures, and technical controls. Each requirement is linked to a specific control check—whether procedural, automated, or both—and is reviewed periodically to ensure continued alignment.
Pre-Issuance Linting and Simulation
Before any ICA or end-entity certificate is issued, it undergoes pre-issuance linting using different linter tools aligned with CAB Forum and Mozilla Root Store requirements. Simulated issuance is performed in a staging environment to validate all extensions and distribution points.
Mandatory SME Review
All changes to certificate profiles, ICA configurations, and issuance parameters require documented SME review and dual approval from Compliance and Technical Operations. These gates are now enforced through change management tooling and cannot be bypassed./lf
Apart from these technical controls, IdenTrust has established an internal CA/Browser Forum Advisory Board composed of subject matter experts from Compliance, Engineering, PKI and Policy teams. This board monitors all CA/B Forum updates, collaborates with stakeholders, and ensures timely integration of new requirements into IdenTrust’s policies, procedures, and technical controls.

(Q3): In the Timeline we see: “2025-09-24: 16:17 – Received notification from CCADB that the revoked Subordinate CA is not present in the root CRL” but no additional commentary. We’ve seen past IdenTrust incidents related to revocation services (e.g., 1709192, 1636544, etc.). What was the specific root cause for failing to publish this revocation, and how do your Action Items for this failure differ from previous commitments to improve the monitoring and reliability of your revocation infrastructure?
IdenTrust:
We revoked the certificate and added it to the issuing root CA's Certificate Revocation List (CRL), which was appropriately published on 2028-09-18. The corresponding CCADB record was updated to reflect the revocation on the same date. Initially, the certificate did not appear as revoked in CCADB due to the incorrect CDP value embedded in the certificate—this misconfiguration was the reason for the revocation. The monitoring and revocation infrastructure referenced in the referenced incidents is not pertinent to the root cause of this particular issue.

(Q4): Reviewing past IdenTrust incident reports, and somewhat reflective of the questions above, we’re seeing recurring, functionally similar failures in certificate profiling, procedural enforcement, and revocation systems. It seems, issues persist despite numerous incident reports and commitments to improve. Is IdenTrust considering or planning to conduct a holistic, top-down review of its compliance culture, quality assurance, and control environment to address these recurring systemic weaknesses, rather than continuing to address them on an incident-by-incident basis?
IdenTrust:
Our systems, practices, and controls are subject to ongoing review and evaluation against multiple highly regulated standards.
At a minimum of once per year, an independent auditor assesses our systems, practices, and controls for compliance with both AICPA WebTrust and CA/Browser Forum requirements.
In addition to these formal audits, we undergo separate annual assessments conducted by multiple customers. These audits focus on evaluating our quality assurance processes and identifying any recurring systemic weaknesses in our control environment.
Following each audit, IdenTrust conducts a thorough analysis of any identified issues or recommendations. We promptly implement corrective actions and establish preventive measures to mitigate future risks.
We actively review and enhance our compliance framework to address common root causes of recurring issues by introducing technical automated controls as feasible. This includes managing certificate profiles, linting, issuance, revocation systems, and internal controls are integrated and monitored across teams.
These measures reflect IdenTrust’s commitment to continuous improvement and proactive risk mitigation.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Correct the Subordinate CA certificate profile Corrective Root Cause Issue resolved 2025-09-17 Completed
Add systematic enforcement to approval workflow Preventive Root Cause Prevent recurrence 2025-10-31 Completed

Report Closure Summary

  • Incident description:
    Issuance of an Issuing subordinate CA (ICA) with invalid CRL Distribution Point (CDP)
  • Incident Root Cause(s):
    An incorrect value string was inadvertently copied into the AIA (Authority Information Access) and CRL Distribution Point extensions for a new ICA. The discrepancy went undetected because review and approval within the established workflow were not enforced.
  • Remediation description:
    The profile was promptly corrected, and a systemic enforcement mechanism was added to the existing approval workflow. This measure ensures that the implementation team cannot access the profile until all predefined reviews and approvals are completed
  • Commitment summary:
    IdenTrust will ensure that any new certificate profile is not implemented until all required subject matter experts have reviewed and approved it before delivery to the implementation team.

All Action Items disclosed in this report have been completed as described, and we request its closure.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-11-21.

Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [ca-misissuance] → [close on 2025-11-21] [ca-compliance] [ca-misissuance]
Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Flags: needinfo?(incident-reporting)
Resolution: --- → FIXED
Whiteboard: [close on 2025-11-21] [ca-compliance] [ca-misissuance] → [ca-compliance] [ca-misissuance]
You need to log in before you can comment on or make changes to this bug.