Closed Bug 2004492 Opened 9 months ago Closed 7 months ago

IdenTrust: CA Certificate not published in DER Encoded Format

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: roots, Assigned: roots)

Details

(Whiteboard: [ca-compliance] [policy-failure])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

Steps to reproduce:

Preliminary Incident Report

Summary

  • Incident Description
    During review of a cross-signed root CA in a staging environment, it was identified that the file referenced in id-ad-caIssuers within the Authority Information Access (AIA) extension of a production server subordinate CA certificate is provided in PEM format rather than the DER-encoded format expected by RFC 5280 and the CA/Browser Forum Server Certificate Baseline Requirements (BR)
  • Relevant policies:
    RFC-5280 Section 4.2.2.1 and BR Section 7.1.2.10.3
  • Source of incident disclosure:
    CA Owner self-disclosed

Impact

This issue does not affect the CA server certificate itself, and is limited to the file referenced, in the id-ad-caIssuers access method (https://crt.sh/?id=3234157095). This corrected file is now uploaded, and is in the expected format (DER). All other production subordinate CA certificates have been reviewed, and their AIA id-ad-caIssuers values have been confirmed to be in DER format.
The root cause is under investigation, and a full incident report will be provided by 2025-12-19.

Assignee: nobody → roots
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [policy-failure]

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000036

  • Incident description:

During review of a cross-signed root CA in a staging environment, it was identified that the file referenced in id-ad-caIssuers within the Authority Information Access (AIA) extension of a production server subordinate CA certificate is provided in PEM format rather than the DER-encoded format expected by RFC 5280 and the CA/Browser Forum Server Certificate Baseline Requirements (BR)

  • Timeline summary:

    • Non-compliance start date: 2020-08-12
    • Non-compliance identified date: 2025-12-03
    • Non-compliance end date: 2025-12-18
  • Relevant policies: RFC-5280 Section 4.2.2.1

  • Source of incident disclosure: CA Owner self-disclosed

Impact

  • Total number of certificates: 1

  • Total number of "remaining valid" certificates: 1

  • Affected certificate types: TLS Subordinate CA

  • Incident heuristic: https://crt.sh/?id=3234157095

  • Was issuance stopped in response to this incident, and why or why not?:
    Issuance was not stopped as there was no certificate misissuance involved.

  • Analysis: N/A

  • Additional considerations: N/A

Timeline

All times are in UTC
2020-08-12: 19:52 - Issued Subordinate CA with id-ad-caIssuers in PEM format
2025-12-03: 14:31 - Noticed the discrepancy
2025-12-04: 14:00 - Validated that all active subordinate CA's have the id-ad-caIssuers file in DER format
2025-12-06: 01:06 – Disclosed preliminary incident
2025-12-18: 11:30 – Corrected the id-ad-caIssuers file to be DER format

Related Incidents

Bug Date Description
https://bugzilla.mozilla.org/show_bug.cgi?id=1637093 2020-05-11 AIA CA Issuer field pointing to PEM encoded cert
https://bugzilla.mozilla.org/show_bug.cgi?id=1637854 2020-05-13 AIA CA Issuer field pointing to PEM encoded cert
https://bugzilla.mozilla.org/show_bug.cgi?id=1884461 2024-03-08 CA Certificates not published in DER Encoded Format
https://bugzilla.mozilla.org/show_bug.cgi?id=1914466 2024-08-22 CA Certificates not published in DER Encoded Format

Root Cause Analysis

Contributing Factor #1: Missing monitoring

  • Description:
    Our current process for issuing new Subordinate CAs includes a manual verification to ensure that the id-ad-caIssuers file referenced in the “CA Issuers” field of the AIA extension is in DER format.
    At the time of this incident (August 12, 2020), this control was not yet implemented, and the issue went unnoticed primarily due to the very low volume of certificates issued by the affected Subordinate CA—approximately one test certificate per month.
    Additionally, we did not have an alerting tool in place to monitor the published id-ad-caIssuers file; such a tool would have identified this discrepancy within hours.

Contributing Factor #2: Missing automation

  • Description:
    The key generation utility lacks an automated script to generate the id-ad-caIssuers file in DER-encoded format. Implementing such a tool would significantly reduce the likelihood of this issue recurring.

Lessons Learned

  • What went well:
    No customer were affected

  • What didn’t go well:
    Not detected until 5+ years later

  • Where we got lucky:
    Minimal certificate issuance

  • Additional:

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Corrected file to be in DER format Correct Fixing the issue Successful validation 2025-12-18 Completed
Add monitoring tool Detect Root Cause #1 Successful validation 2026-01-30 Pending
Incorporate automation Prevent Root Cause #2 Successful validation 2026-01-30 Pending

Appendix

https://crt.sh/?id=3234157095

We are track to deliver the promised items by 2026-01-30

This report has gone stale. As a reminder, CA Owners may request the “Next update” Whiteboard field be set by a Root Store Operator to align with a specific date related to an open Action Item.

Flags: needinfo?(roots)

Thank you for the reminder. As of 2026-01-05, we have implemented automation to address Root Cause #2. We remain on track to deploy the validation intended to address Root Cause #1 by 2026-01-30. We would like to request that the “Next update” Whiteboard field be set to align with this expected completion date.

Flags: needinfo?(roots)

Action Items - Updated

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Corrected file to be in DER format Correct Fixing the issue Successful validation 2025-12-18 Completed
Incorporate automation Prevent Root Cause #2 Successful validation 2026-01-30 Completed on 2026-01-05
Add validation Prevent Root Cause # 1 Successful validation 2025-12-30 Completed on 2026-01-13

Report Closure Summary

  • Incident description:
    During review of a cross-signed root CA in a staging environment, it was identified that the file referenced by the id-ad-caIssuers field within the Authority Information Access (AIA) extension of a production subordinate CA certificate was provided in PEM format instead of the DER-encoded format required by RFC 5280 and the CA/Browser Forum Baseline Requirements (BRs). This misconfiguration had remained undetected for over five years.

  • Incident Root Cause(s):
    Our current process for issuing new Subordinate CAs includes a manual check to verify that the id-ad-caIssuers file referenced in the “CA Issuers” field of the AIA extension is encoded in DER format.

At the time of this incident (August 12, 2020), this control had not yet been implemented, and the issue went undetected primarily because the affected Subordinate CA issued certificates at a very low frequency—approximately one test certificate per month.

Furthermore, we did not yet have an alerting mechanism to monitor the published id-ad-caIssuers file; such a tool would have detected this discrepancy within hours.

Remediation description:
We have since remediated the issue by correcting the file format and implementing two new controls:

  1. Automated validation within the key ceremony process to ensure the file is encoded in DER format.
  2. A post‑issuance verification to confirm that the published file remains in DER format.

Commitment summary:
All Action Items disclosed in this report have been completed as described, and we request its closure.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Flags: needinfo?(bwilson)

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-02-05.

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [policy-failure] → [Close on 2026-02-05] [ca-compliance] [policy-failure]
Status: ASSIGNED → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Whiteboard: [Close on 2026-02-05] [ca-compliance] [policy-failure] → [ca-compliance] [policy-failure]
You need to log in before you can comment on or make changes to this bug.