Open Bug 2005399 Opened 2 months ago Updated 6 days ago

CFCA: DV OCA caIssuers Returns PEM Encoded Certificate (RFC 5280 Section 4.2.2.1 Violation)

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: songxinlei, Assigned: songxinlei)

Details

(Whiteboard: [ca-compliance] [policy-failure])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

Steps to reproduce:

Preliminary Incident Report

Incident Title: CFCA: DV OCA caIssuers Returns PEM Encoded Certificate (RFC 5280 Section 4.2.2.1 Violation)

Summary

  • Incident description: The caIssuers HTTP link in the CFCA DV OCA certificate (https://crt.sh/?id=7833196483) was returning a PEM encoded certificate instead of DER or BER encoded format as required by RFC 5280. The affected URI was http://gtc.cfca.com.cn/evroot/evroot.cer. This issue was identified through a community member's report after they reviewed a related Bugzilla report (https://bugzilla.mozilla.org/show_bug.cgi?id=2004492). CFCA has completed remediation by updating the caIssuers link to return DER encoded certificates and conducted a comprehensive investigation across the entire root certificate hierarchy to ensure compliance.
  • Relevant policies: RFC 5280 Section 4.2.2.1 - Authority Information Access (AIA) extension. The policy specifies that when certificate information is available via HTTP or FTP, the accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in RFC 2585 or a collection of certificates in a BER or DER encoded format. PEM encoding does not comply with this requirement.
  • Source of incident disclosure: Third Party Reported

Impact

This issue does not affect the CA server certificate itself, and is limited to the file referenced in the id-ad-caIssuers access method of the CFCA DV OCA certificate (https://crt.sh/?id=7833196483). The affected URI was http://gtc.cfca.com.cn/evroot/evroot.cer. This corrected file has been updated and is now in the expected format (DER). All other production subordinate CA certificates have been reviewed, and their AIA id-ad-caIssuers values have been confirmed to be in DER format.

The root cause is under investigation, and a full incident report will be provided later.

Assignee: nobody → songxinlei
Type: defect → task
Whiteboard: [ca-compliance] [policy-failure]
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

We’re preparing the full report, will update within this week.

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000272
  • Incident description:

During review of a community member's report, it was identified that the file referenced in id-ad-caIssuers within the Authority Information Access (AIA) extension of the CFCA DV OCA certificate is provided in PEM format rather than the DER-encoded format expected by RFC 5280 and the CA/Browser Forum Server Certificate Baseline Requirements (BR).

The affected certificate is CFCA DV OCA (SHA-256: da738a474ee7473c9699ecba8eb5f483ada967988185a05975c4ba0c01b39559), and the caIssuers link (http://gtc.cfca.com.cn/evroot/evroot.cer) was returning a PEM encoded certificate instead of DER or BER encoded format.

  • Timeline summary:
    • Non-compliance start date: 2022-10-17
    • Non-compliance identified date: 2025-12-06
    • Non-compliance end date: 2025-12-10
  • Relevant policies: RFC 5280 Section 4.2.2.1 - Authority Information Access (AIA) extension. The policy specifies that when certificate information is available via HTTP or FTP, the accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in RFC 2585 or a collection of certificates in a BER or DER encoded format.
  • Source of incident disclosure: Third Party Reported (Community member Dexter Dopping)

Impact

  • Total number of certificates: 1
  • Total number of "remaining valid" certificates: 1
  • Affected certificate types: TLS Subordinate CA (CFCA DV OCA)
  • Incident heuristic: https://crt.sh/?id=7833196483
  • Was issuance stopped in response to this incident, and why or why not?:

Issuance was not stopped as there was no certificate misissuance involved. The issue was limited to the format of the certificate file referenced in the AIA extension, not the certificate itself.

  • Analysis:

This issue does not affect the CA server certificate itself, and is limited to the file referenced in the id-ad-caIssuers access method. The affected URI (http://gtc.cfca.com.cn/evroot/evroot.cer) has been corrected and is now in the expected format (DER). All other production subordinate CA certificates have been reviewed, and their AIA id-ad-caIssuers values have been confirmed to be in DER format.

  • Additional considerations:

A comprehensive investigation was conducted across the entire root certificate hierarchy, and unified remediation measures have been implemented to ensure all certificates comply with RFC 5280 specifications.

Timeline

All times are in UTC+8 (China Standard Time)

  • 2025-12-06 09:26 - Received CPR from community member Dexter Dopping reporting that the caIssuers link in CFCA DV OCA certificate returns PEM encoded certificate
  • 2025-12-06 09:42 - Confirmed the problem and acknowledged the issue. Initiated investigation to determine scope across certificate hierarchy
  • 2025-12-06 to 2025-12-10 - Conducted comprehensive investigation across entire root certificate hierarchy to identify all affected certificates
  • 2025-12-10 14:35 - Completed remediation by updating the caIssuers link to return DER encoded certificate. Implemented unified remediation measures across all affected certificates. Notified CPR sender of completion

Related Incidents

Bug Date Description
2004492 2025-12-06 IdenTrust: CA Certificate not published in DER Encoded Format - Similar issue where caIssuers link returned PEM instead of DER format
1637093 2020-05-11 AIA CA Issuer field pointing to PEM encoded cert
1637854 2020-05-13 AIA CA Issuer field pointing to PEM encoded cert
1884461 2024-03-08 CA Certificates not published in DER Encoded Format
1914466 2024-08-22 CA Certificates not published in DER Encoded Format
2004668 2025-12-08 Telekom Security: Root-CA certificates published in PEM encoded format
2004732 2025-12-23 Certigna: AIA CA issuer field pointing to PEM encoded cert

Root Cause Analysis

Contributing Factor #1: Insufficient awareness and implementation of RFC 5280 requirements

  • Description:

We acknowledge that we did not fully recognize or implement the specific requirement in RFC 5280 Section 4.2.2.1 that mandates DER or BER encoding for certificates accessible via HTTP/FTP in the AIA extension's id-ad-caIssuers field. While we were aware of the general requirement for DER encoding in certificate contexts, we failed to apply this requirement specifically to the certificate files referenced in the AIA extension. This oversight resulted in the PEM-encoded certificate being published at the caIssuers URI. Our process for issuing and maintaining Subordinate CAs did not include validation checks to ensure compliance with this specific RFC requirement for AIA extension resources. Additionally, we did not have an alerting tool in place to monitor the published id-ad-caIssuers file format; such a tool would have identified this discrepancy promptly.

  • Timeline:

This gap in understanding and implementation existed from the time the certificate was issued (2022-10-17) until the issue was identified through a third-party report on 2025-12-06.

  • Detection:

The issue was detected through a community member's report after they reviewed a related Bugzilla report (Bug 2004492) regarding a similar issue with another CA. This external review highlighted our oversight.

  • Interaction with other factors:

The lack of awareness of this specific requirement, combined with the absence of automated validation and monitoring during certificate issuance and maintenance processes, contributed to the issue remaining undetected for an extended period.

  • Root Cause Analysis methodology used:

5 Whys analysis and timeline reconstruction

Contributing Factor #2: Lack of automated validation and monitoring in certificate deployment process

  • Description:

The certificate management and deployment process lacks automated validation scripts to ensure that the id-ad-caIssuers file is generated and published in DER-encoded format. Specifically, we did not implement:

  1. Pre-deployment validation: Automated scripts to verify the encoding format of caIssuers files before they are published to production URIs
  2. Format conversion automation: Automated tools to convert certificate files to DER format during the deployment process, ensuring consistency
  3. Periodic monitoring: Scheduled checks (e.g., daily or weekly) that use tools like OpenSSL to fetch and validate the encoding format of all published caIssuers URIs, with alerting mechanisms for non-compliance

Manual processes are prone to human error, and the absence of these automated checks increases the likelihood of format inconsistencies going undetected. While we have processes for certificate issuance, we did not extend these controls to validate the format of referenced resources in the AIA extension.

  • Timeline:

This gap has existed throughout the certificate lifecycle management process, from initial deployment through ongoing maintenance.

  • Detection:

Identified during root cause analysis after the incident was reported. The absence of automated checks meant this issue could only be detected through manual review or external reporting.

  • Interaction with other factors:

The lack of automation compounds the monitoring gap identified in Root Cause #1. Without automated validation at deployment time and continuous monitoring post-deployment, format issues can persist undetected even when processes are followed correctly.

  • Root Cause Analysis methodology used:

Process analysis and gap identification

Lessons Learned

  • What went well:
  1. Rapid response to the third-party report - we acknowledged the issue within 16 minutes of receiving the CPR
  2. Comprehensive investigation - we conducted a full review of the entire root certificate hierarchy, not just the reported certificate
  3. Unified remediation - we implemented measures across all certificates to ensure compliance
  4. Transparent communication - we maintained open communication with the reporter throughout the remediation process
  • What didn't go well:
  1. The issue was not detected internally - it required a third-party report to identify
  2. No automated monitoring was in place to detect format compliance issues
  3. The problem may have existed for an extended period before discovery
  4. Manual processes lacked sufficient validation controls
  • Where we got lucky:
  1. The issue was limited to the format of a referenced file, not the certificate itself, so no certificate misissuance occurred
  2. The community member's proactive review and reporting helped identify the issue
  3. The remediation was straightforward and could be completed quickly
  • Additional:

This incident highlights the importance of:

  • Implementing automated monitoring and validation for all certificate-related resources
  • Regular compliance audits of certificate infrastructure
  • Learning from similar incidents reported by other CAs in the community
  • Proactive review of certificate infrastructure, not just reactive response to reports

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Corrected caIssuers file to be in DER format Correct Fixing the issue Successful validation and verification 2025-12-10 Completed
Comprehensive review of all root and intermediate certificates Correct Fixing the issue All certificates verified to be in DER format 2025-12-10 Completed
Develop pre-deployment validation scripts Prevent Root Cause #2 Scripts that verify DER encoding format before caIssuers files are published to production URIs 2025-12-25 Completed
Implement automated format conversion in deployment process Prevent Root Cause #2 Automated tools integrated into deployment workflow to ensure DER format conversion 2026-01-30 Pending
Establish regular compliance audit schedule Prevent Root Cause #1, #2 Quarterly audits scheduled and documented 2026-03-31 Pending

Appendix

Our deployment process follows a structured pipeline for each version or file change. By implementing validation checks at both the testing stage and the production deployment stage, we can ensure DER format compliance while aligning with our existing deployment workflow. This approach provides multiple checkpoints to catch format issues early, before they reach production, and is more practical given our current infrastructure and processes. So we're updating the Action Items as follows:

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Corrected caIssuers file to be in DER format Correct Fixing the issue Successful validation and verification 2025-12-10 Completed
Comprehensive review of all root and intermediate certificates Correct Fixing the issue All certificates verified to be in DER format 2025-12-10 Completed
Develop pre-deployment validation scripts Prevent Root Cause #2 Scripts that verify DER encoding format before caIssuers files are published to production URIs 2025-12-25 Completed
Add format validation checks in testing and production deployment processes Prevent Root Cause #2 Validation checks integrated into testing and production deployment workflows to verify DER encoding format of caIssuers files 2025-12-26 Completed
Establish regular compliance audit schedule Prevent Root Cause #1, #2 Quarterly audits scheduled and documented 2026-03-31 Pending

#1 In consideration of the fact that this violation has persisted since three years, I request that you provide the specific audit workpapers or statements of your auditor. It must be explained how this URI was verified during the audit cycles of 2023, 2024, and 2025. If no verification did happen, please explain why these audits shall be seen as valid representations of your compliance or why this auditor should be considered competent for such evaluation of public trust.

#2 Regarding your CCADB Compliance Self-Assessments of 2024 and 2025: Have you indicated therein that your AIA extensions conform to RFC 5280? If this is so, upon which internal evidence did you rely for making such claim? Furthermore, why did this evidence not succeed to reveal the PEM encoding?

#3 Following the failures identified in Bug 1886135 and Bug 1793059, CFCA has given a pledge to improve the "SSL Certificate Standard Tracking Specification" . I must ask: Why did this specification not include the encoding requirements for AIA URIs? These requirements are explicitly listed in the RFCs which you claim to be tracking.

#4 Which specific individual—identifiable by their professional role—is responsible for the final verification of changes to the infrastructure configuration? Does this person stand under the report of the Operations team or the Compliance team? This incident is suggestive of a breakdown in the Segregation of Duties.

#5 Before the format was "fixed" to DER, have you conducted a thorough analysis why it was originally PEM? Was this a manual configuration error, or did the software for certificate issuance output a PEM file per default?. If the latter is true, are there other outputs from this software which are also non-conform to the standards?

#6 This issue was found by the community member and not by your own monitoring. In the Bug 1778035 and the Bug 1886135, you were promising to improve the post-issuance linting and the verification. Why did these systems not find the PEM encoding error? Is your linting only for the end-entity certificates and not for the subordinate CA certificates?

#7 The timeline provided indicates an awareness of the problem on December 6, yet the bug was not filed until December 11. This is exceeding the 72-hour reporting requirement which is specified in the CCADB and Mozilla policies . I request an explanation of the internal breakdown that caused this delay. Please also give confirmation if an internal decision was made to wait for the completion of remediation before the public report was filed.

Flags: needinfo?(songxinlei)

(In reply to Dean F. Reed from comment #4)

#1 In consideration of the fact that this violation has persisted since three years, I request that you provide the specific audit workpapers or statements of your auditor. It must be explained how this URI was verified during the audit cycles of 2023, 2024, and 2025. If no verification did happen, please explain why these audits shall be seen as valid representations of your compliance or why this auditor should be considered competent for such evaluation of public trust.

#2 Regarding your CCADB Compliance Self-Assessments of 2024 and 2025: Have you indicated therein that your AIA extensions conform to RFC 5280? If this is so, upon which internal evidence did you rely for making such claim? Furthermore, why did this evidence not succeed to reveal the PEM encoding?

#3 Following the failures identified in Bug 1886135 and Bug 1793059, CFCA has given a pledge to improve the "SSL Certificate Standard Tracking Specification" . I must ask: Why did this specification not include the encoding requirements for AIA URIs? These requirements are explicitly listed in the RFCs which you claim to be tracking.

#4 Which specific individual—identifiable by their professional role—is responsible for the final verification of changes to the infrastructure configuration? Does this person stand under the report of the Operations team or the Compliance team? This incident is suggestive of a breakdown in the Segregation of Duties.

#5 Before the format was "fixed" to DER, have you conducted a thorough analysis why it was originally PEM? Was this a manual configuration error, or did the software for certificate issuance output a PEM file per default?. If the latter is true, are there other outputs from this software which are also non-conform to the standards?

#6 This issue was found by the community member and not by your own monitoring. In the Bug 1778035 and the Bug 1886135, you were promising to improve the post-issuance linting and the verification. Why did these systems not find the PEM encoding error? Is your linting only for the end-entity certificates and not for the subordinate CA certificates?

#7 The timeline provided indicates an awareness of the problem on December 6, yet the bug was not filed until December 11. This is exceeding the 72-hour reporting requirement which is specified in the CCADB and Mozilla policies . I request an explanation of the internal breakdown that caused this delay. Please also give confirmation if an internal decision was made to wait for the completion of remediation before the public report was filed.

Hi Dean,

Thank you for your detailed questions. We take these concerns seriously and will address each point comprehensively.

#1 Audit Workpapers and Verification During Audit Cycles

We don’t have the workpapers in hands and we believe it would be restricted for the auditor to disclose them under the contractual requirements.

#2 CCADB Self-Assessment Claims

Yes, we did claim that our AIA extensions conform to RFC 5280 in both the 2024 and 2025 self-assessments, but in Section 7.1.2.10.3, based on our understanding, there's no direct description for the encoding format of objects referenced by AIA URIs. Thus, we did not specifically validate the encoding format of objects referenced by AIA URIs.

#3 SSL Certificate Standard Tracking Specification

We acknowledge that the specification did not explicitly include encoding requirements for objects referenced by AIA URIs. For tracking audit requirements and BRs, we designed the specification primarily focused on certificate content compliance and did not explicitly cover compliance requirements for resources referenced by AIA extensions.

Remediation:

  • We are updating the specification to explicitly include RFC 5280 Section 4.2.2.1 requirements regarding AIA extensions and AIA URI encoding format verification requirements
  • Target completion: 2026-01-09

#4 Individual Responsible for Final Verification

Michael(songxinlei@gmail.com), of the Compliance team under the Product Department, is responsible for the final verification of changes to the CA system configurations and profiles now.

#5 Root Cause Analysis of PEM Format

The PEM formatted certificate was manually exported from CA system by operation team to give to the product team, and the product team applied to request the operation team to upload to the address referenced by the AIA URIs afterwards.

Product team used the PEM format because not aware of this specific file encoding requirement from RFC 5280 Section 4.2.2.1 and thus did not have a validation step to ensure the files are DER encoded.

#6 Why Linting Systems Did Not Find PEM Encoding Error

We acknowledge that our current post-issuance linting system has limitations. Our existing monitoring mechanisms primarily focus on certificate content and issuance processes, and the linting primarily targets end-entity certificates.

We currently use open-source linting tools, but these tools do not support verification of objects referenced by AIA extension URI encoding formats. The non-compliance occurred at the certificate publication level, and this aspect was not explicitly included in our original monitoring machenism.

Regarding our commitments in Bug 1778035 and Bug 1886135: We did commit to improving the linting system, but the improvements at that time focused primarily on certificate content itself and did not extend to verification of objects referenced by AIA extension URIs.

#7 72-Hour Reporting Requirement Delay

We acknowledge that we violated the 72-hour reporting requirement. The issue was identified on December 6, 2025 at 09:26, but the bug report was not filed until December 11, 2025. This exceeded the 72-hour requirement.

Due to an incorrect assumption that remediation should be completed before filing the report. We acknowledge this was an awareness and process deficiency and that we should not have waited for remediation completion before reporting.

We'll file another incident for this delay.

Flags: needinfo?(songxinlei)

(In reply to Dean F. Reed from comment #4)

#7 The timeline provided indicates an awareness of the problem on December 6, yet the bug was not filed until December 11. This is exceeding the 72-hour reporting requirement which is specified in the CCADB and Mozilla policies . I request an explanation of the internal breakdown that caused this delay. Please also give confirmation if an internal decision was made to wait for the completion of remediation before the public report was filed.

Hi Dean, we've filed a new bug for this here: Bug #2009134

Hello,

Your Full Incident Report includes the following:

“This incident highlights the importance of:
…
• Learning from similar incidents reported by other CAs in the community.”

It is good to see that you recognize the value of learning from incidents disclosed by other CAs. At the same time, it is not clear whether you are making any updates to your internal processes to ensure that you are consistently reviewing other incident reports. Please describe how you plan to incorporate these lessons proactively, rather than relying on third‑party reports.

Thank you.

(In reply to Dustin Hollenback (Apple) from comment #7)

Hello,

Your Full Incident Report includes the following:

“This incident highlights the importance of:
…
• Learning from similar incidents reported by other CAs in the community.”

It is good to see that you recognize the value of learning from incidents disclosed by other CAs. At the same time, it is not clear whether you are making any updates to your internal processes to ensure that you are consistently reviewing other incident reports. Please describe how you plan to incorporate these lessons proactively, rather than relying on third‑party reports.

Thank you.

Hi Dustin Hollenback (Apple),

Yes, learning from the community is of great help.
Our orignal process was by written email, which sent from a personel weekly specifically assigned for this job, but we found this may not be the best practice, compliance members may acknowledge the email but don't go through details. In order to improve this, we're establishing a process that will meetings/discussions will be held 2-4 times per month, to go through recent incidents for self inspection, attended by compliance team, product team and vetting team, or other related teams.

Updated Action Items:

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Corrected caIssuers file to be in DER format Correct Fixing the issue Successful validation and verification 2025-12-10 Completed
Comprehensive review of all root and intermediate certificates Correct Fixing the issue All certificates verified to be in DER format 2025-12-10 Completed
Develop pre-deployment validation scripts Prevent Root Cause #2 Scripts that verify DER encoding format before caIssuers files are published to production URIs 2025-12-25 Completed
Add format validation checks in testing and production deployment processes Prevent Root Cause #2 Validation checks integrated into testing and production deployment workflows to verify DER encoding format of caIssuers files 2025-12-26 Completed
Establish regular compliance audit schedule Prevent Root Cause #1, #2 Quarterly audits scheduled and documented 2026-03-31 Pending
Establish regular meetings to go through recent incidents Improve Root Cause #1, #2 Internal meeting logs 2026-02-14 Pending

Hi,
We're still monitoring this, no updates yet.

Hi,
We're still monitoring this, no updates yet.

Hi,
One Action Item Update: we've finalized the draft Incident Handling Standards for review.

You need to log in before you can comment on or make changes to this bug.