SHECA: CA Certificate not published in DER Encoded Format
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: wangjiatai, Assigned: wangjiatai)
Details
(Whiteboard: [ca-compliance] [policy-failure])
Preliminary Incident Report
Summary
- Incident Description
The file referenced in id-ad-caIssuers within the Authority Information Access (AIA) extension of some subscriber certificates issued by SHECA is provided in PEM format rather than the DER-encoded format expected by RFC 5280. - Relevant policies:
RFC-5280 Section 4.2.2.1
Where the information is available via HTTP or FTP, accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in [RFC2585] or a collection of certificates in a BER or DER encoded "certs-only" CMS message as specified in [RFC2797].
- Source of incident disclosure:
Third Party Reported.
A preliminary list of the intermediate CA certificates involved and their corresponding id-ad-caIssuers is shown in the table below. The hosted CA certificate files have been updated to the DER format, and a full investigation is in progress. SHECA will release a full incident report by December 17, 2025.
| CA certificates | id-ad-caIssuers |
|---|---|
| SanFront DV TLS RSA CA G1 | http://certs.global.sheca.com/sanfrontdvg1.cer |
| SanFront DV TLS RSA CA G2 | http://certs.global.sheca.com/sanfrontdvg2.cer |
| SanFront OV TLS RSA CA G1 | http://certs.global.sheca.com/sanfrontovg1.cer |
| SanFront OV TLS RSA CA G2 | http://certs.global.sheca.com/sanfrontovg2.cer |
|
||
Updated•9 months ago
|
Full Incident Report
Summary
-
CA Owner CCADB unique ID:
A000261 -
Incident description:
The file referenced in id-ad-caIssuers within the Authority Information Access (AIA) extension of some subscriber certificates issued by SHECA is provided in PEM format rather than the DER-encoded format expected by RFC 5280.
-
Timeline summary:
- Non-compliance start date:
2025-09-18 - Non-compliance identified date:
2025-12-09, 22:15 - Non-compliance end date:
2025-12-10, 00:27
- Non-compliance start date:
-
Relevant policies:
RFC 5280 Section 4.2.2.1 -
Source of incident disclosure:
Third Party Reported
Impact
- Total number of certificates:
N/A, as there were no certificates misissued - Total number of "remaining valid" certificates:
N/A, as there were no certificates misissued - Affected certificate types:
N/A, as there were no certificates misissued - Incident heuristic:
This will affect all subscriber certificates issued by the four Subordinate listed in the table below before 00:27 on December 10, 2025.Subordinate certificates id-ad-caIssuers SanFront DV TLS RSA CA G1 http://certs.global.sheca.com/sanfrontdvg1.cer SanFront DV TLS RSA CA G2 http://certs.global.sheca.com/sanfrontdvg2.cer SanFront OV TLS RSA CA G1 http://certs.global.sheca.com/sanfrontovg1.cer SanFront OV TLS RSA CA G2 http://certs.global.sheca.com/sanfrontovg2.cer - Was issuance stopped in response to this incident, and why or why not?:
As there were no certificates misissued, issuance was not stopped. - Analysis:
N/A - Additional considerations:
N/A
Timeline
All times are UTC+8
2025-09-18, 00:00 – Issued two Subordinate Certificates (SanFront DV TLS RSA CA G1, SanFront OV TLS RSA CA G1).
2025-09-18, 09:00 – Published the PEM-encoded certificates to the storage system and refreshed the CDN cache.
2025-11-12, 00:00 – Issued another two Subordinate Certificates (SanFront DV TLS RSA CA G2, SanFront OV TLS RSA CA G2).
2025-11-12, 10:00 – Published the PEM-encoded certificates to the storage system and refreshed the CDN cache.
2025-12-09, 22:15 – Received the issue identified and reported by a third party.
2025-12-09, 22:20 – Forwarded the issue report to the Root Certificate/Compliance Team.
2025-12-09, 22:30 – Conducted a preliminary check on the certificate encoding formats of all published Subordinate Certificates and Root Certificates.
2025-12-10, 00:27 – Initially confirmed that four Subordinate Certificates were affected; immediately republished the DER-encoded Subordinate Certificates.
2025-12-10, 16:19 – Released the preliminary incident report.
2025-12-11, 10:00 – Conducted a thorough review of the encoding formats of all published Subordinate Certificates, Root Certificates and CRLs to ensure no omissions.
2025-12-15, 09:00 – Confirmed that only the above four Subordinate Certificates had format issues, which have now been resolved.
Related Incidents
Our research indicates that the following recent bugs in Bugzilla are affected by the same issue:
| Bug | Date | Description |
|---|---|---|
| 2005567 | 2025-12-11 | CA Certificates Published in PEM format |
| 2005399 | 2025-12-10 | caIssuers Returns PEM Encoded Certificate |
| 2004845 | 2025-12-08 | CA Certificates Published in PEM format |
| 2004733 | 2025-12-08 | CA Certificate not published in DER Encoded Format |
| 2004732 | 2025-12-08 | AIA CA issuer field pointing to PEM encoded cert |
| 2004699 | 2025-12-08 | CA in AIA in PEM format |
| 2004521 | 2025-12-05 | CA Certificate not published in DER Encoded Format |
| 2004492 | 2025-12-05 | CA Certificate not published in DER Encoded Format |
Root Cause Analysis
The standard CA issuance and Subscriber certificate profile scripts do not include a check for certificate encoding, and the certificate upload program likewise lacks automated validation of the certificate encoding.
Contributing Factor #1: The script used for issuing CA certificates lacks a check item for verifying the certificate encoding format.
-
Description:
SHECA provides execution scripts for standard CA certificate issuance and Subscriber certificate profile settings, which include a chec items for each step, but it lacks a check item for the encoding of the certificate download file.
-
Timeline:
From 2016 to the present (SHECA officially launched its SSL certificate business in 2016)
-
Detection:
After the incident, we found that the CA certificate issuance script was missing a certificate encoding format check item during our inspection.
-
Interaction with other factors:
This factor creates a fundamental gap, which directly enables Contributing Factor #2 to trigger the incident, as the subsequent certificate publication process also lacks encoding verification function, leading to the release of incorrectly formatted certificates.
-
Root Cause Analysis methodology used:
5-Whys
Contributing Factor #2: The certificate upload process in the CA system lacks a function to check the certificate encoding format.
-
Description:
After SHECA issues Root Certificates or Subordinate Certificates, the CA system administrator uploads the certificates to the storage system via the CA system and automatically refreshes the CDN cache. However, the relevant program does not check the encoding format of the certificates.
-
Timeline:
From 2016 to the present (SHECA officially launched its SSL certificate business in 2016)
-
Detection:
Following the incident, we examined the certificate issuance program and found that it lacked automated controls related to certificate encoding format verification.
-
Interaction with other factors:
This factor acts as a critical secondary barrier failure after the initial gap from Contributing Factor #1. Even if the issuance script omits encoding format checks, a functional verification mechanism in the publication process could have intercepted the incorrectly encoded certificates before they were released. However, the absence of this verification means the defect from Factor #1 is not mitigated, resulting in the publication of non-compliant certificates and the subsequent incident.
-
Root Cause Analysis methodology used:
5-Whys
Lessons Learned
-
What went well:
After receiving the third-party email, SHECA responded quickly, completed the initial verification within one hour, and fixed the issue promptly.
-
What didn’t go well:
N/A -
Where we got lucky:
N/A -
Additional:
N/A
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Replace the four problematic subordinate certificates. | Prevent | #1 | Download the certificate and confirm that its encoding format is DER format. | 2025-12-11 | Complete |
| Write a script to check the encoding format of all CA certificates and CRLs. | Prevent | #1 | Check passed. | 2025-12-15 | Complete |
| Add a step for verifying the certificate encoding format in the CA certificate issuance scrip | Prevent | #1 | The script has been updated | 2025-12-17 | Complete |
| Add a certificate encoding format verification function to the CA certificate upload program of the CA system. | Prevent | #2 | The function has been luanched | 2025-12-26 | Ongoing |
| Add monitoring for the encoding formats of CA certificates and CRLs | Prevent | #2 | The function has been luanched | 2025-12-31 | Ongoing |
Appendix
N/A
|
||
Comment 2•8 months ago
|
||
Hello. I noticed that this bug is Third‑Party Reported and the Related Incidents section was missing several similar incidents from last year. Since similar issues were disclosed by several other CAs last year, it’s unclear whether you are monitoring incident reports from the broader community. Public disclosures are intended to help all CA operators learn from each other’s findings.
Could you confirm whether you review other CAs’ incident reports as part of your internal processes? If so, a brief overview of how you track these reports and incorporate relevant lessons would be helpful. It would also be useful to understand how this issue was ultimately identified through a third‑party report rather than your internal monitoring.
(In reply to Dustin Hollenback (Apple) from comment #2)
Hello. I noticed that this bug is Third‑Party Reported and the Related Incidents section was missing several similar incidents from last year. Since similar issues were disclosed by several other CAs last year, it’s unclear whether you are monitoring incident reports from the broader community. Public disclosures are intended to help all CA operators learn from each other’s findings.
Could you confirm whether you review other CAs’ incident reports as part of your internal processes? If so, a brief overview of how you track these reports and incorporate relevant lessons would be helpful. It would also be useful to understand how this issue was ultimately identified through a third‑party report rather than your internal monitoring.
Dear Dustin Hollenback (Apple),
Thank you for your attention and questions.
SHECA has been consistently tracking updates on the Bugzilla Incident Dashboard and ballots on the CA/B Forum, and actively participating in working group discussions as well as providing feedback on browser root inclusion policy revisions prior to their release. We are fully aware that root certificates must be uploaded in DER format, and all uploads prior to September 2025 complied with this requirement. Although human error played a role in recent certificate upload operations, the root cause lies in gaps in our exsiting processes and systems. Historically, our risk mitigation relied mainly on manual processes—with the compliance team issuing reminders and communicating requirements to relevant personnel—without implementing institutional or systematic enhancements.
we have recognized the importance of automated controls and are optimizing our CA system to implement automated processes, minimizing errors caused by human intervention. We have now explicitly incorporated format requirements into the root certificate signing script, and system verification functions will be added subsequently to ensure full compliance with RFC 5280 standards.
Best Regard!
|
|
||
Comment 4•8 months ago
|
||
Thank you for the update. Your response did not fully address the questions raised. Could you please clarify:
- Whether you formally review other CAs’ incident reports as part of your internal processes. e.g “Tracking updates on the Bugzilla Incident Dashboard” is unfortunately too vague for the questions I had asked.
- How lessons from those reports are incorporated into your controls.
- Why this issue was detected by a third party rather than your internal monitoring.
(In reply to Dustin Hollenback (Apple) from comment #4)
Dear Dustin Hollenback (Apple),
Thank you for your follow-up questions. We clarify the points as follows:
- Whether you formally review other CAs’ incident reports as part of your internal processes. e.g “Tracking updates on the Bugzilla Incident Dashboard” is unfortunately too vague for the questions I had asked.
- How lessons from those reports are incorporated into your controls.
We want to start by confirming that we have integrated reviews of other CAs' incident reports into our internal compliance management processes: Each week, our compliance and business teams work together to compile public incident reports from all CAs listed on the Bugzilla Incident Dashboard, and we conduct self-reviews and gap analyses for every type of issue during our biweekly compliance meetings.
For instance, when we noted other CAs' certificate upload issues back in 2024, we reviewed our own certificates and did not identify similar risks at the time. As a result, we chose to verbally remind our operators to reinforce manual checks. Thanks to our ongoing attention to these requirements, we did not experience similar errors prior to September 2025.
However, this process had a flaw: We failed to fully adopt our peers' corrective actions (in line with industry best practices) to strengthen system-level automated controls, which meant we could not mitigate similar risks at their root. This is the key gap this incident has brought to light, and we take full responsibility for this oversight.
To remedy this, SHECA has revised its approach: Moving forward, we will thoroughly summarize the root causes and action items for each incident type, translating identified risk points into actionable, peer-informed best practices. We will assess whether our existing systems can prevent similar issues through automation; if feasible, we will draft detailed requirement documents, and our compliance team will submit formal improvement requests to the product R&D department. We will also track the entire optimization process closely to ensure we resolve issues at their root—and reduce our reliance on manual controls.
- Why this issue was detected by a third party rather than your internal monitoring.
As a direct result of the process flaw outlined above, we previously lacked effective internal controls to proactively identify format upload issues (most notably, we had no system-level automatic checks in place). Instead, we relied solely on our operators' voluntary compliance awareness—a manual control method we now recognize is unreliable. We deeply regret that this gap meant we were unable to catch the issue internally.
To address this immediately, as the action items, we have now embedded format verification logic into the root CA signing script and initiated the development of system-level automatic verification and monitoring functions, which will be launched in the near future.
Add an action item #6 with a deadline of 2026-01-15.
Action Item #5 has not yet been released due to the holidays and has been delayed until 2026-01-07.
Action Items update
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Replace the four problematic subordinate certificates. | Prevent | #1 | Download the certificate and confirm that its encoding format is DER format. | 2025-12-11 | Completed |
| Write a script to check the encoding format of all CA certificates and CRLs. | Prevent | #1 | Check passed. | 2025-12-15 | Complete |
| Add a step for verifying the certificate encoding format in the CA certificate issuance scrip | Prevent | #1 | The script has been updated | 2025-12-17 | Complete |
| Add a certificate encoding format verification function to the CA certificate upload program of the CA system. | Prevent | #2 | The function has been luanched | 2025-12-26 | Complete |
| Add monitoring for the encoding formats of CA certificates and CRLs | Prevent | #2 | The function has been luanched | 2026-01-07 | Delayed |
| Add a lint to verify that the certificate encoding format returned by id-ad-caIssuers in the certificate AIA extension and the Content-Type of its response comply with the specifications. | Prevent | #2 | The function has been luanched | 2026-01-15 | Ongoing |
Action Items update
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Replace the four problematic subordinate certificates. | Prevent | #1 | Download the certificate and confirm that its encoding format is DER format. | 2025-12-11 | Completed |
| Write a script to check the encoding format of all CA certificates and CRLs. | Prevent | #1 | Check passed. | 2025-12-15 | Complete |
| Add a step for verifying the certificate encoding format in the CA certificate issuance scrip | Prevent | #1 | The script has been updated | 2025-12-17 | Complete |
| Add a certificate encoding format verification function to the CA certificate upload program of the CA system. | Prevent | #2 | The function has been luanched | 2025-12-26 | Complete |
| Add monitoring for the encoding formats of CA certificates and CRLs | Prevent | #2 | The function has been luanched | 2026-01-07 | Complete |
| Add a lint to verify that the certificate encoding format returned by id-ad-caIssuers in the certificate AIA extension and the Content-Type of its response comply with the specifications. | Prevent | #2 | The function has been luanched | 2026-01-15 | Ongoing |
Action Items update
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Replace the four problematic subordinate certificates. | Prevent | #1 | Download the certificate and confirm that its encoding format is DER format. | 2025-12-11 | Completed |
| Write a script to check the encoding format of all CA certificates and CRLs. | Prevent | #1 | Check passed. | 2025-12-15 | Complete |
| Add a step for verifying the certificate encoding format in the CA certificate issuance scrip | Prevent | #1 | The script has been updated | 2025-12-17 | Complete |
| Add a certificate encoding format verification function to the CA certificate upload program of the CA system. | Prevent | #2 | The function has been luanched | 2025-12-26 | Complete |
| Add monitoring for the encoding formats of CA certificates and CRLs | Prevent | #2 | The function has been luanched | 2026-01-07 | Complete |
| Add a lint to verify that the certificate encoding format returned by id-ad-caIssuers in the certificate AIA extension and the Content-Type of its response comply with the specifications. | Prevent | #2 | The function has been luanched | 2026-01-15 | Complete |
Report Closure Summary
- Incident description: The file referenced in id-ad-caIssuers within the Authority Information Access (AIA) extension of some subscriber certificates issued by SHECA is provided in PEM format rather than the DER-encoded format expected by RFC 5280.
- Incident Root Cause(s): The script used for issuing CA certificates lacks a check item for verifying the certificate encoding format. The certificate upload process in the CA system lacks a function to check the certificate encoding format.
- Remediation description: SHECA replaced non-compliant PEM files with DER-encoded versions per RFC 5280, developed encoding check scripts for CA certificates/CRLs, added verification steps to issuance scripts and upload programs, implemented real-time monitoring, and added a lint tool for AIA extension compliance.
- Commitment summary: SHECA commits to continuously maintaining CA certificate issuance scripts, persisting in promoting the automation of CA systems, and enhancing real-time monitoring of industry-related issues as well as self-inspections.
All Action Items disclosed in this report have been completed as described, and we request its closure.
|
|
||
Comment 10•7 months ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, it will be closed on approximately 2026-01-27.
|
|
||
Updated•7 months ago
|
Description
•