1970728 - eMudhra: Invalid CRL signatures

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Rob Stradling]

https://crt.sh/mozilla-disclosures reports that eMudhra is currently serving 6 CRLs with invalid ECDSA signatures:

https://crl.emsign.com/?EMDVTLSCAG3B.crl
https://crl.emsign.com/?EMEVTLSCAG3B.crl
https://crl.emsign.com/?EMOVTLSCAG3B.crl
https://crl.emsign.com/?EMDVTLSCAG3C.crl
https://crl.emsign.com/?EMEVTLSCAG3C.crl
https://crl.emsign.com/?EMOVTLSCAG3C.crl

Comment 1

[Naveen Kumar ML]

Preliminary Incident Report

Summary

• Incident description:
  On June 5, 2025, Mozilla’s monitoring service reported that six Certificate Revocation Lists (CRLs) published by eMudhra were signed with an incorrect private key, resulting in invalid ECDSA signatures. As a consequence, clients attempting to verify these CRLs would fail verification.
  No end entity certificates were issued under this issuing CA.
  A full incident report will be submitted no later than June 9, 2025.

• Relevant policies:
  Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates, Section 4.9 (Certificate Revocation Requirements)

• Source of incident disclosure:
  Third party report via Mozilla Bugzilla Bug 1970728 opened by Sectigo team and ticket raised as a result of email received on June 5, 2025 from Sectigo.

Comment 2

[Naveen Kumar ML]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A005678
• Incident description: On June 6, 2025, it was identified that CRLs published on May 29, 2025, for six of eMudhra’s Intermediate CAs were signed using incorrect CA key configuration, resulting in invalid CRL signatures. The issue was initially reported via email by Sectigo on June 3, 2025, but the message was quarantined by internal filters and not seen. A follow-up message was received on June 6, prompting immediate investigation and remediation.
• Timeline summary:
  • Non-compliance start date: May 29, 2025 09:46 UTC (CRLs signed with incorrect CA key configuration)
  • Non-compliance identified date: June 5, 2025, 20:12:16 UTC (external alert received in inbox)
  • Non-compliance end date: June 6, 2025 07:30 UTC (correct CRLs published)
• Relevant policies: CA/Browser Forum Baseline Requirements, Section 4.9
• Source of incident disclosure: External report by Sectigo via email and Bugzilla; confirmation via crt.sh dashboard (Mozilla: CA Certificate Disclosures in CCADB)

Impact

• Total number of certificates: None affected as no end entity certificates issued; CRLs were malformed due to incorrect signing.
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: CRLs for six Intermediate CAs
• Incident heuristic: Operational / Procedural error related. Since no end-entity certificates were issued and the issue was limited to CRLs generated using incorrect CA key configuration for six new Intermediate CAs, it is not possible for third parties to identify affected CRLs using a public heuristic.
• Was issuance stopped in response to this incident, and why or why not?: No, issuance of certificates was not impacted. The issue was limited to CRL generation and no End entity certificates were issued under these six new Intermediate CAs. Prompt corrective action ensured that updated, correctly signed CRLs were published to restore revocation integrity.
• Analysis: The affected CRLs were published on May 29, 2025, through a manual process that incorrectly referenced a non-corresponding CA key configuration. This process lacked strict validation mechanisms to confirm key correctness prior to signing. The discrepancy was not caught during the publication workflow, primarily due to failure of maker checker post-signature verification process for this new set of Intermediate CA certificates CRL generation path. Consequently, the malformed CRLs were publicly available until the issue was externally reported. While the underlying keys were from the issuing environment, they did not match the exact keys expected for those Intermediate CA CRLs, thereby resulting in signature validation failure. The problem was exacerbated by lack of redundancy in alerting from external email quarantine, delaying internal awareness. Remediation involved verifying the correct CA key configuration and regenerating the CRLs using the appropriate private key associated with each Intermediate CA.
• Additional considerations: N/A

Timeline

Timeline (All times in UTC)
2025-05-29 09:46:00 : CRLs generated and published with incorrect key configuration
2025-06-03 10:05:00 : Sectigo sends initial email alert to problem reporting (quarantined and not delivered to inbox)
2025-06-05 20:12:16 : Sectigo sends follow-up alert; issue acknowledged
2025-06-06 04:05:00 : Ticket was alerted to critical and was passed to internal PKI team and compliance team
2025-06-06 04:30:00 : Internal incident was created and PKI team started to analyse the CRLs and root cause for the signature invalid
2025-06-06 06:00:00 : PKI team re-traced the complete log for the occurrence and how the manual process caused the generation of CRLs using incorrect CA key.
2025-06-06 06:20:00 : Compliance team reviewed the root cause and sent for the internal clearance for generating the correct CRLs and publish.
2025-06-06 06:40:00 : Internal management confirmed for the CRL generation and publication
2025-06-06 07:10:00 : CRLs generation process completed with 6 new CRLs with right signature.
2025-06-06 07:30:00 : Correct CRLs published after validation
2025-06-06 09:04:00 : Preliminary report was submitted.
2025-06-06 15:00:00 : Reviewed the compliance of the CRL generation process and the root cause of incident.

Related Incidents

Bug	Date	Description
1793441	2023-04-19	Bug 1793441 similarly involved a CRL that appeared valid but failed verification not due to a wrong key, but because the signature algorithm OID was incorrect. As confirmed by GlobalSign, the CRL was produced by an ECC Issuing CA signed under an RSA root, but it mistakenly retained the root’s SHA256withRSA OID instead of the intended ECC algorithm.

In our case at eMudhra, the issue stemmed from using the wrong private key, yet both incidents resulted in publicly published CRLs that could not be validated by relying parties.

Though the root causes differ misconfigured algorithm OID versus outright key mismatch the operational impact was the same: invalid CRLs and broken revocation chains across both cases.
|

Root Cause Analysis

Contributing Factor 1: Incorrect Key Configuration Used During Manual CRL Generation

• Description: A manual CRL generation process used incorrect CA key configuration due to insufficient verification steps.

• Timeline:
  2025-05-29 09:46:00 : CRLs for six Intermediate CAs were generated manually using the wrong CA private key. No validation failure was flagged during the manual signing workflow.
  2025-06-03 10:05:00 : Initial CPR alert email from Sectigo sent to the problem-reporting address but was quarantined and not acted upon.
  2025-06-06 04:30:00 : Upon receipt of a follow-up alert, internal teams began investigation and traced the issue to incorrect key usage during the manual CRL generation
  2025-06-06 07:30:00 : Correct CRLs signed with the appropriate keys were published.

• Detection: Issue was detected on June 6, 2025, via external alert by Sectigo

• Interaction with other factors: Late detection of CRL signature failure as a result of late detection of inbound emails sent that went into quarantine.

• Root Cause Analysis methodology used: 5-Whys

Lessons Learned

• What went well:
  o Upon identifying the issue on June 6, corrective actions were initiated promptly.
  o The CRLs were regenerated and published within the same day using the correct CA key configuration.

• What didn’t go well:
  o First time CRL generation follows a manual process reliant on configurations and CA key mapping set in the system to drive the CRL generation.
  o This manual CRL generation path relies on maker checker controls to verify the accuracy of the CRL’s generated and was not augmented with automated checks to enforce or verify signature validation. This allowed CRLs to be published with invalid signatures.
  o The initial email alert from Sectigo was quarantined and not noticed until a follow-up.

• Where we got lucky:
  o These Intermediate CAs were recently created and had not yet issued any end-entity certificates, minimizing risk exposure.
  o No downstream relying parties reported failures due to the invalid CRL signature. External monitoring and proactive disclosure by Sectigo ensured the issue was brought to attention.

• Additional:
  o The affected Intermediate CAs were newly created and no end-entity certificates had been issued under them, which limited the operational impact.
  o This incident underscores the importance of replacing manual processes (even though infrequent) with automation as part of various stages of certificate lifecycle management, including CRL generation.
  o Strengthening email monitoring for exceptions such as quarantine, spam filters on a more frequent basis are necessary to ensure that external incident notifications are not missed given need to respond quickly and effectively.
  o The incident has triggered a broader review of manual checkpoints and how to automate and augment verification of critical steps in Issuing CA creation such as CRL generation.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Replace manual CRL signing process with an automated workflow to eliminate reliance on human configuration of key mappings	Prevent	Root Cause # 1	All CRLs generated via automated, validated flow	2025-06-20	In Progress
Introduce automated post-signing signature validation checks in CRL publishing steps, particularly during onboarding of new Issuing CAs	Prevent	Root Cause # 1	Every CRL is cryptographically validated immediately after signing and before publication	2025-06-20	In Progress

Appendix

Attached 6 CRLs
Issuer details of 6 CRLs:

1. EM DV TLS CA - G3B
2. EM DV TLS CA - G3C
3. EM EV TLS CA - G3B
4. EM EV TLS CA - G3C
5. EM OV TLS CA - G3B
6. EM OV TLS CA - G3C

Comment 3

[Naveen Kumar ML]

Attachment: EM EV TLS CA - G3B.crl

Comment 4

[Naveen Kumar ML]

Attachment: EM DV TLS CA - G3C.crl

Comment 5

[Naveen Kumar ML]

Attachment: EM EV TLS CA - G3B.crl

Comment 6

[Naveen Kumar ML]

Attachment: EM EV TLS CA - G3C.crl

Comment 7

[Naveen Kumar ML]

Attachment: EM OV TLS CA - G3B.crl

Comment 8

[Naveen Kumar ML]

Attachment: EM OV TLS CA - G3C.crl

Comment 9

[Naveen Kumar ML]

Following action items are anticipated to be completed on 2025-06-20:

1. Replace manual CRL signing process with an automated workflow to eliminate reliance on human configuration of key mappings
2. Introduce automated post-signing signature validation checks in CRL publishing steps, particularly during onboarding of new Issuing CAs

Comment 10

[Naveen Kumar ML]

All action items have been completed.

Action Item 1: The manual CRL signing process has been replaced with an automated workflow, eliminating reliance on manual key mapping configuration.
Action Item 2: Automated post-signing signature validation checks have been implemented in the CRL publishing workflow, especially during the onboarding of new Issuing CAs.

Comment 11

[Naveen Kumar ML]

No further action required at this time.

Comment 12

[Naveen Kumar ML]

No further action required at this time.

Comment 13

[incident-reporting]

Please provide a Closure Report if you feel this is ready for closure.

Comment 14

[Naveen Kumar ML]

Report Closure Summary

• Incident description:
  On May 29, 2025, CRLs were generated and published for six newly created Intermediate CAs through a manual process. This process incorrectly referenced a non-corresponding CA key configuration, resulting in CRLs that failed signature validation. The issue was initially reported by Sectigo via email on June 3, 2025, but the message was inadvertently quarantined by internal filters and not seen by the compliance team. A follow-up message was received on June 5, 2025, which triggered investigation. The issue was confirmed, and updated CRLs were generated and published using the correct CA key configuration on June 6, 2025.
• Incident Root Cause(s):
  The CRL generation workflow for the six Intermediate CAs was conducted manually, and due to insufficient validation controls, an incorrect CA key configuration was used. The signing operation succeeded without triggering a validation error, resulting in malformed CRLs. The post-signing verification process for this specific onboarding was not fully enforced, allowing the issue to go undetected until reported externally. The delay in internal identification was further impacted by the quarantining of the original alert email.
• Remediation description:
  Upon detection, the CRLs were re-validated, regenerated, and published using the appropriate private keys associated with the respective Intermediate CAs. The manual CRL generation workflow has been replaced with an automated process that enforces key validation and includes automated post-signing cryptographic verification. Additionally, monitoring of critical mailboxes has been strengthened to include daily quarantine reviews, ensuring timely processing of future external alerts.
• Commitment summary:
  All corrective and preventive actions disclosed in the incident report have been completed or are in progress. These include transitioning to automated CRL generation, introducing mandatory post-signing validation, and improving external alert monitoring. We respectfully request the closure of this incident.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 15

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-14.