1965559 - eMudhra: Delayed Publication of Issuing CA Certificates In CCADB

Assignee

Description

•

11 months ago

Incident Report

Summary

On 30th April 2025 eMudhra generated an Issuing CA certificate that was intended for publication to the CCADB in accordance with Mozilla's Root Store Policy. As per Section 5.3.2 of the Mozilla policy, Issuing CA certificates must be published within 1 week of generation. However, publication was completed on 09th May 2025, resulting in a delay of two days beyond the policy-mandated timeframe.

Impact

The Issuing CA certificate was published 2 days later than required by Mozilla's Root Store Policy.
No certificates were issued using this CA prior to its publication.
No impact on relying parties, subscribers, or certificate validity.

Timeline

All times are IST.

2025-04-30: Issuing CA certificate was generated.
2025-05-07: Deadline for publishing the certificate as per the 7-day requirement.
2025-05-10: Certificate was published to the CCADB.

Root Cause Analysis

The delay occurred due to a gap in the Maker-Checker process. The designated Maker completed the certificate artefact preparation and performed the initial validations as per internal compliance standards. However, the Checker’s review, which includes an independent validation against updated internal policies and externally mandated guidelines, was delayed. Given the increasing complexity and stringency of these guidelines, additional time was required to thoroughly complete the compliance review. This extended review period led to the delay in final approval and subsequent publication.
This incident also highlights the importance of incorporating more detailed advanced planning and internal feedback checkpoints during the preparation phase. Doing so will ensure that necessary resources, including validation reviewers, are aligned and available within the required timeframe.

Lessons Learned

Maker-Checker dependencies must be complemented by defined escalation and fallback roles. Time-based triggers and calendar alerts are essential to maintain compliance with strict publishing timelines. Advanced scheduling and alignment of review personnel is necessary to avoid future delays.

What went well

No certificates were issued using the delayed CA, ensuring there was no security or operational impact. The delay was identified and transparently.

What didn't go well

Reliance on manual coordination caused an avoidable delay. Publication deadline was not tracked with automated reminders or checkpoints.

Where we got lucky

The CA was not used before being published, avoiding potential compliance violations beyond disclosure. The oversight was caught internally and proactively reported.

Action Items

Action Item	Kind	Due Date
Formalized Maker-Checker timeline enforcement with escalation thresholds.	Prevent	2025-05-08
Introduced automated calendar-based alerts to track 5-day and 7-day checkpoints.	Prevent	2025-05-08

Based on Incident Reporting Template v. 2.0

chrome-root-program

Comment 1

•

11 months ago

Please provide an updated Full Incident Report aligned with Version 3.0 of the IRGs, which became effective on March 1, 2025.

Assignee: nobody → naveen.ml

Status: UNCONFIRMED → ASSIGNED

Ever confirmed: true

Summary: Delayed Publication of Issuing CA Certificates In CCADB → eMudhra: Delayed Publication of Issuing CA Certificates In CCADB

Whiteboard: [ca-compliance] [disclosure-failure]

chrome-root-program

Comment 2

•

10 months ago

This report has gone "stale" without a "Next Update" date set, and has not yet acknowledged or actioned the request in Comment 1.

Flags: needinfo?(naveen.ml)

Naveen Kumar ML

Assignee

Comment 3

•

10 months ago

Full Incident Report

Summary

CA Owner CCADB unique ID: A005678
Incident description:
On 30th April 2025 eMudhra generated an Issuing CA certificate that was intended for publication to the CCADB in accordance with Mozilla's Root Store Policy. As per Section 5.3.2 of the Mozilla policy, Issuing CA certificates must be published within 1 week of generation. However, publication was completed on 10th May 2025, resulting in a delay of two days beyond the policy-mandated timeframe.

-Timeline summary:

Non-compliance start date: 8 May 2025 (1 week, 1day after certificate generation)
-Non-compliance identified date: 10 May 2025 (when publication occurred and delay became apparent)
Non-compliance end date: 10 May 2025 (publication date)
Relevant policies: Mozilla Root Store Policy § 5.3.2 (“Publicly Disclosed and Audited”)
Source of incident disclosure: Self discovered as documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1965559

Impact

Total number of certificates: 24 Issuing CAs
Total number of "remaining valid" certificates: 24 Issuing CAs
Affected certificate types: Issuing CA Certificate
Incident heuristic:: Disclosure-failure (CCADB publication beyond allowed window)
Was issuance stopped in response to this incident, and why or why not?: No. No certificates had been issued under this CA prior to its publication since they were not yet in use, so there was no operational requirement to halt issuance after discovery.
Analysis: The late CCADB update constitutes a compliance breach of Mozilla’s “one week” rule, but posed no security or operational risk since the certificate was not yet in use.
Additional considerations: NA

Timeline

Timeline (All times in IST)
• 2025-04-30 – Issuing CA certificate was generated.
• 2025-05-07 – Deadline for publishing the certificate as per the 7-day requirement.
• 2025-05-10 – Certificate was published to the CCADB.

Related Incidents

There are no related eMudhra incidents

Root Cause Analysis

Contributing Factor 1: Delayed Checker Review

Description: The Checker’s compliance validation required after the Maker’s initial preparation was postponed due to manual coordination and resource availability constraints.
-Timeline: Checker review period extended beyond the planned 2-day window, pushing final approval past the 7-day publication deadline.
-Detection: The delay was detected when the publication done on 10 May 2025.
-Interaction with other factors: Absence of automated alerts allowed the deadline to pass unnoticed.
Root Cause Analysis methodology used: Five Whys analysis

Why was the CCADB update two days late?
Because the Issuing CA certificate wasn’t published within the one-week window mandated by Mozilla’s policy.
Why wasn’t it published on time?
Because the Maker-Checker review step and the subsequent CCADB submission was delayed.
Why was the Maker-Checker review delayed?
Because there were no formal SLAs or escalation thresholds enforcing when the Checker must complete its validation.
Why were there no formal SLAs or escalation thresholds?
Because the Maker-Checker and CCADB-submission process was a manual one, and no enforceable timeline checks or hand-off reminders were configured.
Why was the manual process not configured with timeline checks or reminders?
Because when the Maker-Checker procedure was defined, the Mozilla policy requirement for one-week CCADB publication was not translated into the checkpoints, so no reminders or escalation points were ever enforced.

Lessons Learned

What went well:
No certificates were issued using the delayed CA, ensuring there was no security or operational impact. The delay was identified and transparently addressed.
What didn’t go well:
Reliance on manual coordination caused an avoidable delay. Publication deadline was not tracked with automated reminders or checkpoints.
Where we got lucky:
The CA was not used before being published, avoiding potential compliance violations beyond disclosure. The oversight was caught internally and proactively reported.
Additional: NA

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Formalized Maker-Checker timeline enforcement with escalation thresholds	Prevent	Root Cause # 1	SoP Updated	2025-05-08	Completed
Introduced automated calendar-based alerts to track 5-day and 7-day checkpoints	Prevent	Root Cause # 1	SoP Updated	2025-05-08	Completed

Appendix

Flags: needinfo?(naveen.ml)

chrome-root-program

Comment 4

•

10 months ago

Thank you for providing the full incident report in Comment 3 using Version 3.0 of the IRGs.

While the nature of this incident might be considered relatively minor, eMudhra's continued failure to adhere to the CCADB IRGs is concerning. Specifically, the combination of eMudhra not using the most current version of the IRG templates, missing update deadlines, and seemingly disregarding specific instructions (e.g., “Related Incidents” MUST consider incidents beyond those corresponding to the CA Owner subject of this report.) - leads us to believe they are not monitoring reports filed to Bugzilla and applying lessons learned.

(1) Can you describe in detail how eMudhra is following incident reports from other CA Owners and applying lessons learned to its own policies and practices?

We note that the example provided above for disregarding specific IRG instructions is one of many instances where the guidelines were not followed in this report (e.g., “Relevant policies”, “Timeline”, etc.). We observe similar challenges in adhering to previous generations of the IRGs in past reports.

Further, and aside from other CA Owner incidents, eMudhra had a similar one recently (1924492). It seems that the Prevent Action Items from that incident report and the commitments described in the closure summary have failed to stop this type of incident from recurring.

(2) Considering the failure of these previous Action Items, what other Root Cause Analysis should be performed in this incident to develop more robust Action Items, so that this type of failure does not continue to repeat?

Naveen Kumar ML

Assignee

Comment 5

•

10 months ago

While the nature of this incident might be considered relatively minor, eMudhra's continued failure to adhere to the CCADB IRGs is concerning. Specifically, the combination of eMudhra not using the most current version of the IRG templates, missing update deadlines, and seemingly disregarding specific instructions (e.g., “Related Incidents” MUST consider incidents beyond those corresponding to the CA Owner subject of this report.) - leads us to believe they are not monitoring reports filed to Bugzilla and applying lessons learned.

We acknowledge that several aspects of this incident report including incomplete adherence to the IRG format, delayed updates, and a narrow interpretation of the “Related Incidents” section stemmed from human oversight and a lapse in our internal compliance review process. These were genuine errors, and we take full responsibility for them.

(1) Can you describe in detail how eMudhra is following incident reports from other CA Owners and applying lessons learned to its own policies and practices?

We note that the example provided above for disregarding specific IRG instructions is one of many instances where the guidelines were not followed in this report (e.g., “Relevant policies”, “Timeline”, etc.). We observe similar challenges in adhering to previous generations of the IRGs in past reports.

We had previously followed a decentralized model where members of the compliance team were individually responsible for tracking incident trends, monitoring IRG updates, and preparing reports. While roles were defined, this model lacked a coordinated review mechanism which contributed to oversights in this submission, particularly in referencing past incidents and ensuring complete alignment with IRG version 3.0.
We are addressing this gap through the following immediate steps:

-Increased Oversight: All incident reports will now be reviewed by a senior compliance reviewer before submission. Where reports require
policy interpretation, a second-level internal check will be performed.
-Monthly Learning Reviews: A dedicated session has been introduced where the compliance team reviews recent high-impact incidents from other CAs and discusses their relevance. This ensures shared understanding and better internal alignment on emerging issues.

Further, and aside from other CA Owner incidents, eMudhra had a similar one recently (1924492). It seems that the Prevent Action Items from that incident report and the commitments described in the closure summary have failed to stop this type of incident from recurring.

(2) Considering the failure of these previous Action Items, what other Root Cause Analysis should be performed in this incident to develop more robust Action Items, so that this type of failure does not continue to repeat?

We agree that similarities with incident 1924492 demonstrate the need for deeper reinforcement and better follow-through on preventive actions. While actions were defined, their sustained implementation was not verified adequately.
To prevent this from recurring:

-RCA Revalidation: We have reopened the internal review of earlier controls to verify effectiveness and identify areas of weak implementation.
-Targeted Debriefs: The compliance lead is now directly engaging with key team members to ensure a stronger understanding of reporting expectations and the rationale behind corrective actions.
-Follow-Up Reviews: Preventive actions from any incident will now be reviewed for status and impact within 30 days of implementation. Items not completed or seen as ineffective will be escalated for immediate resolution.

We fully understand that even minor reporting lapses can erode confidence when seen as recurring. This incident has prompted important internal adjustments, and we are committed to implementing them with discipline and transparency.
We appreciate your continued engagement and welcome any further feedback.

Naveen Kumar ML

Assignee

Comment 6

•

10 months ago

No further action required at this time.

incident-reporting

Comment 7

•

10 months ago

If requesting closure of this report, please follow the guidance on CCADB.org.

Flags: needinfo?(naveen.ml)

Naveen Kumar ML

Assignee

Comment 8

•

10 months ago

Report Closure Summary

Incident description:
On 30th April 2025 eMudhra generated 24 Issuing CA certificates that was intended for publication to the CCADB in accordance with Mozilla's Root Store Policy. As per Section 5.3.2 of the Mozilla policy, Issuing CA certificates must be published within 1 week of generation. However, publication was completed on 10th May 2025, resulting in a delay of two days beyond the policy-mandated timeframe.
Incident Root Cause(s):
The delay was caused by a gap in the Maker–Checker process. Specifically, the Checker’s compliance review was extended beyond the planned 2-day window and the absence of formal SLAs or automated reminders, causing the final approval and CCADB submission to miss the policy-mandated timeframe.
Remediation description:
To address this, the Standard Operating Procedures were updated to include:

Formalized Maker–Checker timeline enforcement with defined escalation thresholds.
Automated calendar-based alerts to track 5-day and 7-day publication checkpoints.

Commitment summary:
All remediation actions were completed by 8 May 2025. eMudhra commits to adherence to Mozilla’s CCADB publication timelines for all future certificates and will maintain continuous monitoring via automated alerts to ensure compliance and prevent recurrence.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Flags: needinfo?(naveen.ml)

incident-reporting

Updated

•

10 months ago

Flags: needinfo?(incident-reporting)

Naveen Kumar ML

Assignee

Comment 9

•

9 months ago

We are continuing to monitor this issue.

Naveen Kumar ML

Assignee

Comment 10

•

9 months ago

Weekly Status Update

The closure report associated to this bug has been submitted. Please close if no other comments are provided.

incident-reporting

Comment 11

•

9 months ago

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-01.

incident-reporting

Updated

•

9 months ago

Flags: needinfo?(incident-reporting)

Whiteboard: [ca-compliance] [disclosure-failure] → [close on 2025-07-01] [ca-compliance] [disclosure-failure]

incident-reporting

Updated

•

9 months ago

Status: ASSIGNED → RESOLVED

Closed: 9 months ago

Resolution: --- → FIXED

incident-reporting

Updated

•

9 months ago

Whiteboard: [close on 2025-07-01] [ca-compliance] [disclosure-failure] → [ca-compliance] [disclosure-failure]