1999241 - eMudhra emSign PKI Services : Delayed Publication of Issuing CA Certificates in CCADB

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Naveen Kumar ML]

Attachment: Annexure List of CA Certificates.xlsx

Full Incident Report

Summary

• CA Owner CCADB unique ID:
  A011830

• Incident description:
  Two subordinate issuing CA certificates, emSign TLS CA - G1 and emSign TLS CA - G3, two cross-sign CA certificates emSign Root TLS CA - G3 and emSign Root TLS CA - G1, were created on July 11, 2024 as part of eMudhra’s CA hierarchy design to facilitate issuance of initial certificates for test websites during the browser root program evaluation process.
  As per Section 8 of the Google Chrome Root Program Policy v1.5, issuing CAs must be uploaded to the CCADB within seven days of creation. However, emSign TLS CA - G1 and emSign TLS CA - G3 issuing CAs were uploaded on September 16, 2024, and emSign Root TLS CA - G3 and emSign Root TLS CA - G1 cross-sign CA certificates were uploaded on September 23, 2024, exceeding the allowed timeframe.
  The non-compliance was identified in November 2025 as part of Chrome’s metadata verification activities.

• Timeline summary:

  • Non-compliance start date:
    July 11, 2024 (Issuing CAs and Cross-sign CAs created)
  • Non-compliance identified date:
    November 9, 2025 (identified during metadata verification)
  • Non-compliance end date:
    September 23, 2024 (CAs published in CCADB)

• Relevant policies:
  Google Chrome Root Program Policy v1.5, Section 8 – “All intermediate and issuing CA certificates must be uploaded to CCADB within seven days of creation".

• Source of incident disclosure:
  Chrome Root Program metadata review during eMudhra’s root inclusion evaluation.

Impact

• Total number of certificates:
  2 (Issuing CA certificates)
  2 (Cross-sign CA certificates)

• Total number of "remaining valid" certificates:
  4

• Affected certificate types:
  TLS issuing CA certificates and Cross-sign CA certificates

• Incident heuristic:
  Administrative / procedural oversight in publication timelines

• Was issuance stopped in response to this incident, and why or why not?:
  Not applicable. These CAs had not being used for customer-facing or production issuance during the non-compliance period.

• Analysis:
  Although the issuing CAs were generated correctly as per cryptographic and ceremony requirements, the procedural workflow for ensuring CCADB publication within seven days was not strictly enforced. This gap stemmed from the lack of a dedicated maker/checker control mechanism and no defined compliance review milestone post issuing CA creation.
  Following the similar incident (Bug 1965559) that was self-reported, eMudhra conducted a comprehensive review of its CCADB publication procedures and has begun enforcing stricter controls and systematic tracking.

• Additional considerations:
  The non compliance was administrative only. All four remain unused for any active issuance during the period of non-compliance, and there was no impact on relying parties or revocation infrastructure.

Timeline

All times are IST.
11-07-2024 20:22:21 to 11-07-2024 22:12:13: emSign TLS CA - G1 and G3 CA certificates, emSign Root TLS CA - G3 and emSign Root TLS CA - G1 cross-sign CA certificates were generated.
16-09-2024: emSign TLS CA - G1 and G3 CA issuing CA certificates uploaded to CCADB
23-09-2024: emSign Root TLS CA - G3 and emSign Root TLS CA - G1 cross-sign CA certificates uploaded to CCADB
08-11-2025 01:29: Chrome Root Program identified delay during metadata verification
08-11-2025 12:45: An interim response was sent to the Google Root Program team about raising the incident report.
08-11-2025 13:30: Internally raised an incident for delayed publication of CA certs to CCADB
08-11-2025 20:30: Prior incident (Bug 1965559) reviewed to ensure process alignment and avoid recurrence
10-11-2025 : eMudhra prepared and submitted this incident disclosure

Related Incidents

Bug	Date	Description
1965559	08-11-2025 01:29	Similar incident involving delayed publication of issuing CA certificates in CCADB.
1904041	21-06-2024 18:31	Intermediate CA Certificate not disclosed to CCADB
1921596	28-09-2024 14:48	Failure to disclose intermediate certificate within 7 days in CCABD.

Root Cause Analysis

Contributing Factor #: 1:
Absence of a dedicated compliance checkpoint for newly created test or validation issuing CAs

• Description:
  The publication of issuing CA certificates to CCADB was a manual process, lacking a structured approval checkpoint or reminder system to enforce the seven-day rule.
• Timeline:
  Creation: July 11, 2024 → Publication for emSign TLS CA - G1 and emSign TLS CA - G3: September 16, 2024
  Creation: July 11, 2024 → Publication for emSign Root TLS CA - G3 and emSign Root TLS CA - G1: September 23, 2024
• Detection:
  Identified during Chrome Root Program metadata verification in November 2025.
• Interaction with other factors:
  The absence of downstream use or end-entity issuance under these CAs delayed any urgency or review.
• Root Cause Analysis methodology used:
  Internal procedural audit and compliance review mapping against Chrome Root Program policy timelines.

Lessons Learned

• What went well:
  The issuing CAs were generated securely following all ceremony requirements. No end entity or customer certificates were issued during the period of non-compliance.
• What didn’t go well:
  CCADB publication timelines were not cross checked as part of the post issuance compliance verification process for validation and it was not enforced due to missing procedural controls and no automated or human second-level check
• Where we got lucky:
  The CAs were not used for any operational or production issuance, and no ecosystem or relying party dependency existed during the delay.
• Additional:
  The second such oversight prompted a thorough review of issuing CA publication governance. A revised process has been implemented with tracking checkpoints and maker-checker responsibilities.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Establish mandatory maker/checker workflow for all CCADB publications	Prevent	Root Cause # 1	Verified via checklist enforcement for new CAs	2025-05-08	Completed
Add CCADB publication tracking step for all new issuing CAs (including test or validation CAs)	Prevent	Root Cause # 1	Verified publication logs for each new CA creation	2025-05-08	Completed
Conduct refresher training for PKI operations and compliance staff on CCADB publication timelines	Prevent	Root Cause # 1	Training attendance and awareness check completed	2025-11-21	In Progress

Appendix

Affected certificates are published in the attachment

Comment 1

[Wayne]

This incident originally occurred in 2024-07-11 and was corrected in 2024-09-23 when the records were updated on CCADB.

Q1: If eMudhra are paying attention to other CA incidents, why did the incident raised 2024-09-28 get overlooked?

Previously eMudhra have stated they "previously followed a decentralized model where members of the compliance team were individually responsible".
Q2: Was there any personnel overlap for CCADB updates and reviewing other CA incidents in that timeframe?

Q3: Did eMudhra have any meetings to discuss similar incidents around this timeframe?

Notably in 2024-10-14 a different eMudhra CCADB incident occurred: #1924492: eMudhra emSign PKI Services: Failure To Update CA Owner Information In CCADB. Amongst the action items include:

Action Item	Kind	Due Date
Review and improve processes for CCADB updates	Prevent	2024-10-11
Conduct periodic review of CCADB data	Prevent	2024-12-15

Q4: Was the process review and periodic review of CCADB data solely restricted to ownership information for that incident?

It is also concerning that an identical incident occurred this May but no retrospective review occurred to highlight this issue proactively.
Q5: Does eMudhra believe that refresher training will stop a recurrence of this incident for a third time?

Q6: How confident are eMudhra that this is the extent of their non-compliance to date?

Q7: When does eMudhra think that non-compliance ended - when the data was updated onto CCADB, when they were made aware of the non-compliance, or when they raised an incident to show that non-compliance occurred?

As of now I believe the action items to not cover anywhere near the required improvements to show trust in eMudhra's internal processes.

Comment 2

[Naveen Kumar ML]

Q1: If eMudhra are paying attention to other CA incidents, why did the incident raised 2024-09-28 get overlooked?
Previously eMudhra have stated they "previously followed a decentralized model where members of the compliance team were individually responsible".

eMudhra operated with a decentralised responsibility model: external CA incidents were monitored by the Compliance function, while CCADB submissions, including publication of Issuing CA certificates, were handled separately by the operational team.
Although the Compliance team did review the 2024-09-28 incident raised by another CA Owner, the learnings were not effectively transmitted to the operational team. This resulted from insufficient cross-team process linkage rather than a lack of attention.
At that time, our internal process had gaps which did not effectively map external incidents to update operational runbooks or publication checklists. This gap meant that while the issue was understood conceptually, it was not guaranteed to be implemented operationally.
This decentralised model is precisely what we corrected in 2025 after Bug 1965559, when we moved CCADB responsibilities under centralised compliance oversight with mandatory cross-team review.

Q2: Was there any personnel overlap for CCADB updates and reviewing other CA incidents in that timeframe?

There was no personnel overlap.
• External incident monitoring (Bugzilla, CCADB announcements, industry disclosures) was owned by the Compliance team.
• CCADB publication of Issuing CA certificates was performed by the operational PKI team.
The earlier process relied on information handoff between these teams, which was inconsistent due to decentralisation and changes in personnel. As a result, awareness gained from reviewing other CA incidents did not get effectively transmitted.
This structural weakness, rather than individual error, is the root reason why learnings were not applied consistently. This has now been corrected by placing CCADB responsibilities under a single accountable function, with mandatory review gates.

Q3: Did eMudhra have any meetings to discuss similar incidents around this timeframe?
Notably in 2024-10-14 a different eMudhra CCADB incident occurred: #1924492: eMudhra emSign PKI Services: Failure To Update CA Owner Information In CCADB. Amongst the action items include:
| Action Item | Kind | Due Date |
| Review and improve processes for CCADB updates| Prevent | 2024-10-11 |
| Conduct periodic review of CCADB data| Prevent | 2024-12-15 |

Yes, multiple internal meetings took place.
However, those meetings lacked structured oversight and did not include:
• A clear mapping of external incidents to internal obligations
• A mandatory checklist with reviews specific to Issuing CA creation
• A senior management approval gate
In 2024, our process lacked sufficient maturity and was decentralised, and while issues were discussed, there was no strong governance mechanism ensuring that the discussions translated into procedural updates particularly for CCADB updates.
This is exactly what was corrected post-May 2025. Now:
• Incident reviews are conducted jointly by Compliance + PKI operations + Governance.
• A mandatory procedure runbook defines “publication to CCADB within 7 days” as a required step in Issuing CA creation.
• Senior management explicitly enforces checklist completion.

Q4: Was the process review and periodic review of CCADB data solely restricted to ownership information for that incident?
It is also concerning that an identical incident occurred this May but no retrospective review occurred to highlight this issue proactively.

No. The review initiated after the 2024-10-14 incident (Bug 1924492) was not limited only to ownership information. That incident prompted the compliance team to reassess CCADB-related tasks more broadly, but the process in 2024 still operated under a decentralized model, where CCADB updates, including publication of newly created Issuing CAs, were not governed by a formal maker/checker workflow.
Because CCADB publication for newly created Issuing CAs was not embedded as a required checkpoint in the operational runbook, the review conducted in 2024 did not surface this specific gap. The earlier review improved accuracy of existing CCADB entries but did not extend deeply enough into the workflow for onboarding new Issuing CAs.
A comprehensive and centralized review of all CCADB obligations, specifically including publication timelines for Issuing CA certificates, was only enforced after the second occurrence identified in May 2025 (Bug 1965559). Those strengthened controls now address the exact procedural oversight that existed in 2024.

Q5: Does eMudhra believe that refresher training will stop a recurrence of this incident for a third time?

Refresher training alone is not sufficient, and we do not rely on training as the only preventive measure. The recurrence of this issue demonstrated that the underlying problem was not a lack of knowledge, but the absence of a clearly defined and consistently enforced procedural checkpoint specifically for CCADB publication of newly created Issuing CA certificates.
Since May 2025, following the voluntarily reported incident in Bug 1965559, eMudhra implemented a strengthened maker/checker step, centralized responsibility under a smaller accountable group, and introduced an explicit CCADB-update checkpoint into the issuance workflow for every new Issuing CA. These controls are procedural rather than training-dependent, and they are now mandatory in the updated runbook reviewed by senior management.
Refresher training supports awareness, but the core prevention mechanism is the formalized and enforced publication checkpoint, which did not previously exist in the 2024 decentralized model. We are therefore confident that the current structure materially reduces the likelihood of recurrence.

Q6: How confident is eMudhra that this is the extent of their non-compliance?

Our review focused on the Issuing CAs created in 2024, as they were part of the same workflow that led to this delay. Within this scope, no additional delayed CCADB publications were identified. As noted in Bug 1965559, our earlier decentralized structure contributed to process gaps, but since mid-2025 we have strengthened and centralized controls around CCADB updates. Based on these improvements and the checks performed within the relevant timeframe, we are confident that the identified issues have been addressed, while remaining committed to promptly correcting any future concerns if they arise.

Q7: When does eMudhra believe the non-compliance ended?

We consider the technical non-compliance to have ended on:
23-September-2024
when the Issuing CA certificates (G1 and G3) were successfully published in CCADB.
However, in the spirit of transparency, we acknowledge that:
• Awareness of the issue occurred only during Chrome's metadata verification in November 2025.
• The formal incident disclosure is being made now to ensure full community transparency.
This aligns with the approach we previously adopted in Bug 1965559, where we reported earlier deviations voluntarily even after they had been corrected.

Comment 3

[Naveen Kumar ML]

Weekly Status Update

We are pleased to report that all action items identified in the incident report have been successfully completed as per the schedule. The details are as follows:

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Establish mandatory maker/checker workflow for all CCADB publications	Prevent	Root Cause # 1	Verified via checklist enforcement for new CAs	2025-05-08	completed
Add CCADB publication tracking step for all new issuing CAs (including test or validation CAs)	Prevent	Root Cause # 1	Verified publication logs for each new CA creation	2025-05-08	completed
Conduct refresher training for PKI operations and compliance staff on CCADB publication timelines	Prevent	Root Cause # 1	Training attendance and awareness check completed	2025-11-21	completed

Comment 4

[Naveen Kumar ML]

Weekly Status Update

No further action required at this time.

Comment 5

[Naveen Kumar ML]

Weekly Status Update

No further action required at this time.

Comment 6

[incident-reporting]

All Action Items have been marked as completed and there are no unanswered questions from the community.

Please submit a closure report if this report is ready to be closed.

Comment 7

[Naveen Kumar ML]

Report Closure Summary

• Incident description:
  Two issuing CA certificates (emSign TLS CA – G1 and emSign TLS CA – G3) and two cross-sign CA certificates (emSign Root TLS CA – G3 and emSign Root TLS CA – G1) were created on July 11, 2024, were created on 11 July 2024 as part of the hierarchy setup for browser root program evaluation. Under Section 8 of the Google Chrome Root Program Policy v1.5, all issuing CA certificates must be published in the CCADB within seven days of creation. The issuing CAs were published on September 16, 2024, and the cross-sign CAs on September 23, 2024, which exceeded the permitted timeframe.
  This non-compliance was identified on November 9, 2025, during Chrome’s metadata verification activities. The affected CAs had not been used for any active issuance, and therefore no impact occurred to relying parties or production infrastructure.
  The delay stemmed from an absence of a structured compliance checkpoint and lack of a dedicated maker/checker control for CCADB publication timelines. Following the identification of a similar earlier incident (Bug 1965559), eMudhra strengthened its internal governance, implemented publication tracking, and introduced mandatory procedural controls to prevent any recurrence.

• Incident Root Cause(s):
  The primary root cause was a procedural and governance gap: CCADB publication of newly created issuing and cross-sign CAs was handled as a manual task without a formal, mandatory compliance checkpoint or maker/checker control. Responsibilities were decentralized between Compliance (monitoring external incidents and policies) and PKI Operations (performing CCADB updates), with no robust mechanism to ensure that external learning’s and requirements translated into specific updates to operational run books. This structural weakness allowed the 7-day publication requirement to be overlooked, despite earlier industry incidents and internal reviews.

• Remediation description:
  eMudhra implemented the following corrective and preventive measures:
  • Centralized accountability for CCADB updates under a single, clearly designated function with joint review by Compliance, PKI Operations and Governance.
  • Established a mandatory maker/checker workflow for all CCADB publications, with checklist-based enforcement for every new issuing CA, including test or validation CAs.
  • Embedded an explicit “publish to CCADB within 7 days” checkpoint into the issuing-CA creation runbook, along with tracking of publication logs for each newly created CA.
  • Conducted refresher training for PKI operations and compliance staff on CCADB publication timelines and expectations under the Chrome Root Program Policy.

All action items defined in the incident report are now marked as completed, with verification through updated checklists, logs, and training records.

• Commitment summary:
  eMudhra considers the technical non-compliance to have ended on 23 September 2024, when the affected issuing and cross-sign CA certificates were successfully published in CCADB.
  Going forward, eMudhra is committed to:
  • Maintaining the strengthened centralized governance model for all CCADB-related activities, including periodic reviews of CCADB data.
  • Enforcing the maker/checker workflow and 7-day publication checkpoint for every new issuing CA.
  • Continuing joint incident reviews across Compliance, PKI Operations and Governance to ensure that external incidents and policy updates are rapidly translated into internal procedures.
  • Promptly disclosing any future deviations, even if already technically corrected, in line with the transparency expectations of root program operators.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 8

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-12-24.