Closed Bug 1914466 Opened 1 year ago Closed 1 year ago

eMudhra emSign PKI Services: CA Certificates not published in DER Encoded Format

Tracking

(Not tracked)

Status:

RESOLVED FIXED

People

(Reporter: naveen.ml, Assigned: naveen.ml)

Details

(Whiteboard: [ca-compliance] [policy-failure])

Naveen Kumar ML

Assignee

Description

•

1 year ago

Incident Report

Summary

An external researcher advised eMudhra that one of the CA certificates published to the repository at a specified AIA location were PEM encoded instead of the required DER encoding, which does not comply with RFC 5280 Section 4.2.2.1. Upon investigation it was determined that all CA certificates published to the repository at encoded AIA URIs from 2018-02-20 - 2024-03-14 were PEM encoded due to a manual error in our publication process. The process lacked sufficient validation controls to ensure the correct DER format file was used, leading to the publication of PEM-encoded files instead, that are typically produced to be provided on demand to customers.

Impact

The impact of this incident was minimal, as no customers reported issues with certificate usage during the investigation. However, the publication of 56 PEM-encoded certificates instead of the required DER format at the URI location posed a potential risk of non-compliance with industry standards (RFC 5280) and could have led to interoperability issues if left unaddressed. The swift resolution ensured that the correct encoding of the certificates were published, mitigating any potential negative effects.

Timeline

All times are IST.

2018-02-20 - 2024-03-14 : The emSign CA issued and published a total of 56 Root and CA certificates to the cacert AIA file repository, which are the subject of this incident. These certificates, detailed in the Appendix, were uploaded as part of our standard certificate publishing process but have now been identified as requiring further review due to the concerns raised in this incident.

2024-08-21 13:00 : We internally raised an incident (Incident No. EMINCPKI0018) with our emSign PKI and compliance group regarding one of the CA issuer certificates (http://repository.emsign.com/certs/emSignEVSSLCAG1.crt) that was published to the AIA file repository in it’s PEM format. This incident was prompted by an email from an external researcher, sent to our emSign problem reporting address. The incident was categorized as low priority and initiated for investigation, as no issues had been reported to date by any customers regarding the certificate's usage prior to the incident being raised.

2024-08-21 15:15 : The emSign PKI and compliance group initiated an investigation to determine whether the AIA links were encoded in PEM or DER format. The investigation involved a thorough review of all certificates published in the repository to ensure that those referenced in AIA were correctly encoded in DER format. During this process, it was identified that 56 Root and CA certificates had been published to the repository encoded in PEM format.

2024-08-21 16:45 : The emSign PKI and compliance group conducted a detailed investigation of the certificates published in PEM format, which revealed that 56 CA certificates were not DER encoded.

2024-08-21 16:55 : The emSign PKI and compliance group notified a formal communication was sent to PA committee on the incident and transferred the incident to emSign PKI core team.

2024-08-22 14:30 : The emSign PKI core team took charge of the incident and identified the DER format files for all 56 certificates. These were then published in the pre-live environment and underwent testing to ensure compliance with the guidelines outlined in RFC 5280.

2024-08-22 21:20 : The emSign PKI core team successfully published the 56 DER-encoded certificates to the production repository at the specified AIA locations and made them live.

Root Cause Analysis

The issue stemmed from the emSign CA's manual publication process, which manages both PEM and DER formats for certificate distribution. Although PEM-encoded certificates are typically provided to customers on demand, a DER-encoded certificate file intended for publication was mistakenly replaced with the PEM-encoded file in the corresponding publishing procedure. This error was due to a procedural gap in the manual workflow, where the distinction between the formats was not adequately enforced, leading to the incorrect selection of the PEM-encoded certificate for publishing.

The problem arose because, during the manual certificate publishing process, a PEM-encoded file normally supplied to customers on request was selected to be published instead of the required DER-encoded version. This occurred because the process lacked sufficient checks to ensure that the DER format was used, resulting in the PEM-encoded format of the certificate being provided by the repository at the AIA location.

Lessons Learned

Lessons learned from this incident include the importance of implementing stricter validation checks during the certificate publication process to ensure compliance with encoding standards. The incident highlighted the need for enhanced automation and review mechanisms to prevent similar errors in the future. Additionally, it underscored the value of proactive monitoring and swift response to external reports to maintain the integrity and reliability of our PKI services.

What went well

The seamless coordination between the PKI and compliance teams ensured a thorough review and quick resolution. The successful transition of corrected certificate formats to the live environment, with adherence to RFC 5280 standards, demonstrated the team's ability to efficiently manage and resolve incidents.

What didn't go well

The initial oversight in the publication process, where the PEM-encoded certificates were selected to be published instead of the required DER format. Additionally, the lack of automated validation checks allowed this error to go unnoticed until it was reported externally.

Where we got lucky

We got lucky that the issue was discovered by an external researcher before any customers reported interoperability problems, preventing potential operational disruptions. The other mitigating circumstances here seems to be that most modern crypto handling software appears to gracefully handle retrieving CA certificate files in either PEM or DER format from a valid location specified in the AIA extension of a certificate. Also, the timely detection allowed us to correct the non-compliance quickly. Moreover, the swift resolution helped us avoid any significant impact on customer trust or service reliability.

Action Items

Action Item	Kind	Due Date
Review all the CA certificates published in repository to check encoding format.	Mitigated	2024-08-21
Replace 56 certificates that had PEM-encoded version with their DER encoded formatted version of certificates.	Mitigated	2024-08-22
Implementation of an enhanced process that includes additional validation checks when publishing CA certificates in DER-encoded format.	Mitigate	2024-08-26

Appendix

Details of affected certificates

Below are the AIA URIs for the above certificates:

Impacted Requirements:

CA/B Forum Baseline Requirements:
Section 7.1.2 Certificate Content and Extensions
“all certificates that it issues MUST comply with one of the following certificate profiles, which incorporate, and are derived from RFC 5280. Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”
RFC 5280:
Section 4.2.2.1. Authority Information Access
“Where the information is available via HTTP or FTP, accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in [RFC2585] or a collection of certificates in a BER or DER encoded "certs-only" CMS message as specified in [RFC2797].”

Based on Incident Reporting Template v. 2.0

Ben Wilson

Updated

•

1 year ago

Assignee: nobody → naveen.ml

Status: UNCONFIRMED → ASSIGNED

Ever confirmed: true

Whiteboard: [ca-compliance] [policy-failure]

Naveen Kumar ML

Assignee

Comment 1

•

1 year ago

Our team has addressed all the items related to this incident. We kindly request that the incident now be marked as resolved.

Action Items

Action Item	Kind	Due Date
Review all the CA certificates published in repository to check encoding format.	Mitigated	2024-08-21
Replace 56 certificates that had PEM-encoded version with their DER encoded formatted version of certificates.	Mitigated	2024-08-22
Implementation of an enhanced process that includes additional validation checks when publishing CA certificates in DER-encoded format.	Mitigated	2024-08-26

Ben Wilson

Comment 2

•

1 year ago

I'll look at closing this on Friday, 30-Aug-2024.

Flags: needinfo?(bwilson)

Ben Wilson

Updated

•

1 year ago

Status: ASSIGNED → RESOLVED

Closed: 1 year ago

Flags: needinfo?(bwilson)

Resolution: --- → FIXED

Chris Clements

Updated

•

1 year ago

Summary: emSign PKI Services: CA Certificates not published in DER Encoded Format → eMudhra emSign PKI Services: CA Certificates not published in DER Encoded Format

You need to log in before you can comment on or make changes to this bug.

Bugzilla

eMudhra emSign PKI Services: CA Certificates not published in DER Encoded Format

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

People

(Reporter: naveen.ml, Assigned: naveen.ml)

References

Details

(Whiteboard: [ca-compliance] [policy-failure])

Crash Data

Security

(public)

User Story

Description

Incident Report

Summary

Impact

Timeline

Root Cause Analysis

Lessons Learned

What went well

What didn't go well

Where we got lucky

Action Items

Appendix

Details of affected certificates

Updated

Comment 1

Action Items

Comment 2

Updated

Updated