1763700 - eMudhra: emSign CA Invalid AIA Extension Value

Assignee

Description

•

3 years ago

Summary:
84 SSL certificates were wrongly issued with wrong Authority Information Access (AIA) values.

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On 06-Apr-2022 13:36 (Indian Time), we received an email reporting a certificate with invalid AIA value. Connected to this was the sslmate utility report on OCSP watch (thanks to Andrew Ayer) referenced through a discussion in mozilla.dev.security.policy.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
(All times are in Indian Time / UTC+5:30)
22-Mar-2022 10:43 The reported issue started.
06-Apr-2022 13:36 We received the reporting email.
06-Apr-2022 15:00 We have confirmed the certificate is actually mis-issued and start the investigation.
06-Apr-2022 15:30 The system configuration inspection and resolution to mitigate the issue is initiated.
06-Apr-2022 15:50 The system configuration changes are confirmed and completed towards mitigate the issue. All new issuances are verified to be proper.
06-Apr-2022 19:00 The system scanning to identify all affected issuances is completed. The results were found containing 84 certificates with similar issue.
07-Apr-2022 16:00 Notification sent to Subscribers of these affected certificates.
07-Apr-2022 17:27 The affected certificate revocation completed.
07-Apr-2022 18:00 The replacement Certificates issued to these affected Subscribers are verified for successful issue mitigation.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We have stopped issuing certificates with the problem, and resolved this for future issuances.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
We have checked all the valid certificates and found 84 SSL certificates containing invalid AIA values as per above issue. These certificates are dated between 22-Mar-2022 and 06-Apr-2022.

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
CRT.SH IDs:
6488407374
6488398191
6487869516
6487163999
6486565975
6481663433
6481002897
6480997231
6480966762
6480560164
6477457581
6475521640
6475447468
6474512753
6471227772
6470485209
6464164503
6461800724
6461223775
6459394945
6456860756
6456754494
6454616421
6453188008
6453151197
6452982252
6452712203
6448926794
6448515642
6448005882
6447585555
6447334178
6447168949
6447158818
6446662526
6446086117
6440466478
6440361377
6440042042
6436311533
6434718753
6434632388
6434059978
6433867900
6433848054
6428690037
6428628260
6428155102
6428013035
6427574124
6427499938
6427247469
6427247469
6417667821
6417155497
6416813549
6416459178
6416300382
6415401840
6415219224
6415212160
6412889386
6412516333
6410783079
6410355685
6409720066
6409359279
6408803891
6408675663
6408631995
6404592404
6404365633
6404055211
6403049017
6396891551
6396385241
6396299873
6392439672
6390613280
6389894423
6389764049
6389689150
6389689255
6389600732

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
For issuance of these certificates, there was a misconfiguration of AIA values, due to a human error. Given that this issue was not reflected as part of system checks / results, the issue went undetected. The lint tests were actively there, but they do not verify the URL accuracies.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
A human error possibility of causing this issue was unexpected and unforeseen, as the configuration of each field goes via dual checks. However, it looks it has caused misconfiguration resulting certificate misissuances through website based requests. Hence, considering the severity of such mistakes, the configuration have been scrutinized in detail to not result into this issue. Additional training is provided to all personnel involved in such operation. The configuration changes have been moved to another layer of validation, before applying in production.

Ben Wilson

Updated

•

3 years ago

Assignee: bwilson → vijay

Status: UNCONFIRMED → ASSIGNED

Ever confirmed: true

Whiteboard: [ca-compliance]

Ryan Sleevi

Comment 1

•

3 years ago

(In reply to Vijay Kumar from comment #0)

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
For issuance of these certificates, there was a misconfiguration of AIA values, due to a human error. Given that this issue was not reflected as part of system checks / results, the issue went undetected. The lint tests were actively there, but they do not verify the URL accuracies.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
A human error possibility of causing this issue was unexpected and unforeseen, as the configuration of each field goes via dual checks. However, it looks it has caused misconfiguration resulting certificate misissuances through website based requests. Hence, considering the severity of such mistakes, the configuration have been scrutinized in detail to not result into this issue. Additional training is provided to all personnel involved in such operation. The configuration changes have been moved to another layer of validation, before applying in production.

From https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

For example, it’s not sufficient to say that “human error” of “lack of training” was a root cause for the incident, nor that “training has been improved” as a solution. While a lack of training may have contributed to the issue, it’s also possible that error-prone tools or practices were required, and making those tools less reliant on training is the correct solution. When training or a process is improved, the CA is expected to provide specific details about the original and corrected material, and specifically detail the changes that were made, and how they tie to the issue. Training alone should not be seen as a sufficient mitigation, and focus should be made on removing error-prone manual steps from the system entirely.

Can you redo the answers here to provide clearer, and more concrete, root cause analysis?

Also,

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Flags: needinfo?(vijay)

Vijay Kumar

Assignee

Comment 2

•

3 years ago

Sure Ryan. An update to this thread will be posted within next few days. (before 14-Apr-2022).

Flags: needinfo?(vijay)

Mathew Hodson

Comment 3

•

3 years ago

The list of certificates in this report doesn't use the recommended form https://crt.sh/?sha256=[sha256-hash] discussed in https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/1NQb9Ktpr3E and added to https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Was there a reason this report didn't use the recommended form? Does eMudhra follow the dev-security-policy@mozilla.org mailing list?

Vijay Kumar

Assignee

Comment 4

•

3 years ago

Hi Mathew. Thanks for bringing that up. We follow public discussions and try our best to keep a summary/outcome of conversation, wherever applicable. Sorry that this incident report missed updating the full URLs.

We are adding below as an update to the thread.

Vijay Kumar

Assignee

Comment 5

•

3 years ago

(In reply to Ryan Sleevi from comment #1)

Can you redo the answers here to provide clearer, and more concrete, root cause analysis?

Also,

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

This an update from the Root Cause Analysis with more information. While the cause has been human error of wrong configuration for this particular field, this has been the result of lack of user interface intuitiveness in configuration screen.

The administrators managing this configuration have undergone necessary training, as well as have user manuals to perform such configuration. However, we have identified lack of intuitiveness which can result in such errors, as the fields are more technically named, instead of appropriate user-friendly naming conventions in the UI. (Eg: Field 1 and 2 under AIA).

As a resolution against future occurrence, we have initiated change request to the admin application to update descriptive fields names, so that the user and reviewer can make sure that the configurations are done as intended.

We have an expected update in this regard (patch) by 30-Apr-2022 and we will update this thread once we have successfully completed the update.

In terms of the issue resolution, the effected certificates are already revoked and new issuance are reviewed to conform to requirements.

Vijay Kumar

Assignee

Comment 6

•

3 years ago

We have nothing new to report this week. The software update is scheduled next week, by when we will update the closure of our pending actions.

Vijay Kumar

Assignee

Comment 7

•

3 years ago

(In reply to Vijay Kumar from comment #5)

We have an expected update in this regard (patch) by 30-Apr-2022 and we will update this thread once we have successfully completed the update.

The software update as stated above has been deployed successfully, last week. We have examined that the new descriptive interface implemented should avoid erroneous configuration found in the RCA of this issue.

We confirm that all remediation's identified in this bug have now been completed. We request for this bug to please be marked as Resolved, in case there are no other comments.

Ben Wilson

Updated

•

3 years ago

Status: ASSIGNED → RESOLVED

Closed: 3 years ago

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Updated

•

3 years ago

Product: NSS → CA Program

Chris Clements

Updated

•

2 years ago

Summary: eMudhra emSign CA: Invalid AIA Extension Value → eMudhra: emSign CA Invalid AIA Extension Value

David Lawrence [:dkl]

Updated

•

2 years ago

Whiteboard: [ca-compliance] → [ca-compliance] [dv-misissuance]

Bugzilla

eMudhra: emSign CA Invalid AIA Extension Value

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

People

(Reporter: vijay, Assigned: vijay)

References

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

Crash Data

Security

(public)

User Story

Description

Updated

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Updated

Updated

Updated

Updated