1876871 - IdenTrust: test certificates inadvertently published in production environment

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

Incident Report

This is a preliminary report

Summary

We've identified some test S/MIME and TLS test certificates that were mistakenly issued in the CA production environment instead of the designated CA test environment, bypassing the CA/B Forum BRs vetting process. All uncovered certificates have either expired or were revoked within minutes of issuance.
We are still investigating the root cause and will post a complete incident report by February , 2024

Comment 1

[IdenTrust]

We will post a complete incident report no later than February 16 , 2024

Comment 2

[IdenTrust]

Summary

On 2024-01-22, IdenTrust confirmed that 3 S/MIME certificates containing test data had been inadvertently published in the production environment instead of the test environment, bypassing the CA/B Forum S/MIME Baseline Requirements vetting processes, and also runs contrary to the ICANN convention of using "www.example.net" outside of documentation.

Although the certificates were automatically revoked by the test script in less than a minute of publication, their issuance still violated subsections of S/MIME Baseline Requirements section 3.2.2 regarding initial identity validation, Mozilla Root Store Policy section 2.2, and IdenTrust TrustID CP and CPS requirements. Additionally, the use of "www.example.net" violated RFC 2606.

Impact

Because the certificates lived for less than a minute before revocation, IdenTrust believes that the impact on the trustworthiness of certificates is minimal.

Timeline - all times are MST (UTC-7:00)

2024-01-23

• 13:58 MST Additional IdenTrust internal checks reported that 2 S/MIME certificates using "www.example.net" had been issued and then almost immediately revoked.

2024-01-23

• 14:40 MST Further investigation confirmed that 1 additional S/MIME certificate had been identified, for a total of 3 of this type that were created and revoked immediately. Since the number of certificates had increased, IdenTrust decided to make a comprehensive examination of its certificate database to provide assurance that only these certificates were involved.

2024-01-26

• 07:12 MST IdenTrust learned that the QA team had mistakenly published test scripts for automated processes in the production environment. Because the scripts had been intended for a testing environment, they were written in such a way that validation processes were bypassed so that the automated processes could be tested alone.
  -08:20 MST The comprehensive examination of the certificate database revealed that 1 TLS certificate had been similarly issued, making a total of 3 S/MIME certificates and 1 TLS certificate, or 4 certificates altogether. As with the S/MIME certificates, this was revoked within seconds after being published, using the test script that created it.
• 11:06 MST IdenTrust found that the QA team had been granted temporary access to the production environment certificate application API for one-time work that had already been completed, and that this was the source of the wrong-environment issue. QA access to the production environment was revoked, and reauthorization of access was blocked going forward using technical means. At no time did QA have access to legitimate certificate data in production, and it was never able to modify or delete legitimate certificate data.
• 16:19 MST IdenTrust disclosed the incident to CCADB via this bug report.

2024-02-11

• 12:00 IdenTrust completed its comprehensive analysis of the certificate database and found no evidence of other S/MIME or TLS certificates issued in this manner.

Root Cause Analysis

The offending certificates were the result of QA test automation scripts being mistakenly run in the production environment instead of the testing environment. The scripts were created such that the certificates were revoked within seconds of creation, but they should never have been run there in the first place.

A contributing cause was that some members of the QA team were new employees who may not have been fully aware of the need for strict separation of the two environments to assure certificate trustworthiness.

Lessons Learned

What went well

• The test scripts were designed to double-check that certificate had been successfully and then to delete them almost immediately. This resulted in almost no impact on the internet or on overall trustworthiness of certificates.

What didn't go well

• The realization that our QA team had unneeded access to the production environment and were insufficiently trained to not use it for testing.

Where we got lucky

• Even though we were not spefically looking for this error, we were able to catch in in time to prevent a more widespread issue.

Action Items

Action Item	Kind	Due Date
Remove QA access to production environment for any use	Prevent	Completed
Perform comprehensive examination of certificate database to determine if additional misissued certificates were present	Discover	Completed
Retrain QA team on permitted environments	Prevent	Completed

Appendix

Details of affected certificates

S/MIME
SERIALNUMBER=A01410C0000018AE79EF66C0000010C
SERIALNUMBER=A01410D0000018AE79F6878000000F1
SERIALNUMBER=A01410C0000018B6841DA200006083E

TLS
https://crt.sh/?id=10985927863

Comment 3

[IdenTrust]

we have no further remediation actions pendings for this issue.

Comment 4

[Ben Wilson]

I'll close this on or about Friday, 15-March-2024, unless there are additional items to discuss.