1859694 - SHECA: Issuance of test certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[chenxiaotong]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15

Steps to reproduce:

During regression testing after a new system went live, SHECA mis-issued test certificates. We have revoked all mis-issuance certificates and initiated an internal investigation.

Actual results:

During regression testing after a new system went live, SHECA mis-issued test certificates. We have revoked all mis-issuance certificates and initiated an internal investigation.

Expected results:

We will provide a full incident report by Friday, October 20, 2023, at the latest.

Comment 1

[jasmine.tang]

Incident Report

Summary

SHECA has recently been developing a new CA system. During the verification process after the new CA system went online, due to the incorrect use of some switches in the new CA system program, certificates containing informal information were issued.

Impact

12 certificates containing informal information were issued. Since our new CA system is still in the internal testing stage and has not been used publicly, there has been no other impact.

Timeline

All times are UTC. Our times are UTC+8.

2023-10-17:

• 10:10 SHECA auditors discovered problems when checking the correctness of issued certificates and notified the compliance department.
• 10:30 SHECA started certificate revocation.
• 10:31 SHECA began internal investigation procedures.
• 10:35 SHECA confirmed that it had issued a pre-certificate containing informal information due to procedural issues in the new CA system and immediately suspended the issuance of certificates by the new CA system.
• 11:10 SHECA confirmed that all incorrectly issued certificates had been revoked.
• 11:30 SHECA completed the second confirmation, and all the certificates involved were revoked.

2023-10-18:

• 03:17 SHECA confirmed the cause of the problem.
• 07:21 SHECA reviewed the incident and confirmed subsequent solutions.

Root Cause Analysis

• As part of our development process, we created a new CA system. After the system passed testing, it was deployed in the production environment for validation.

• To ensure functionality meets our expectations, we requested DV/OV/EV SSL certificates in the RA production environment. According to SHECA's CPS requirements, these certificate requests were to be processed by the new CA system after passing domain control validation and organization validation.

• During testing in our testing environment, we utilized automated scripts for comprehensive system testing. Our program utilizes switches to control whether the automated testing module is executed. When the switch is turned on, the automated testing module is triggered. This module also overwrites the requested data and inserts fixed values into the certificate (untrusted). After generating the certificate, the script verifies whether the information in the certificate matches the expected values, thus automatically validating the correctness of the program execution logic.

• After deploying the code for the system upgrade to the production environment, the developers failed to switch the toggle correctly, resulting in the new CA system program executing the automated testing module and overwriting the original correct certificate application information, ultimately issuing the incorrect certificates.

• When inspecting the certificates collectively after all the certificate requests were completed, we detected the issue. The auditor reported the incident, and we revoked the erroneous certificates according to the procedure.

Lessons Learned

What went well

• Due to the requirements of our internal regulations, internal personnel must carefully check the accuracy of the certificate information after each application for a certificate, so we immediately discovered the problem and revoked it.

What didn't go well

• We are deeply aware that during the process of major system upgrades, we should conduct more detailed inspections of every aspect of the system to control risks to the greatest extent.

Where we got lucky

• N/A

Action Items

Action Item	Kind	Due Date
In the subsequent testing process, we will completely delete this switch logic and strictly control the differences between production and test code through git branches.	Prevent	2023-10-27
Our testers are required to conduct a comprehensive verification of the test case data, and compliance personnel are required to conduct a secondary confirmation of the test case data. The data of the testing environment must also be standardized.	Prevent	2023-10-28
In response to two problems that have occurred during the development of our new CA system, we are deeply aware that measures must be taken to avoid these unexpected situations. We have formulated some plans, which are as follows:	Mitigate	2023-10-28

• 1.Retrain all testers and standardize the writing of test cases.
• 2.An approval process is added to the switch configuration and configuration information. Unapproved configurations cannot be put online.
• 3.Extend the testing cycle for all our functions, and then compliance personnel will participate in the entire testing process to control risks.
• 4.Adjust our code review rules so that the code must be reviewed by at least two engineers to pass, reducing the occurrence of code bugs.
• 5.Extend the system grayscale time and promptly follow up on the alarms of various important indicators to achieve timely discovery, timely processing, and timely reporting.

Appendix

Details of affected certificates

Firt Time And Last Time

• First issuance time https://crt.sh/?id=10810557325 2023-10-17 09:30:44 UTC

• Last issuance time https://crt.sh/?id=10810601379 2023-10-17 10:08:59 UTC

Certificate list

https://crt.sh/?id=10810557325
https://crt.sh/?id=10810557648

https://crt.sh/?id=10810567975
https://crt.sh/?id=10810568935

https://crt.sh/?id=10810577035
https://crt.sh/?id=10810579104

https://crt.sh/?id=10810584377
https://crt.sh/?id=10810587581

https://crt.sh/?id=10810589652
https://crt.sh/?id=10810589515

https://crt.sh/?id=10810598711
https://crt.sh/?id=10810601379

Comment 2

[jasmine.tang]

Hi Ben，we have completed all actions based on the action items.And have nothing else pending to correct on our side regarding this issue and considered it close.

Comment 3

[jasmine.tang]

Hi Ben, we have no further improvements to make regarding this case. I wonder if you still have any questions? If not, we would appreciate it if you could close this case.Thank you.

Comment 4

[Ben Wilson]

I will close this on Wed. 29-Nov-2023 unless there are further questions.