1839105 - SHECA: Non-compliant Subject Fields problem in OV TLS certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[chenxiaotong]

Steps to reproduce:

SHECA became aware of one certificate issued with Non-compliant Subject Fields. The certificate has been revoked already.
https://crt.sh/?id=9632622798 revoked at:2023-06-15 09:40:00 UTC

Actual results:

We have identified one violation that need to be addressed in these certificates:
Non-compliant Subject Fields: The subject fields within the certificates must only contain printable control characters as per the requirements.

Expected results:

The certificate has been revoked already.We are beginning a full investigation across our certificate population. We will provide a full report ASAP.

Comment 1

[chenxiaotong]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   On June 15, 2023, SHECA was made aware of the issue via a certificate-issue report email.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

20230615 05:13 CST (UTC+8) SHECA received an email about certificate problem report.
20230615 09:30 CST (UTC+8) The investigation was initiated by the Information Security and Compliance Department.
20230615 10:00 CST (UTC+8) The certificates issuance error is confirmed.
20230615 10:30 CST (UTC+8) Review of historically issued certificates and outstanding requests. Initiated root cause analysis.
20230615 17:40 CST (UTC+8) The certificate is revoked.
20230615 18:00 CST (UTC+8) Root cause confirmed.
20230620 15:00 CST (UTC+8) Historically issued certificates and outstanding requests have been reviewed, and no similar problematic certificates have been found.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

When SHECA became aware of the problem certificate, we launched an internal investigation immediately. At the same time, we stopped the issuance of related OV certificates.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g., OCSP failures, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help measure the severity of each problem.

Number of certificates: 1
First Issue Certificate: 2023-06-12 https://crt.sh/?id=9632622798

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list “https://crt.sh/?sha256=[sha256-hash]”, unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=9632622798

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In the manual review of the information which submitted by the certificate applicant, we review the organization information based on the submitted organization information, not the organization information contained in the CSR.
When the CSR contains garbled characters which submitted by the user, our certificate issuance system fail to correctly handle the subsequent certificate issuance logic process.

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

1.Before users submit a CSR, we will provide clear guidance and requirements to ensure they use the right way to fill the CSR information. This can reduce the occurrence of garbled characters.
2.We will upgrade our system to validate and filter error messages. It will ensure only CSRs which match the expected format and encoding pass validation and are submitted successfully.
Validation and filtering are performed in the system to ensure that only CSRs that meet the expected format can pass the validation and submit successfully. It is expected to be completed on 6.26.
3.We will improve the our system’s certificate-issuance-process to ensure the system can handle the corresponding process correctly and will not cause similar situations due to similar problems. It is expected to be completed on 6.30.

Comment 2

[Mathew Hodson]

(In reply to chenxiaotong from comment #1)

In the manual review of the information which submitted by the certificate applicant, we review the organization information based on the submitted organization information, not the organization information contained in the CSR.
When the CSR contains garbled characters which submitted by the user, our certificate issuance system fail to correctly handle the subsequent certificate issuance logic process.

This is concerning. SHECA is responsible for validating the information that is actually included in a certificate issued. Based on your comments SHECA was alerted to the problem by an external party and wasn't able to detect this problem themselves.

What assurance can you provide that other certificates issued with data from a CSR actually matched the information that was validated? Your resolution steps talk about detecting encoding errors, but how do you detect if the information submitted by the applicant is different from what is in the CSR?

Have you checked if there are any other certificates where there is a mismatch between what was validated and what was issued? If you have done that investigation, provide details on how you determined whether or not a certificate was affected by this issue.

Comment 3

[zhangxiao]

非常感谢您的关注与回复

首先我们要说明一下：
我们当前的系统中是不以CSR中的数据（组织名称，城市，省份，国家等）进行审核验证并按照CSR中的数据（组织名称，城市，省份，国家等）进行签发的。
我们是审核验证用户额外提交的（组织名称，城市，省份，国家等）信息，并按照这个信息进行签发证书。
所以我们当前是不会去校验CSR中的数据与验证的信息是否匹配的，两者信息会存在不一样
理论上 CSR中数据（组织名称，城市，省份，国家等）是不会签发到证书中的，虽然CSR中的（组织名称，城市，省份，国家等）在大多数情况下与用户额外提交的（组织名称，城市，省份，国家等）信息一致
造成本次错误原因是CSR中包含乱码，在签发证书时候出现了错误，虽然之前的验证审核是按照用户提交的额外信息并且已经完成审核
我们在上述的7：List of steps your CA is taking to resolve the situation中的第三条会解决这个签发时候的错误问题；
再次感谢你的回复与关注，我们会在签发证书之前，以及签发证书之后，将生成证书的信息，证书中的信息，我们审核验证的信息（用户提交的额外信息）进行比对，保证签发证书的准确性，如果出现问题，我们也会在第一时间收到相应的警报

我们在时间线中的20230620 15：00 CST （UTC+8）对之前已经签发的证书进行了复查，
首先我们通过我们的订单系统列出了所有签发过的OV、EV 证书，包含用户提交的额外信息，证书信息
之后我们通过工具解析证书信息，获取其中的组织名称，城市，省份，国家等信息，与用户提交的额外信息进行比对
目前发现存在问题的证书有
https://crt.sh/?id=9607061049
https://crt.sh/?id=9632622798
根据错误原因我们提交了
bug id: 1839105

Comment 4

[zhangxiao]

(In reply to zhangxiao from comment #3)

非常感谢您的关注与回复

首先我们要说明一下：
我们当前的系统中是不以CSR中的数据（组织名称，城市，省份，国家等）进行审核验证并按照CSR中的数据（组织名称，城市，省份，国家等）进行签发的。
我们是审核验证用户额外提交的（组织名称，城市，省份，国家等）信息，并按照这个信息进行签发证书。
所以我们当前是不会去校验CSR中的数据与验证的信息是否匹配的，两者信息会存在不一样
理论上 CSR中数据（组织名称，城市，省份，国家等）是不会签发到证书中的，虽然CSR中的（组织名称，城市，省份，国家等）在大多数情况下与用户额外提交的（组织名称，城市，省份，国家等）信息一致
造成本次错误原因是CSR中包含乱码，在签发证书时候出现了错误，虽然之前的验证审核是按照用户提交的额外信息并且已经完成审核
我们在上述的7：List of steps your CA is taking to resolve the situation中的第三条会解决这个签发时候的错误问题；
再次感谢你的回复与关注，我们会在签发证书之前，以及签发证书之后，将生成证书的信息，证书中的信息，我们审核验证的信息（用户提交的额外信息）进行比对，保证签发证书的准确性，如果出现问题，我们也会在第一时间收到相应的警报

我们在时间线中的20230620 15：00 CST （UTC+8）对之前已经签发的证书进行了复查，
首先我们通过我们的订单系统列出了所有签发过的OV、EV 证书，包含用户提交的额外信息，证书信息
之后我们通过工具解析证书信息，获取其中的组织名称，城市，省份，国家等信息，与用户提交的额外信息进行比对
目前发现存在问题的证书有
https://crt.sh/?id=9607061049
https://crt.sh/?id=9632622798
根据错误原因我们提交了
bug id: 1839105

I apologize. We will provide a complete response in English later. You can ignore this message.

Comment 5

[chenxiaotong]

(In reply to Mathew Hodson from comment #2)

(In reply to chenxiaotong from comment #1)

In the manual review of the information which submitted by the certificate applicant, we review the organization information based on the submitted organization information, not the organization information contained in the CSR.
When the CSR contains garbled characters which submitted by the user, our certificate issuance system fail to correctly handle the subsequent certificate issuance logic process.

This is concerning. SHECA is responsible for validating the information that is actually included in a certificate issued. Based on your comments SHECA was alerted to the problem by an external party and wasn't able to detect this problem themselves.

What assurance can you provide that other certificates issued with data from a CSR actually matched the information that was validated? Your resolution steps talk about detecting encoding errors, but how do you detect if the information submitted by the applicant is different from what is in the CSR?

Have you checked if there are any other certificates where there is a mismatch between what was validated and what was issued? If you have done that investigation, provide details on how you determined whether or not a certificate was affected by this issue.

Thank you for your attention and response.
After our internal investigation, our certificate issuance system relies on the additional organization information provided by users, such as organization name, city, province, and country, to verify and validate the organization details. This verification process is not solely dependent on the organization name, city, province, and country data in the CSR. Although in most cases, the information in the CSR is consistent with the additional information provided by the users (organization name, city, province, country).
We have identified that the two certificates issued in error were due to garbled codes contained in the CSR, which resulted in the certificate issuance error.
While the verification process for the organization information, based on the user's additional information, had been successfully completed previously. This issue would not have occurred if the CSR did not contain garbled codes.
The third step outlined in the #comment1, queestion7: List of steps your CA is taking to resolve the situation" will address the error when issuing certificates.
In the future, prior to certificate issuance, we will compare the information used to generate the certificate with the information verified during our validation process (the user's additional information) to ensure the accuracy of the issued certificates. If any issues arise, we will promptly receive corresponding alerts.

Within the aforementioned timeline: 20230620 15:00 CST (UTC+8),We conducted a thorough review of the previously issued certificates.
Firstly, we utilized our order system to compile a list of all the issued OV certificates, including the additional information provided by users and the certificate details.
Subsequently, we parsed the certificate information using tools and compared the organization name, city, province, and country details with the user's submitted additional information.
Currently, we have identified only the two certificates reported earlier as problematic:
●Certificate ID: 9607061049 (Link: https://crt.sh/?id=9607061049)
●Certificate ID: 9632622798 (Link: https://crt.sh/?id=9632622798)
Based on the error reasons, we have subsequently submitted the following bug IDs:
●Bug ID: 1838765
●Bug ID: 1839105
Thank you for your response and attention again.

Comment 6

[Ben Wilson]

Please provide an update on the status of your remediation steps related to this bug. As required by root store policies (https://www.ccadb.org/cas/incident-report), incident reports must be updated on a weekly basis "until you confirm that the resolution steps have been completed, unless a Root Store Operator has agreed to a different schedule by setting a 'Next Update' date in the 'Whiteboard' field of the bug or has announced they consider closing the bug and no further comments have been posted."

Comment 7

[jasmine.tang]

(In reply to Ben Wilson from comment #6)

Please provide an update on the status of your remediation steps related to this bug. As required by root store policies (https://www.ccadb.org/cas/incident-report), incident reports must be updated on a weekly basis "until you confirm that the resolution steps have been completed, unless a Root Store Operator has agreed to a different schedule by setting a 'Next Update' date in the 'Whiteboard' field of the bug or has announced they consider closing the bug and no further comments have been posted."

6. Explain how and why errors or introduced errors occur, and how they have been avoided so far.

When manually reviewing the information submitted by certificate applicants, we review the organizational information based on the submitted organizational information, rather than the organizational information included in the CSR. When the CSR submitted by the user contains garbled code, our certificate issuance system cannot properly handle the subsequent certificate issuance logic process.

7. A list of steps taken by your CA to address the situation and ensure that such notifications will not be repeated in the future, along with a timeline for your CA's expected completion of these tasks.

7.1. Provide correct use of openSsl to generate CSR documents:
The CSR that caused the error was caused by the customer's incorrect use of OpenSSL. Therefore, we will provide the user with the correct usage method and mark it on our certificate application page to prevent the customer from generating similar invalid CSR again. Please refer to the attached document for documentation.

7.2 Upgrade the system to verify CSR:
We will upgrade the system to verify and filter error messages. It will ensure that only CSR that matches the expected format and encoding through validation can be successfully submitted. The system will verify and filter to ensure that only CSR that meets the expected format can pass verification and be successfully submitted. CSR that does not meet the requirements will prompt the user that they cannot apply. The prompt is shown in the attached screenshot.

7.3 Improving the certificate issuance system:
We will handle exceptions and message email alerts for this type of CSR garbled code situation, and will not continue with subsequent certificate issuance operations.

Remediation Status Date：
Provide correct use of openSsl to generate CSR documents, completed, June 25th, 2023
Adjust the order system's verification of CSR,completed, June 26, 2023
Improve certificate issuance system, completed, July 10, 2023

This is our latest status.Please let us know if there are any further questions or concerns about this bug.Thank you.

Comment 8

[jasmine.tang]

Attachment: Generate the private key and CSR using OpenSSL.pdf

Comment 9

[jasmine.tang]

Attachment: Screenshot of prompt.jpg

Comment 10

[Ben Wilson]

I plan to close this item on Wed. 11-Oct-2023, unless there are additional questions or concerns expressed.