1838765 - SHECA: Outdated Organizational Units (OUs) problems in OV TLS certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[chenxiaotong]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15

Steps to reproduce:

SHECA became aware of two certificates issued with Outdated Organizational Units (OUs). These certificates have been revoked already.
https://crt.sh/?id=9632622798 revoked at:2023-06-15 09:40:00 UTC
https://crt.sh/?id=9607061049 revoked at:2023-06-15 09:40:36 UTC

Actual results:

we have identified a misissuance that need to be addressed in these certificates:
Outdated Organizational Units (OUs): According to the CA Browser Forum Baseline Requirements, the use of OUs has been prohibited since September 1, 2022.

Expected results:

These certificates have been revoked already.We are beginning a full investigation across our certificate population. We will provide a full report ASAP.

Comment 1

[Matthias]

Hi,

Looking at the certificates in crt.sh (and on my windows certificate inspector, and on several other certificate decoder services) it looks like the certificates contain garbled / misencoded data.

Assuming the issue is related to the OU inclusion, could you also include in your incident report an explanation why the problematic certificates contain garbled data [0] and unprintable control characters [1]?

[0] https://crt.sh/?id=9607061049&opt=zlint
[1] https://crt.sh/?id=9632622798&opt=zlint

Comment 2

[lee_yiu_chung]

I successfully reconstructed the subjects. Both cases are caused by injecting Chinese characters with incorrect encoding into issuing process, but what scares me is they both have different encoding errors.

[0] https://crt.sh/?id=9607061049&opt=zlint
The command I reconstructed the subject is
xxd -r -p | iconv -f utf8 -t iso88591 | iconv -f gb18030 -t utf8

O = 上市浦南东街社卫服中
OU = 上市浦南东街社卫服中

In this case, Chinese characters with GB18030 is fed directly into the program, but the program assumed the input was ISO-8859-1 and converted into UTF8String.

[1] https://crt.sh/?id=9632622798&opt=zlint
The command I reconstructed the subject is
xxd -r -p | iconv -f utf8 -t iso88591

O = 上海市大数据中心
OU = 卫健委和民族宗教局团队

In this case, Chinese characters are now correctly encoding into UTF-8, but the program still assumed the input was ISO-8859-1 and converted into UTF8String.

Comment 3

[lee_yiu_chung]

Correction:

[1] https://crt.sh/?id=9632622798&opt=zlint
The command I reconstructed the subject is
xxd -r -p | iconv -f utf8 -t iso88591

O = 上海市大数据中心
OU = 卫健委和民族宗教局团队

In this case, Chinese characters are now correctly encoding into UTF-8 before feeding into issuing program, but the program still assumed the input was ISO-8859-1 and converted into UTF8String.

Comment 4

[chenxiaotong]

(In reply to Matthias from comment #1)

Hi,

Looking at the certificates in crt.sh (and on my windows certificate inspector, and on several other certificate decoder services) it looks like the certificates contain garbled / misencoded data.

Assuming the issue is related to the OU inclusion, could you also include in your incident report an explanation why the problematic certificates contain garbled data [0] and unprintable control characters [1]?

[0] https://crt.sh/?id=9607061049&opt=zlint
[1] https://crt.sh/?id=9632622798&opt=zlint

Hi Matthias, SHECA had aware of the certificate issued with Non-compliant Subject Fields, The certificate has been revoked already.
Thanks for bringing that to our attention again.Please see the bug id=1839105，We will provide a full report ASAP.

Comment 5

[chenxiaotong]

Incident report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   On June 15, 2023, SHECA was made aware of the issue via a certificate-issue report email.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   20230615 05:13 CST (UTC+8) SHECA received an email about certificate problem report.
   20230615 09:30 CST (UTC+8) The investigation was initiated by the Information Security and Compliance Department.
   20230615 10:00 CST (UTC+8) The certificates issuance error is confirmed.
   20230615 10:30 CST (UTC+8) Review of historically issued certificates and outstanding requests. Initiated root cause analysis.
   20230615 17:40 CST (UTC+8) The certificate is revoked.
   20230615 18:00 CST (UTC+8) Root cause confirmed.
   20230620 15:00 CST (UTC+8) Historically issued certificates and outstanding requests have been reviewed, and no similar problematic certificates have been found.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

When SHECA became aware of the problem certificate, we launched an internal investigation immediately. At the same time, we stopped the issuance of related OV certificates.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g., OCSP failures, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help measure the severity of each problem.

Number of certificates: 2
First Issue Certificate: 2023-06-09 https://crt.sh/?id=9607061049
Second issue certificate: 2023-06-12 https://crt.sh/?id=9632622798

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list “https://crt.sh/?sha256=[sha256-hash]”, unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=9607061049
https://crt.sh/?id=9632622798

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In the manual review of the information which submitted by the certificate applicant, we review the organization information based on the submitted organization information, not the organization information contained in the CSR.
When the CSR contains garbled characters which submitted by the user, our certificate issuance system fail to correctly handle the subsequent certificate issuance logic process.
This error also caused the problem of Bug id=1839105.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

1.Before users submit a CSR, we will provide clear guidance and requirements to ensure they use the right way to fill the CSR information. This can reduce the occurrence of garbled characters.
2.We will upgrade our system to validate and filter error messages. It will ensure only CSRs which match the expected format and encoding pass validation and are submitted successfully.
Validation and filtering are performed in the system to ensure that only CSRs that meet the expected format can pass the validation and submit successfully. It is expected to be completed on 6.26.
3.We will improve the our system’s certificate-issuance-process to ensure the system can handle the corresponding process correctly and will not cause similar situations due to similar problems. It is expected to be completed on 6.30.

Comment 6

[Mathew Hodson]

(In reply to chenxiaotong from comment #5)

In the manual review of the information which submitted by the certificate applicant, we review the organization information based on the submitted organization information, not the organization information contained in the CSR.
When the CSR contains garbled characters which submitted by the user, our certificate issuance system fail to correctly handle the subsequent certificate issuance logic process.
This error also caused the problem of Bug 1839105.

Has SHECA done an investigation to determine if other certificates have similar issues? From your description it sounds like any fields in the CSR would have been included in the issued certificate without any verification that they comply with your CPS or with the BRs.

Comment 7

[chenxiaotong]

(In reply to Mathew Hodson from comment #6)

(In reply to chenxiaotong from comment #5)

In the manual review of the information which submitted by the certificate applicant, we review the organization information based on the submitted organization information, not the organization information contained in the CSR.
When the CSR contains garbled characters which submitted by the user, our certificate issuance system fail to correctly handle the subsequent certificate issuance logic process.
This error also caused the problem of Bug 1839105.

Has SHECA done an investigation to determine if other certificates have similar issues? From your description it sounds like any fields in the CSR would have been included in the issued certificate without any verification that they comply with your CPS or with the BRs.

20230620 15:00 CST (UTC+8), we conducted a thorough review of the previously issued certificates. Currently, we have identified only the two certificates reported earlier as problematic:
● Certificate ID: 9607061049 (Link: https://crt.sh/?id=9607061049)
● Certificate ID: 9632622798 (Link: https://crt.sh/?id=9632622798)

After our internal investigation, our certificate issuance system relies on the additional organization information provided by users, such as organization name, city, province, and country, to verify and validate the organization details. This verification process is not solely dependent on the organization name, city, province, and country data in the CSR. Our verification complies with the requirements of CPS and BR.The two certificates issued in error were due to garbled codes contained in the CSR, which resulted in the certificate issuance error.This issue would not have occurred if the CSR did not contain garbled codes.

Comment 8

[Ben Wilson]

Please provide an update on the status of your remediation steps related to this bug. As required by root store policies (https://www.ccadb.org/cas/incident-report), incident reports must be updated on a weekly basis "until you confirm that the resolution steps have been completed, unless a Root Store Operator has agreed to a different schedule by setting a 'Next Update' date in the 'Whiteboard' field of the bug or has announced they consider closing the bug and no further comments have been posted."

Comment 9

[jasmine.tang]

(In reply to Ben Wilson from comment #8)

Please provide an update on the status of your remediation steps related to this bug. As required by root store policies (https://www.ccadb.org/cas/incident-report), incident reports must be updated on a weekly basis "until you confirm that the resolution steps have been completed, unless a Root Store Operator has agreed to a different schedule by setting a 'Next Update' date in the 'Whiteboard' field of the bug or has announced they consider closing the bug and no further comments have been posted."

Hi Ben, I will reply to your previous question. We have conducted tests that only result in garbled information being sent to the ATTCH certificate from the CSR. However, this situation does not occur when there is garbled information in the middle of the CSR. We have conducted a comprehensive inspection of the previous certificates and have listed all the certificates that have caused this problem.

1、Provide a document for correctly using OpenSSL to generate CSR:
The incorrect CSR is caused by the customer's incorrect use of OpenSSL. Therefore, we will provide the user with the correct usage method and mark it on our certificate application page to prevent customers from generating similar invalid CSR again. Please refer to the attached document.

2、Upgrade the order system to verify CSR:
We will upgrade the system to verify and filter error messages. It will ensure that only CSR that matches the expected format and encoding through validation can be successfully submitted. The system will verify and filter to ensure that only CSR that meets the expected format can pass
verification and be successfully submitted. CSR that does not meet the requirements will prompt the user that they cannot apply. The prompt is shown in the attached screenshot.

3. Improve the certificate issuance system:
   We will handle exceptions and message email alerts for this type of CSR garbled code, and will not proceed with subsequent certificate issuance operations.

Remediation Status Date：
Provide documentation for correctly using openSsl to generate CSR, completed, June 25th, 2023
Adjust the order system's verification of CSR，completed, June 26, 2023
Improve certificate issuance system, completed, July 10, 2023

This is our latest status.Please let us know if there are any further questions or concerns about this bug.Thank you.

Comment 10

[jasmine.tang]

Attachment: Generate the private key and CSR using OpenSSL.pdf

Comment 11

[jasmine.tang]

Attachment: Screenshot of prompt.jpg

Comment 12

[Ben Wilson]

I plan to close this item on Wed. 11-Oct-2023, unless there are additional questions or concerns expressed.