1902947 - SHECA: The certificate's cpsURI is empty

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[SHECA]

Incident Report

This is a preliminary report.

Summary

In another SHECA case, https://bugzilla.mozilla.org/show_bug.cgi?id=1902592, we were informed that there was a problem with the cps address of SHECA's certificate. We are investigating this matter. SHECA will release a complete investigation report by 2024-06-20 09:00 (UTC+8)

Comment 1

[SHECA]

Sorry, the complete incident report is still being prepared and will be released on 2024-06-22 (UTC+8).

Comment 2

[SHECA]

Attachment: OK

Comment 3

[Wayne]

Before you repost that properly please double-check the stated timeline

Comment 4

[SHECA]

Incident Report

Summary

SHECA received a reminder in another case (1902592 - SHECA: EV certificate subject RDN order is incorrect) that at least one SHECA certificate had an empty cpsURI. We immediately began an investigation and discovered that due to a certificate template issue, six problematic certificates were issued.

The CP item format of the erroneous certificates is as follows. It can be seen that the policyIdentifier of the second PolicyInformation is the EV OID specified by BR, but the cpsURI is empty.

[1]Certificate Policy:
     Policy Identifier=1.2.156.112570.1.1.3
     [1,1]Policy Qualifier Info:
          Policy Qualifier Id=CPS
          Qualifier:
               https://www.sheca.com/repository
[2]Certificate Policy:
     Policy Identifier=2.23.140.1.1
     [2,1]Policy Qualifier Info:
          Policy Qualifier Id=CPS
          Qualifier

Impact

Certificates issued by SHECA contained an empty cpsURI in the CP item, resulting in an abnormal certificate format. After investigation, SHECA found a total of six certificates with the same issue. All six erroneous certificates have either been revoked or have expired. Since these certificates were for SHECA's internal use, they did not cause significant impact.

Timeline

All timestamps are in Beijing Time (UTC+8)

• 2023-06-14 21:00 SHECA learned from another case (crt.sh | 12421770586) that the cpsURI was empty.
• 2023-06-15 10:00 Investigation began.
• 2023-06-15 11:07 The compliance department sent an email to the certification department to stop issuing certificates.
• 2023-06-15 10:05 Using the latest version of zlint to check all unexpired certificates, no errors were found.
• 2023-08-15 13:00 Discovered that the reporter used pkilint to find the error, immediately wrote a script to use pkilint to scan all certificates locally.
• 2023-09-17 10:12 Completed the script and began scanning all unexpired certificates.
• 2024-06-17 13:50 The scan found a total of six problematic certificates, among which https://crt.sh/?id=12421770586 and https://crt.sh/?id=12421770586 had not been revoked.
• 2024-06-17 13:55 Immediately revoked the two certificates.
• 2024-06-17 14:10 Confirmed the issue was due to a certificate template configuration error. In SHECA's EV certificates, two PolicyInformation items are included: the first with a policyIdentifier declared in CPS, cpsURI being SHECA's cps address, and the second containing only the policyIdentifier value specified by the EV OID in BR. During the template modification on November 9, 2023, the second PolicyInformation's cpsURI attribute was checked but not filled in, and the program did not handle this situation, leading to an empty cpsURI value.
• 2024-06-17 14:15 Checked all certificate templates and adjusted any problematic ones.
• 2024-06-17 14:15 Confirming that only the EV certificate template has issues, we promptly adjusted the template and resumed issuing all other types of certificates except EV SSL certificates.
• 2024-06-18 21:15 Fix the issuance program to ensure that if the cpsURI attribute value in the CP item extension is empty, this item is not included in the certificate. Additionally, check for and fix similar issues in other extension items.
• 2024-06-18 15:00 Began drafting the incident report.
• 2024-06-19 09:00 Sent the draft report to relevant personnel for review.
• 2024-06-22 21:05 Released the complete incident report.

Root Cause Analysis

Background: In our CA system, all certificate extension items are configurable and can be adjusted through our CA system's interface.

• Why were certificates with an empty cpsURI issued?

SHECA's EV certificate template includes two PolicyInformation items: the first has a policyIdentifier declared in CPS, and the cpsURI is the CPS address; the second has a policyIdentifier specified by BR for the EV OID. On November 9, 2023, during a template adjustment, the cpsURI attribute for the second PolicyInformation was checked but not filled in. The program did not warn about this, resulting in an empty cpsURI value.

• Why was this issue not discovered earlier?

We use zlint as a detection tool and update it regularly. However, zlint does not currently support checking for this type of error. We have already raised an issue with zlint regarding this matter.

• Why did the certificate template adjustment error occur?

Due to the complexity of the certificate template interface design, mis-selections may occur during adjustments. The main issue is the lack of a secondary confirmation process after modifying or adding templates.

Lessons Learned

What went well

• SHECA responded quickly upon receiving the alert.

What didn't go well

• The certificate template modification and addition process lacked a secondary confirmation step, increasing the risk of errors.
• Based on the previous case, relying solely on one lint tool does not fully meet our certificate compliance checking needs. We will incorporate pkilint and certlint, and SHECA's extended lint tool mentioned in previous reports, into a single service to avoid issues due to lint tools not being timely updated or missing certain error checks.
• Over-reliance on lint tools. After each system or certificate template change, in addition to using the integrated lint tools mentioned above to check new certificates, we should add manual verification steps to confirm the attributes of issued certificates manually.

Where we got lucky

• The number of issued certificates was small, minimizing the impact on subscribers.

Action Items

Action Item	Type	Due Date
Check all certificate templates and adjust problematic ones	Prevent	2024-06-17
Fix the issuing system to ensure that certificates will not include extension items with empty values	Prevent	2024-06-24
Add a confirmation and approval process to the certificate template modification and addition workflow, ensuring that the person making the changes is not the same as the person approving them.	Detect	2024-06-28
Rebuild the lint tool to integrate zlint, pkilint, and certlint, and include the custom lint extensions from Shanghai CA, as mentioned in the previous case.	Detect	2024-06-28

Appendix

Details of Affected Certificates

https://crt.sh/?id=12421772198

https://crt.sh/?id=12573150826

https://crt.sh/?id=12421770452

https://crt.sh/?id=12421770586

https://crt.sh/?id=11917833535

https://crt.sh/?id=11044872118

Comment 5

[jasmine.tang]

Reply

Hello, Everyone.

SHECA has completed all the required rectifications in this case. If you have any further questions, please feel free to ask. Your suggestions will help SHECA enhance its compliance standards.

Action Items

Action Item	Type	Due Date	state
Check all certificate templates and adjust problematic ones	Prevent	2024-06-17	complete
Fix the issuing system to ensure that certificates will not include extension items with empty values	Prevent	2024-06-24	complete
Add a confirmation and approval process to the certificate template modification and addition workflow, ensuring that the person making the changes is not the same as the person approving them.	Detect	2024-06-28	complete
Rebuild the lint tool to integrate zlint, pkilint, and certlint, and include the custom lint extensions from Shanghai CA, as mentioned in the previous case.	Detect	2024-06-28	complete

Comment 6

[SHECA]

Reply

I think all the problems in this case have been solved. At the same time, SHECA has recently connected to the new lint project pkimetal https://github.com/pkimetal/pkimetal. Thank you very much for the hard work of the project developers. SHECA requests to close this case.

Comment 7

[Ben Wilson]

I'll look to close this case sometime next week (Aug. 19-23).