Open Bug 1946921 Opened 1 month ago Updated 12 days ago

SHECA: DV SSL certificate format is abnormal

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: wangjiatai, Assigned: wangjiatai)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

Incident Report

All timestamps are in Beijing Time (UTC+8):

https://crt.sh/?id=16556132396

https://crt.sh/?id=16546315389

https://crt.sh/?id=16535669659

2025-02-07 17:21 SHECA Compliance Department received an alert from the linting scan task (performed once every Friday). The alert showed that three certificates had format errors and all three certificates were issued by the intermediate root (Keymatic Secure Domain ECC CA G1). SHECA immediately stopped using the intermediate root (Keymatic Secure Domain ECC CA G1) to issue certificates and began to investigate the issue.

2025-02-07 23:44 SHECA received a third-party email notification that some certificates had format problems. After verification, they were consistent with the above certificates.

The above three certificates have all been revoked by the customers on their own initiative.

Summary

The above is the initial report. SHECA will provide a more detailed incident investigation report within three days!

Assignee: nobody → wangjiatai
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [dv-misissuance]

Incident Report

Summary

SHECA issued a new intermediate root certificate for a partner and configured a new certificate product. However, when issuing DV SSL certificates under the intermediate root Keymatic Secure Domain ECC CA G1, an operator error caused the DV SSL certificate product to mistakenly reference the OV SSL certificate template. Additionally, due to the absence of the product code in the whitelist, the pre-issuance linting process was not executed. As a result, the issued certificates contained the OV certificate OID in the certificatePolicies extension and included only CN and C attributes in the certificate's DN.

Impact

The following three certificates contained formatting errors. All have since been revoked upon the customer's request.

Timeline

All times are UTC+8.

2025-01-05:

  • 15:00: SHECA issued a new custom intermediate root certificate Keymatic Secure Domain ECC CA G1 for the partner.

2024-01-24:

  • 21:00: A new DV SSL certificate product was added. The CA system operator configured the certificate template for the intermediate root (Keymatic Secure Domain ECC CA G1) and refreshed the cache to activate the template.

2024-02-05:

  • 15:00: A subscriber requested ECC algorithm certificates, and SHECA issued three certificates.

2024-02-07:

  • 17:21: SHECA's compliance department received an alert indicating format issues with three DV certificates issued by Keymatic Secure Domain ECC CA G1. The alert originated from a scheduled linting task that runs every Friday to lint all unrevoked certificates. Detected issues trigger alert notifications.
  • 17:30: The compliance department informed the business team to stop issuing certificates under Keymatic Secure Domain ECC CA G1.
  • 23:44: SHECA received a third-party email notification about formatting issues with the certificates. Upon verification, the issues matched those in the previously identified certificates.

2024-02-10:

  • 10:02: The CA system operator identified that the DV SSL certificate product associated with Keymatic Secure Domain ECC CA G1 had been mistakenly configured to reference the OV SSL certificate template.
  • 10:10: The operator reviewed all certificate product configurations and confirmed no other products had similar issues.
  • 10:40: The operator reconfigured the DV SSL certificate product template for Keymatic Secure Domain ECC CA G1 and refreshed the cache.
  • 11:00: Investigated the reason why pre-issuance linting was not executed. SHECA mandates pre-issuance linting for all certificates to verify compliance with BR requirements.
  • 15:01: Identified that pre-issuance linting was skipped because the product code for the DV SSL certificate product associated with Keymatic Secure Domain ECC CA G1 was missing from the linting whitelist.

Root Cause Analysis

Cause 1:
The CA system operator mistakenly configured the DV SSL certificate product associated with Keymatic Secure Domain ECC CA G1 to reference the OV SSL certificate template.

Cause 2:
SHECA uses a whitelist to define which certificate products are subject to pre-issuance linting. After adding the new product, the engineers were not notified to add the product code to the whitelist, resulting in the failure of the pre-issuance linting check to catch this misconfiguration.

Lessons Learned

What went well

  • SHECA's scheduled linting task for unrevoked certificates promptly identified the issue.

What didn't go well

  • The certificate template configuration process lacked a secondary review mechanism.
  • The new product addition process was not standardized.

Where we got lucky

  • Not applicable.

Action Items

Action Item Kind Due Date
Standardize the process for adding new products by including the step to add product codes to the linting whitelist. Prevent 2025-02-21
Implement a secondary review mechanism for certificate template configurations. Templates configured by an operator must be reviewed and approved by an auditor before activation. Prevent 2025-02-30

Appendix

Wouldn't it be preferable to err on the side of caution and request linting by default, instead of "whitelisting" on every new product? If you are so sure about cases where you don't need linting (assuming this for non Publicly-Trusted Certificates), you could "exclude" those cases.

You also mentioned:

Implement a secondary review mechanism for certificate template configurations.

Can you please describe your current (primary) review mechanism for certificate template configurations?

Finally, when you refer to "intermediate root certificate", do you mean "Subordinate CA" as defined in the TLS Baseline Requirements? Combining "root" and "intermediate" seems a bit confusing. Is "Keymatic Secure Domain ECC CA G1" operated internally or externally by SHECA?

Thank you for your reminder. Below are my detailed responses to your questions:

Why isn't linting applied to all certificate products?

Thank you very much for your suggestion. I will discuss the feasibility of the solution you mentioned with my colleagues!

SHECA's Current Certificate Template Review Mechanism

CA system operators configure a set of templates for all subscriber SSL certificates, which include:

  • RSA DV SSL Subscriber Certificate Template
  • ECDSA DV SSL Subscriber Certificate Template
  • RSA OV SSL Subscriber Certificate Template
  • ECDSA OV SSL Subscriber Certificate Template
  • RSA EV SSL Subscriber Certificate Template
  • ECDSA EV SSL Subscriber Certificate Template

The compliance department reviews the relevant templates after each BR update to ensure they comply with the latest BR requirements. When adding new SSL certificate products, CA system operators must select an appropriate template from the above list. The error occurred because the operator mistakenly chose the ECDSA OV SSL Subscriber Certificate Template for a DV SSL certificate product, and there was no secondary review mechanism in place.

Definition of "Intermediate Root Certificate"

Thank you for your reminder!The "Intermediate Root Certificate" mentioned earlier actually refers to the "Subordinate CA" defined in the TLS Baseline Requirements. I will correct this statement in the future. Keymatic Secure Domain ECC CA G1 is a Subordinate CA operated internally by SHECA, primarily for use by partners.

Thank you for the quick response.

Regarding the Certificate Template Review Mechanism, I will notice that according to the Network and Certificate System Security Requirements and even version 1.7, have rules around change management.

Version 1.7 (section 1h)

Ensure that the CA’s security policies encompass a change management process, following the principles of documentation, approval and review, and to ensure that all changes to Certificate Systems, Issuing Systems, Certificate Management Systems, Security Support Systems, and Front-End / Internal-Support Systems follow said change management process;

From my understanding, it appears that there is a review process which happens when the BRs change, but there is no review process when a certificate profile is added or updated.

Sorry, I missed this part of the description. When adding and updating certificate profiles, the process is as follows

  • The compliance personnel provide the PDF file of the certificate template based on the latest BR, which is reviewed and approved by the "Security Certification Committee"
  • After the "Security Certification Committee" approves, the CA operator configures the relevant template according to the certificate template PDF file
  • The compliance personnel review the relevant configuration of the CA system according to the template
  • Configure the corresponding certificate product
  • Issue the certificate and verify whether pre-issuance linting can pass

(In reply to Dimitris Zacharopoulos from comment #4)

Thank you for the quick response.

Regarding the Certificate Template Review Mechanism, I will notice that according to the Network and Certificate System Security Requirements and even version 1.7, have rules around change management.

Version 1.7 (section 1h)

Ensure that the CA’s security policies encompass a change management process, following the principles of documentation, approval and review, and to ensure that all changes to Certificate Systems, Issuing Systems, Certificate Management Systems, Security Support Systems, and Front-End / Internal-Support Systems follow said change management process;

From my understanding, it appears that there is a review process which happens when the BRs change, but there is no review process when a certificate profile is added or updated.

Reply

Response Regarding Pre-Issuance Linting Issue

After discussing with the R&D engineers, we believe that enabling pre-issuance Linting by default for all certificate products is feasible. At the same time, by configuring directories, we exclude certain non-public trusted certificate products from requiring pre-issuance Linting, ensuring that the issue of products not undergoing pre-issuance Linting is avoided. I have adjusted the action plan accordingly.

Clarification on SHECA Internal Updates, New Certificate Templates, and New Certificate Product Processes

As mentioned earlier, SHECA has the following sets of templates for subscriber SSL certificates:

  • RSA DV SSL Subscriber Certificate Template
  • ECDSA DV SSL Subscriber Certificate Template
  • RSA OV SSL Subscriber Certificate Template
  • ECDSA OV SSL Subscriber Certificate Template
  • RSA EV SSL Subscriber Certificate Template
  • ECDSA EV SSL Subscriber Certificate Template

SHECA Update and New Certificate Template Process

  • Stay updated on BR-related updates by subscribing to and following forum polls, etc.;
  • Evaluate the impact of BR updates on certificate templates, and the compliance department will provide the latest version of the certificate template files, initiating a review by the "Security Certification Committee";
  • After approval by the "Security Certification Committee", CA operators will configure the relevant certificate product templates based on the certificate template files;
  • The compliance department will review the certificate template configuration to ensure correctness based on the certificate template files;
  • The certificate will be issued, and a pre-issuance check will be conducted to verify if it passes.

SHECA New Certificate Product Process

  • The business department initiates the new product process with the "Security Certification Committee";
  • After approval by the "Security Certification Committee", CA operators will add the new certificate product and associate it with the corresponding certificate product template;
  • The certificate will be issued, and pre-issuance linting will be checked to ensure it passes.

The error occurred during step 2 of the new certificate product process, where the CA operator mistakenly configured the DV SSL certificate product with the OV template. To prevent human error, SHECA has added a review step in step 2, requiring compliance review and approval before the product is successfully added.

Adjusted SHECA New Certificate Product Process

  • The business department submits a new product application to the "Security Certification Committee";
  • After approval by the "Security Certification Committee", CA operators will add the new certificate product and associate it with the corresponding certificate product template;
  • The compliance department will review the new product’s configuration for correctness;
  • The certificate will be issued, and pre-issuance linting will be checked to ensure it passes.

Action Plan Adjustments

Action Item Type Due Date
Enable pre-issuance linting by default for all certificate products, excluding non-public trusted certificate products that do not require it. Prevent 2025-02-27
A secondary review mechanism is implemented for adding new certificate products. After a new certificate product is added, it must be reviewed and confirmed by the compliance department for the second time before the product can take effect. Prevent 2025-02-27

If there is anything unclear about the above clarification, please feel free to let me know.

Action Items are being executed. If anyone has any other questions, please feel free to ask!

Action Plan Adjustments Update

Type Due Date State
Enable pre-issuance linting by default for all certificate products, excluding non-public trusted certificate products that do not require it. Prevent 2025-02-27 complete
A secondary review mechanism is implemented for adding new certificate products. After a new certificate product is added, it must be reviewed and confirmed by the compliance department for the second time before the product can take effect. Prevent 2025-02-27 complete

If there are no questions about this case, I will publish the Incident Closure Summary on March 17, 2025.

You need to log in before you can comment on or make changes to this bug.