Open Bug 2015383 Opened 1 month ago Updated 6 days ago

SHECA: CRL of root CA not published within 24 hours

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: wangjiatai, Assigned: wangjiatai)

Details

(Whiteboard: [ca-compliance] [crl-failure])

Preliminary Incident Report

Summary

  • Incident Description: SHECA revoked two cross‑signed certificates (UniTrust Global TLS RSA Root CA R1, UniTrust Global TLS ECC Root CA R2) on February 6. Due to system logic issues, this action inadvertently caused all subordinate CA certificates under UniTrust Global TLS RSA Root CA R1 and UniTrust Global TLS ECC Root CA R2 to be erroneously revoked. The incident remained undetected until February 8, resulting in a failure to issue and publish the CRL within 24 hours.
  • Relevant policies: BR 4.9.7 CRL issuance frequency
    CAs issuing CA Certificates:
  1. MUST update and publish a new CRL at least every twelve (12) months; 2. MUST update and publish a new CRL within twenty-four (24) hours after recording a Certificate as revoked.
  • Source of incident disclosure: Self Reported.
Assignee: nobody → wangjiatai
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [crl-failure]

SHECA has published the CRL.
SHECA will submit a full incident report.

A full incident report is being prepared and is expected to be released before February 23, 2026.

Full Incident Report

All times are UTC+8

Summary

  • CA Owner CCADB unique ID:
    A000261

  • Incident description:

    SHECA revoked two cross‑signed certificates (UniTrust Global TLS RSA Root CA R1, UniTrust Global TLS ECC Root CA R2) on February 6. Due to system logic issues, this action inadvertently caused all subordinate CA certificates under UniTrust Global TLS RSA Root CA R1 and UniTrust Global TLS ECC Root CA R2 to be erroneously revoked. The incident remained undetected until February 8, resulting in a failure to issue and publish the CRL within 24 hours.

  • Timeline summary:

    • Non-compliance start date:
      2026-02-06 15:07
    • Non-compliance identified date:
      2026-02-08 09:10
    • Non-compliance end date:
      2026-02-09 00:00

  • Relevant policies:

    • BR 4.9.7 CRL issuance frequency
      CAs issuing CA Certificates:
    1. MUST update and publish a new CRL at least every twelve (12) months; 2. MUST update and publish a new CRL within twenty-four (24) hours after recording a Certificate as revoked.

  • Source of incident disclosure:
    Self Reported

Impact

  • Total number of certificates:

    N/A

  • Total number of "remaining valid" certificates:

    N/A

  • Affected certificate types:

    N/A

  • Incident heuristic:

    Six subordinate CA certificates was erroneously revoked. CRLs are not released within the BR timeframe.

    SHECA DV TLS RSA CA 1A

    SHECA EV TLS RSA CA 1A

    SHECA OV TLS RSA CA 1A

    SHECA DV TLS ECC CA 2A

    SHECA EV TLS ECC CA 2A

    SHECA OV TLS ECC CA 2A

  • Was issuance stopped in response to this incident, and why or why not?:
    Yes. The affected Subordinate CAs have stopped issuance due to their revocation.

  • Analysis:
    N/A

  • Additional considerations:
    N/A

Timeline

2024-03-10, 12:00 The relevant functions were launched with the CA management system, which will cascade revoke all subordinate CA certificates when a CA certificate is revoked.

2026-02-01, 12:00 The EKU of two cross-certificates submitted by SHECA to the CCADB (UniTrust Global TLS RSA Root CA R1[Cross-certified with UCA Global G2 Root] and UniTrust Global TLS ECC Root CA R2 [Cross-certified with UCA Global G2 Root]) fails to comply with the latest requirements of Google's Root Inclusion Policy. SHECA decided to revoke these two cross-certificates.

2026-02-06, 15:07 Revoked the two cross-certificates.

2026-02-06, 18:00 Issued and published the CRL for the root certificate UCA Global G2 Root.

2026-02-08, 06:43 An alert was triggered for the ACME test website under the Subordinate CA (SHECA EV TLS ECC CA 2A - UniTrust Global TLS ECC Root CA R2), with certificates failing to renew normally.

2026-02-08, 09:10 Detected the alert and initiated an internal investigation into the issue.

2026-02-08, 11:58 Confirmed the root cause of the incident: the revocation of the two cross-certificates led to the unintended revocation of all six associated Subordinate CAs (SHECA DV TLS RSA CA 1A, SHECA EV TLS RSA CA 1A, SHECA OV TLS RSA CA 1A, SHECA DV TLS ECC CA 2A, SHECA EV TLS ECC CA 2A, SHECA OV TLS ECC CA 2A), resulting in the failure of subscriber certificate issuance. Furthermore, as these unintentionally revoked Subordinate CAs were not included in the current revocation plan, the CRLs for their associated root certificates (UniTrust Global TLS RSA Root CA R1 and UniTrust Global TLS ECC Root CA R2) were not republished within 24 hours.

2026-02-09, 15:14 Opened a case and released the preliminary incident report.

2026-02-09, 00:00 Issued and published the CRLs for the root certificates UniTrust Global TLS RSA Root CA R1 and UniTrust Global TLS ECC Root CA R2.

2026-02-22, 20:55 Released the full incident report.

Related Incidents

No related incidents were found.

Root Cause Analysis

Contributing Factor #1: System Bug in CA Certificate Revocation

  • Description:

    A bug exists in the CA operation and management system where, if a revoked CA certificate is associated with other CA entities, the system will automatically revoke all subordinate CA certificates of those entities in a cascading manner. The cross-certificates proactively revoked by SHECA were issued by UCA Global G2 Root and associated with two CA entities: UniTrust Global Code Signing RSA Root CA R1 and UniTrust Global Code Signing ECC Root CA R2. Each entity has three intermediate root certificates, all of which were revoked along with the cross-certificates. Due to the unintended nature of the revocation, SHECA failed to detect the issue and only issued the CRL for UCA Global G2 Root, without publishing the CRLs for UniTrust Global Code Signing RSA Root CA R1 and UniTrust Global Code Signing ECC Root CA R2.

  • Timeline:

    The bug has existed since the official launch of the current version of the CA management system in March 2024.

  • Detection:

    At 09:02 on February 8, 2026, TLS business personnel detected a certificate issuance error for the Subordinate CA (SHECA EV TLS ECC CA 2A). They contacted the CA system development and operation team to investigate the issue, and log analysis revealed the corresponding Subordinate CA had been revoked. Further investigation identified that the operation to revoke the cross-certificates on February 6 had triggered the issue, and the system bug was ultimately located at approximately 11:58.

  • Interaction with other factors:

    Three factors jointly caused the delay in CRL issuance. Factors 1 and 2 led to the unintended revocation incident, which indirectly resulted in the CRL issuance delay. Factor 1 directly caused the unintended revocation; however, adequate testing (Factor 2) would have detected the system bug in a timely manner, significantly reducing the probability of the unintended revocation.

  • Root Cause Analysis methodology used:
    5-Whys

Contributing Factor #2: Inadequate Testing Prior to CA Certificate Revocation

  • Description:

    For high-risk operations such as cross-certificate revocation, SHECA performed the operation directly in the production environment without any prior testing in the test environment, which led to this incident. SHECA has added relevant requirements to the Standard Operating Procedures (SOP) for CA operations: all CA certificate-related operations must be verified in the test environment before implementation in the production environment.

  • Timeline:

    This issue existed consistently.

  • Detection:

    During the investigation, SHECA discovered that there was no testing procedures to prevent unintended results caused by CA revocation.

  • Interaction with other factors:

    Three factors jointly caused the delay in CRL issuance. Factors 1 and 2 led to the unintended revocation incident, which indirectly resulted in the CRL issuance delay. Factor 1 directly caused the unintended revocation; however, adequate testing (Factor 2) would have detected the system bug in a timely manner, significantly reducing the probability of the unintended revocation.

  • Root Cause Analysis methodology used:
    5-Whys

Contributing Factor #3: Lack of Monitoring for Abnormal CA Certificate Revocation

  • Description:

    Among all current monitoring measures implemented by SHECA, there is no monitoring for the abnormal revocation of intermediate root certificates, which resulted in the failure to detect this issue in a timely manner. SHECA plans to add relevant monitoring functions.

  • Timeline:

    This issue existed consistently.

  • Detection:

    During the investigation, SHECA discovered that there was no such monitoring, which greatly increased the probability of such problems occurring.

  • Interaction with other factors:

    Three factors jointly caused the delay in CRL issuance. Factors 1 and 2 led to the unintended revocation incident, which indirectly resulted in the CRL issuance delay. When the controls for Factors 1 and 2 failed, Factor 3 could have served as the final line of defense: timely publication of CRLs after the unintended revocation to avoid non-compliance with the Baseline Requirements.

  • Root Cause Analysis methodology used:
    5-Whys

Lessons Learned

  • What went well:

    Following the detection of the unintended revocation of intermediate CA certificates, SHECA quickly identified the root cause of the incident and promptly issued the relevant CRLs.

  • What didn’t go well:
    N/A

  • Where we got lucky:

    Only a small number of demo site certificates were under the affected subordinate CA certificates, resulting in limited business impact.

  • Additional:
    N/A

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Fix the system bug prevent Factor #1 The bug is fixed. 2026-02-28 Ongoing
Establish a dedicated test environment: all subsequent CA certificate-related operations must be verified in the test environment before implementation in the production environment prevent Factor #2 The testing environment is in place. 2026-03-31 Ongoing
Establish a monitoring mechanism for CA certificate status prevent Factor #3 The function is launched. 2026-3-31 Ongoing

Appendix

N/A

Action Items Update

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Fix the system bug prevent Factor #1 The bug is fixed. 2026-02-28 Completed
Establish a dedicated test environment: all subsequent CA certificate-related operations must be verified in the test environment before implementation in the production environment prevent Factor #2 The testing environment is in place. 2026-03-31 Ongoing
Establish a monitoring mechanism for CA certificate status prevent Factor #3 The function is launched. 2026-3-31 Ongoing

The action items are ongoing.

You need to log in before you can comment on or make changes to this bug.