SHECA: Delayed revocation of TLS certificates affected by bug #1993357
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: wangjiatai, Assigned: wangjiatai)
Details
(Whiteboard: [ca-compliance] [leaf-revocation-delay])
Attachments
(4 files)
Preliminary Incident Report
-
CA Owner CCADB unique ID:
SHECA（A000261）
-
Incident description:
According to SHECA's description in case https://bugzilla.mozilla.org/show_bug.cgi?id=1993357, SHECA needed to revoke the certificates affected by this case before 2025-10-13 08:36. However, due to some reasons, the revocation of all certificates has not been completed. SHECA applied for an extension of the revocation, which is scheduled to be postponed to 2025-10-21 23:59:59 And SHECA will synchronize the revocation progress every day.
-
Timeline summary:
- Non-compliance start date: 2025-10-13 20:35
- Non-compliance identified date: 2025-10-13 20:35
- Non-compliance end date: 2025-10-21 23:59:59
-
Relevant policies:
BR 4.9.1.1
With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:
- The Certificate no longer complies with the requirements of Section 6.1.5 and Section 6.1.6 (CRLReason #4, superseded);
- The CA obtains evidence that the Certificate was misused (CRLReason #9, privilegeWithdrawn);
- The CA is made aware that a Subscriber has violated one or more of its material obligations under the Subscriber Agreement or Terms of Use (CRLReason #9, privilegeWithdrawn);
- The CA is made aware of any circumstance indicating that use of a Fully-Qualified Domain Name or IP address in the Certificate is no longer legally permitted (e.g. a court or arbitrator has revoked a Domain Name Registrant's right to use the Domain Name, a relevant licensing or services agreement between the Domain Name Registrant and the Applicant has terminated, or the Domain Name Registrant has failed to renew the Domain Name) (CRLReason #5, cessationOfOperation);
- The CA is made aware that a Wildcard Certificate has been used to authenticate a fraudulently misleading subordinate Fully-Qualified Domain Name (CRLReason #9, privilegeWithdrawn);
- The CA is made aware of a material change in the information contained in the Certificate (CRLReason #9, privilegeWithdrawn);
- The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA's Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);
- The CA determines or is made aware that any of the information appearing in the Certificate is inaccurate (CRLReason #9, privilegeWithdrawn);
- The CA's right to issue Certificates under these Requirements expires or is revoked or terminated, unless the CA has made arrangements to continue maintaining the CRL/OCSP Repository (CRLReason "unspecified (0)" which results in no reasonCode extension being provided in the CRL);
- Revocation is required by the CA's Certificate Policy and/or Certification Practice Statement for a reason that is not otherwise required to be specified by this section 4.9.1.1 (CRLReason "unspecified (0)" which results in no reasonCode extension being provided in the CRL); or
- The CA is made aware of a demonstrated or proven method that exposes the Subscriber's Private Key to compromise or if there is clear evidence that the specific method used to generate the Private Key was flawed (CRLReason #1, keyCompromise).
-
Source of incident disclosure:
Self Reported
|
||
Updated•10 months ago
|
Full Incident Report
All timestamps are Beijing time (UTC+8)
Summary
CA Owner CCADB unique ID:
SHECA（A000261）
Incident description:
According to SHECA's description, SHECA needed to revoke the certificates affected by this case before 2025-10-13 20:36. However, due to some reasons, the revocation of all certificates has not been completed. SHECA applied for an extension of the revocation, which is scheduled to be postponed to 2025-10-21 23:59:59 And SHECA will synchronize the revocation progress every day.
Timeline summary:
- Non-compliance start date: 2025-10-13 20:35
- Non-compliance identified date: 2025-10-13 20:35
- Non-compliance end date: 2025-10-21 23:59:59
Relevant policies:
BR 4.9.1.1
With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:
- The Certificate no longer complies with the requirements of Section 6.1.5 and Section 6.1.6 (CRLReason #4, superseded);
- The CA obtains evidence that the Certificate was misused (CRLReason #9, privilegeWithdrawn);
- The CA is made aware that a Subscriber has violated one or more of its material obligations under the Subscriber Agreement or Terms of Use (CRLReason #9, privilegeWithdrawn);
- The CA is made aware of any circumstance indicating that use of a Fully-Qualified Domain Name or IP address in the Certificate is no longer legally permitted (e.g. a court or arbitrator has revoked a Domain Name Registrant's right to use the Domain Name, a relevant licensing or services agreement between the Domain Name Registrant and the Applicant has terminated, or the Domain Name Registrant has failed to renew the Domain Name) (CRLReason #5, cessationOfOperation);
- The CA is made aware that a Wildcard Certificate has been used to authenticate a fraudulently misleading subordinate Fully-Qualified Domain Name (CRLReason #9, privilegeWithdrawn);
- The CA is made aware of a material change in the information contained in the Certificate (CRLReason #9, privilegeWithdrawn);
- The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA's Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);
- The CA determines or is made aware that any of the information appearing in the Certificate is inaccurate (CRLReason #9, privilegeWithdrawn);
- The CA's right to issue Certificates under these Requirements expires or is revoked or terminated, unless the CA has made arrangements to continue maintaining the CRL/OCSP Repository (CRLReason "unspecified (0)" which results in no reasonCode extension being provided in the CRL);
- Revocation is required by the CA's Certificate Policy and/or Certification Practice Statement for a reason that is not otherwise required to be specified by this section 4.9.1.1 (CRLReason "unspecified (0)" which results in no reasonCode extension being provided in the CRL); or
- The CA is made aware of a demonstrated or proven method that exposes the Subscriber's Private Key to compromise or if there is clear evidence that the specific method used to generate the Private Key was flawed (CRLReason #1, keyCompromise).
Source of incident disclosure:
Self Reported
Impact
Total number of certificates: 6920
Total number of "remaining valid" certificates: 1391
Affected certificate types: OV-TLS (OID:2.23.140.1.2.2) and DV-TLS (OID:2.23.140.1.2.1）
Incident heuristic:
Was issuance stopped in response to this incident?
SHECA has not stopped issuing certificates. SHECA will revoke all certificates issued before 2025-10-09 23:59:59 that violate BR regulations. All APIs involved in violations after 2025-10-09 23:59:59 have been taken offline. Subsequent newly applied certificates are subject to this case.
Analysis:
This revocation process strictly adheres to the requirements outlined in SHECA'sf MRIP&TP (Mass Revocation Incident Preparation and Testing Plan).
The following is SHECA's certificate revocation plan for the relevant certificates. This certificate revocation and reissue process uses a methodology similar to ACME ARI (Automated Certificate Management Environment Automatic Update Infrastructure). When developing SHECA's MRIP&TP, SHECA required all partners connecting to its API to implement a scheduled script to query all order information daily and initiate reissue tasks for orders marked "pending revocation" SHECA will notify partners in advance of these orders:
-
SHECA requires its partners to send certificate revocation notifications to all subscribers through various means.
-
SHECA will mark all orders associated with certificates pending revocation as "pending revocation" in its certificate management system.
-
SHECA and its partners will conduct a comprehensive scan of all orders and initiate reissue requests for orders marked "revoked."
-
Partners submit applications for issuance in batches; after the certificate is successfully issued, the subscriber will be notified of the replacement certificate within 24 hours.
Per the above plan, SHECA was initially able to complete the revocation of all certificates within 5 days. However, the process did not meet the expected timeline due to the following circumstances, necessitating an extension of the revocation period:
1.Network Freeze at China Mobile Cloud (a SHECA Partner)
As of October 12, 2025, China Mobile Cloud — a major telecommunications service provider in China that offers fundamental telecom services — has implemented a Network Freeze to support a large-scale event,The relevant notification email is included in the attachment。During this period, all internal services are prohibited from being updated. Since the script code for certificate re-issuance at China Mobile Cloud requires service updates to function, immediate execution is not feasible.
Additionally, mandatory certificate revocation could pose significant risks to critical communication infrastructure supported by China Mobile Cloud.
Definition of Network Freeze: A period where non-essential configuration changes, version upgrades, and hardware adjustments to network devices and systems are suspended. Only fault repairs are permitted, all to ensure network stability.
2.Some subscribers have not yet adopted automated certificate issuance solutions.
Currently, a considerable number of subscribers have not yet adopted automated certificate issuance solutions. For these subscribers, domain name verification and certificate renewal still rely on manual processes, which are time-consuming. Following this incident, SHECA will notify the relevant customers to switch to automated solutions as soon as possible and reiterate that the CA reserves the right to revoke certificates within a specified timeframe.
3.Holiday Reason
As both October 8 and October 12, 2025, fall on public holidays in China, there was some delay in the response efficiency of both SHECA and our customers. That said, SHECA is fully aware that this cannot be considered the core cause of this incident. We only wish to provide this context to the community for full transparency.
In conclusion, SHECA hereby applies to the community for an extension of the certificate revocation deadline, which is requested to be adjusted to 23:59:59 on October 21, 2025.
Additional considerations:
Not applicable.
Timeline
All timestamps are Beijing time (UTC+8)
2025-10-08 20:35: SHECA promised in the Google discussion group that it will complete the revocation of the affected certificates within the next five calendar days.
2025-10-09 09:00: SHECA in accordance with the relevant requirements of MRIP&TP, sent a certificate revocation notice to affected subscribers and simultaneously established an internal emergency response team to coordinate subsequent work.
2025-10-11 10:00: Certificate reissue for Xinwang's automated support customers has been completed, and relevant subscribers have been notified to complete the certificate replacement within 24 hours.
2025-10-14 03:00: Mobile Cloud completed the code update for the automatic reissue script, making technical preparations for subsequent certificate processing.
2025-10-14 12:00: China Mobile Cloud has initiated the re-issuance application process for certificates that require revocation.
2025-10-14 14:00: SHECA is actively coordinating with users to complete the issuance and replacement of China Mobile Cloud certificates.
Related Incidents
| Bug | Date | Description |
|---|---|---|
| 1885568 | 2024-03-05 | The incident revolved around Viking Cloud's failure to issue TLS certificates in compliance with standards, resulting in a large number of certificates not being revoked in a timely manner as required by the industry. It involved multiple dimensions such as certificate type, timeline, root cause, affected groups, corrective measures, and subsequent communication and follow-up. The core issue was to resolve the "certificate revocation delay" and the compliance and trust issues caused by it. |
| 1886788 | 2024-03-21 | The incident revolved around ACCV's failure to revoke relevant certificates in a timely manner in accordance with the "TLS Baseline Requirements" due to certificate issuance compliance issues. The core issue was to resolve the "revocation expiration due to delayed notification processing" issue. Ultimately, compliance repairs were completed through rectification and the incident was closed. |
| 1872738 | 2024-01-02 | The incident revolved around Buypass (a Norwegian certification authority) using an external DNS resolver for domain name verification, resulting in the non-compliant issuance of a large number of TLS certificates and failure to complete revocation within 24 hours as required by the "TLS Baseline Requirements". The core issue was a problem with AMCE's ARI, which led to the failure to revoke the certificates in a timely manner. |
Root Cause Analysis
Contributing Factor #1: Failure to promptly track the completion progress of partners' large-scale revocation of support capabilities.
-
Description:
Due to Network Freeze, the code update initiated by Mobile Cloud for re-signing could not be released in a timely manner, resulting in only some certificates being re-signed by subscribers, and a large number of replacement certificates still not being issued.
-
Timeline:
2025-08-09 SHECA released the first version of SHECA's MRIP&TP(Mass Revocation Incident Preparation and Testing Plan) in accordance with Mozilla's requirements. Partners connected through the API are required to implement an automated script for mass re-issuance of certificates (as described in the implementation solution above) in accordance with SHECA's requirements. This script is to be run at least once a day.
2025-10-09 SHECA notified China Mobile Cloud to run the script, but was informed that China Mobile Cloud had not yet scheduled the relevant feature implementation.
2025-10-10 China Mobile Cloud urgently implemented the relevant feature, but due to the Network Freeze issue described above, the code update could not be released.
-
Detection:
Not applicable.
-
Interaction with other factors:
Not applicable.
-
Root Cause Analysis methodology used
SHECA uses "5-Whys" to analyze the above problems:
**Why 1: Why couldn't the certificate be revoked promptly? **
Because the new replacement certificate had not yet been fully re-signed.
**Why 2: Why couldn't the replacement certificate be re-signed quickly? **
Because China Mobile Cloud had not implemented the automated script for mass re-signing as required by SHECA.
**Why 3: Why couldn't the script be released? **
Because China Mobile Cloud was in "Network Freeze" mode, restricting code updates and configuration changes, the developed script could not be released immediately.**Why 4: Why did China Mobile Cloud urgently develop the script on October 10th instead of completing it earlier? **
Because SHECA only notified China Mobile Cloud on October 9th to run the script, China Mobile Cloud discovered that the script function had not yet been scheduled for implementation. Therefore, it had to urgently initiate development to meet the request.
**Why 5: Why didn't China Mobile Cloud schedule the script function before SHECA's notification on October 9th? **
SHECA released SHECA's MRIP&TPon August 9th and clarified the relevant script development requirements to partners. However, SHECA did not follow up on Mobile Cloud's script development progress and failed to promptly identify the "unscheduled" issue with Mobile Cloud. SHECA will subsequently conduct SHECA's MRIP&TPesting on each API partner.
Contributing Factor #2: Some subscribers do not use automated solutions and are unable to quickly complete certificate application and replacement.
-
Description:
There are still some users who have not switched to the automated solution, and all operations rely on manual labor, such as manual DNS configuration and manual certificate replacement.
-
Timeline: This issue has existed but was not previously identified.
-
Detection:
Not applicable.
-
Interaction with other factors:
Not applicable.
-
Root Cause Analysis using "5-Whys" Methodology:
-
Why did SHECA fail to revoke the certificates for case https://bugzilla.mozilla.org/show_bug.cgi?id=1993357 within 5 days?
Some certificates have not been reissued.
-
Why couldn’t the certificates be revoked promptly?
Users need to re-verify domain name ownership.
-
Why couldn’t the replacement certificates be re-signed quickly?
Users needed extra time for domain validation and certificate replacement.
-
Why do users still need to manually configure validation values?
They haven’t implemented automated solutions.
-
Why haven’t many users adopted automated solutions?
Feedback from some users indicates the following reasons:
- ACME client scripts don’t support automation with certain small DNS providers, and users needing wildcard certificates cannot use file validation without modification.
- Some certificates are deployed on CDN or WAF services, which don’t support ACME clients.
-
Lessons Learned
-
What went well:
This revocation was carried out in accordance with SHECA's MRIP&TP, and the overall execution was relatively smooth.
-
What didn’t go well:
There are still some customers who do not use automation, and the certificate replacement speed is relatively slow.
-
Where we got lucky: Not applicable.
-
Additional: Not applicable.
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Optimize the holiday handling process in MRIP&TPto improve the collaboration efficiency and responsiveness of relevant teams during holidays, ensuring that emergency incidents during the holidays can be handled quickly and efficiently. | Mitigate | Added SHECA's contingency plan for large-scale revocation incidents during the holiday period in MRIP&TP. | 2025-11-15 | Ongoing | |
| SHECA will test each partner according to SHECA's MRIP&TP to determine whether the partner has the ability to quickly respond to large-scale revocations. | Prevent | Factor #1 | SHECA will complete the testing of the relevant support capabilities of each partner [MRIP&TP](https://assets-cdn.sheca.com/documents/Large-Scale Revocation Event Preparation and Test Plan v1.1.pdf). | 2025-11-15 | Ongoing |
| SHECA will send relevant notifications to all subscribers. Subsequently, SHECA will no longer provide revocation extension services to subscribers who use non-automated solutions. If a subscriber cannot accept automation and cannot meet the revocation requirements, SHECA will require them to apply for a "short-term certificate." | Prevent | Factor #2 | The relevant notifications have been sent and ensured that the customer has understood them. | 2025-11-15 | Ongoing |
Appendix
Appendix
Xinnet cert:（num:5566）
https://raw.githubusercontent.com/SHECA-Alvin/CASE/refs/heads/main/xinet-cert.md
China Mobile Cloud cert（num:1354）
https://github.com/SHECA-Alvin/CASE/blob/main/ecloud-cert.md
Daily Report
As of now:
xinnet certificate revocation progress: 5,315 certificates have been revoked, including 214 expired certificates, and 37 are pending revocation.
China Mobile Cloud certificate revocation has not yet begun.
Overall, I'm seeing nothing that suggests that SHECA has learnt anything from previous high-profile delayed revocation events from other CAs. I'm not going to surgically dissect and discuss every poor practice and failing of this report, however I would like to comment on a few particularly egregious elements I noticed:
Definition of Network Freeze: A period where non-essential configuration changes [...] are suspended. Only fault repairs are permitted, all to ensure network stability.
Replacing a revoked certificate would easily fall under "essential configuration change" and "fault repair", thus "we are in a network freeze" is in no way a valid explanation or excuse for delaying revocation -- not that it ever is, but even by SHECA's customer's own definitions, there is no excuse for it here.
In conclusion, SHECA hereby applies to the community for an extension of the certificate revocation deadline
This is not a thing. It has never been a thing.
SHECA will no longer provide revocation extension services to subscribers who use non-automated solutions
SHECA should never have been providing revocation extension services to any subscriber, regardless of their use (or not) of automation solutions.
Matt is correct.
This incident is bad - no understanding of any other delayed-revocation bug in last 2 years, things that do not exist (apply to community for extension?).
It is clear that the subscribers are not suitable for use of public TLS certificate.
This is a simple problem, no complex linter or ASN1 or reading of Baseline Requirements some way. This is clear, certs should be revoked and SHECA considered untrusted.
(In reply to mpalmer from comment #3)
Overall, I'm seeing nothing that suggests that SHECA has learnt anything from previous high-profile delayed revocation events from other CAs. I'm not going to surgically dissect and discuss every poor practice and failing of this report, however I would like to comment on a few particularly egregious elements I noticed:
Definition of Network Freeze: A period where non-essential configuration changes [...] are suspended. Only fault repairs are permitted, all to ensure network stability.
Replacing a revoked certificate would easily fall under "essential configuration change" and "fault repair", thus "we are in a network freeze" is in no way a valid explanation or excuse for delaying revocation -- not that it ever is, but even by SHECA's customer's own definitions, there is no excuse for it here.
In conclusion, SHECA hereby applies to the community for an extension of the certificate revocation deadline
This is not a thing. It has never been a thing.
SHECA will no longer provide revocation extension services to subscribers who use non-automated solutions
SHECA should never have been providing revocation extension services to any subscriber, regardless of their use (or not) of automation solutions.
Thank you for your attention to this matter!
First, SHECA apologizes for any inconvenience caused by previous imprecise expressions.
To clarify: SHECA has never intended to apply for an extension of certificate revocation, nor has it provided delayed revocation services to subscribers. The reason SHECA opened this case to report the incident is precisely because SHECA recognizes that delayed revocation is non-compliant. SHECA aims to fully disclose the situation, identify uncertainties in business execution (such as unexpected obstacles in customer-side cooperation) that may hinder compliance fulfillment, and eliminate such risks to avoid similar issues in the future. SHECA will refine its wording in future communications to ensure accuracy and avoid misunderstandings—thank you sincerely for your reminder and correction.
Regarding Action Item 3, the purpose of this action is to meet the revocation timeframes by driving the adoption of automated solutions, not to provide any delayed revocation services. For subscribers who have deployed automated certificate management solutions, SHECA will assist them in completing certificate replacement within the specified timeframe to ensure compliance; for those without automated solutions, sudden certificate revocation may lead to business interruptions and operational risks. Therefore, this part of subscribers are encouraged to transition to automated solutions as soon as possible to meet revocation requirements and avoid potential business impacts.
While SHECA's subscriber agreements clearly stipulate that subscribers must cooperate with CAs in revocation procedures, we encountered special and unavoidable practical challenges in this case—particularly with China Mobile Cloud. As a major telecommunications operator in mainland China, China Mobile Cloud is responsible for the operation and maintenance services of critical infrastructure, including public utilities and emergency systems.For these services, "certificate revocation" cannot be simply categorized as a "routine essential configuration change or fault repair" as understood in general scenarios. The core constraint lies in the dependency of downstream systems: these critical services involve hundreds of interconnected subsystems, and sudden certificate revocation (without prior compatibility testing and phased migration) would directly trigger service downtime. Such downtime could disrupt public services, affect emergency response capabilities, and even cause losses to social operation stability. Therefore, China Mobile Cloud needed additional time to formulate a phased replacement plan that prioritizes service stability. This is the fundamental reason for the temporary delay in revocation—not a refusal to perform revocation obligations, but a necessary measure to balance compliance requirements with the stability of critical social infrastructure.
Thank you again for your support.
You misunderstand again, the fault is mississued or compromised certificates and revocation is part of the remedy.
The revocation reason listed in the linked files is key compromise, I have no knowledge of Chinese legislation governing telecommunications, I know that European providers have strict obligations regarding security. As I see it delaying revocation of these compromised certificates puts the communications of the millions that rely on China Mobile Cloud’s services at risk.
This clarification aside I have few questions:
- How does SHECA monitor bugzilla and learn from bugs opened by other CAs?
- How did SHECA not learn that delayed revocation is not allowed? (How could the whole Entrust del rev saga be missed?)
- How did SHECA not learn that webPKI certificates are not appropriate to secure critical infrastructure? (Like this Chungwa Telecom bug)
I think it is worth bringing into the conversation SHECA's Incident from almost exactly 2 years ago. SHECA: Failure to revoke within 5 days
In which they committed to:
we will emphasize our right of active revocation
If they had done so, I would think it would prevent
we encountered special and unavoidable practical challenges in this case—particularly with China Mobile Cloud
Can I get more explanation around how SHECA's commitment from their last incident on this topic was enacted with China Mobile Cloud? How was it unavoidable if they were emphasizing their right to do so?
|
|
||
Comment 8•10 months ago
•
|
||
Please be aware that because this incident is tagged with a whiteboard label of “revocation-delay,” SHECA is required to report on certificates revoked/unrevoked/expired, and provide an estimated revocation completion date. See https://www.ccadb.org/cas/incident-report#when-are-reports-updated. Also, the “Analysis” section of the incident report must also include "the factors and rationales behind the decision to delay revocation (including detailed and substantiated explanations of how extensive harm would result to third parties–such as essential public services or widely relied-upon systems–and why the situation is exceptionally rare and unavoidable)."
The revocation delay attributed to China Mobile Cloud is most concerning. SHECA should have explicitly warned the subscriber against using publicly-trusted TLS server certificates on systems that could not tolerate timely revocation.
|
||
Comment 9•10 months ago
•
|
||
Comment #8 was filed/posted by me.
SHECA is expected to revoke all affected certificates without further delay. Failure to demonstrate timely and complete revocation, along with corrective actions to prevent recurrence, may affect its continued inclusion in the Mozilla root store.
Ben - Mozilla Root Program
|
Assignee | |
Comment 10•10 months ago
|
||
(In reply to Ben Wilson from comment #9)
Comment #8 was filed/posted by me.
SHECA is expected to revoke all affected certificates without further delay. Failure to demonstrate timely and complete revocation, along with corrective actions to prevent recurrence, may affect its continued inclusion in the Mozilla root store.
Ben - Mozilla Root Program
Hi Ben,
On October 16, SHECA assisted China Mobile Cloud in completing certificate replacement. Last night, at 11:00 PM October 16 (UTC+8), SHECA revoked 1,301 China Mobile Cloud certificates. This morning, at 10:00 AM October 17 (UTC+8), SHECA revoked the remaining 50 certificates. 3 certificates were expired. The remaining 37 certificates of Xinnet were revoked at 14:30 October 16 (UTC+8). Currently, SHECA is reviewing the logs to confirm the successful revocation of all affected certificates.
The revocation results will be reported under this case.
SHECA is currently finalizing the analysis for the delayed revocation, and the corrective actions to avoid recurrence. Detailed statement will be released later.
Thank you.
|
|
Assignee | |
Comment 11•10 months ago
|
||
SHECA has confirmed that all 1,391 certificates involved in this case have been revoked.
revoked: 1387
unrevoked: 0
expired: 4
|
|
Assignee | |
Comment 12•10 months ago
|
||
(In reply to JR Moir from comment #4)
This incident is bad - no understanding of any other delayed-revocation bug in last 2 years, things that do not exist (apply to community for extension?).
As SHECA stated in comment #5, this case serves as SHECA reporting to the community regarding the delayed revocation incident. SHECA apologizes for any inconvenience caused by previous imprecise expressions.
It is clear that the subscribers are not suitable for use of public TLS certificate.
For subscribers whose services involve critical infrastructure, SHECA will advise them to use private PKI or short-lived certificates. If they apply for publicly trusted certificates, they are required to adopt automated certificate management services and accept the revocation timelines that comply with the BR.
This is a simple problem, no complex linter or ASN1 or reading of Baseline Requirements some way. This is clear, certs should be revoked and SHECA considered untrusted.
As of now, all affected certificates have been revoked. Thank you.
|
|
Assignee | |
Comment 13•10 months ago
|
||
(In reply to Zacharias from comment #6)
You misunderstand again, the fault is mississued or compromised certificates and revocation is part of the remedy.
The revocation reason listed in the linked files is key compromise, I have no knowledge of Chinese legislation governing telecommunications, I know that European providers have strict obligations regarding security. As I see it delaying revocation of these compromised certificates puts the communications of the millions that rely on China Mobile Cloud’s services at risk.
This clarification aside I have few questions:
- How does SHECA monitor bugzilla and learn from bugs opened by other CAs?
- How did SHECA not learn that delayed revocation is not allowed? (How could the whole Entrust del rev saga be missed?)
- How did SHECA not learn that webPKI certificates are not appropriate to secure critical infrastructure? (Like this Chungwa Telecom bug)
- How does SHECA monitor bugzilla and learn from bugs opened by other CAs?
SHECA assigns dedicated personnel to monitor cases on both the CA/B Forum and Bugzilla, ensuring consistent tracking of discussions and non-compliance incidents across the ecosystem. Learnt from ballots and other CAs' bug cases, SHECA has driven technical and process improvements to enhance compliance in recent years. For example, SHECA has integrated open-source lint tools, developed its own lints, established a dedicated case response team, and implemented automation frameworks like ACME and ACME ARI. These measures work together to minimize the risk of non-compliance incidents in the certificate lifecycle management.
- How did SHECA not learn that delayed revocation is not allowed? (How could the whole Entrust del rev saga be missed?)
First, it is important to clarify that SHECA is fully aware the community has long championed optimization measures to eliminate delayed revocation—and SHECA has aligned with this direction by taking concrete steps over the past two years. These steps include actively guiding subscribers to adopt automated certificate management, developing a Mass Revocation Incident Preparation and Testing Plan (MRIP&TP), and adding explicit clauses to subscriber agreements that outline their obligations during mass revocation events.
In this specific case, SHECA has pre-communicated with partners about the emergency response plan prior to the revocation need arising. However, due to unforeseen circumstances: a partner's network freeze and the failure to prepare a code script for batch reissue of certificates as per the pre-agreed plan, the actual revocation execution was postponed to October 13, 2025,And because the partners' TLS certificates are used for critical infrastructure in the communications industry, forced revocation could lead to disruptions in essential communications services,The above reasons led to the delayed revocation. Throughout the delay, SHECA maintained active communication with the partner and provided support to resolve technical hurdles.
Thanks to the early deployment of the MRIP&TP and automation measures, SHECA completed revocation of all affected certificates by October 17, 2025—validating that these proactive preparations help mitigate delays. That said, SHECA acknowledges that subscriber-side risks remain a critical gap, and SHECA recognizes the need to implement more robust corrective measures to prevent such factors from disrupting revocation timelines moving forward. SHECA will publish detailed new action items to address this gap shortly.
- How did SHECA not learn that webPKI certificates are not appropriate to secure critical infrastructure? (Like this [Chungwa Telecom]
For subscribers with critical infrastructure use cases: SHECA will advise them to use private PKI or short-lived certificates. If they apply for publicly trusted certificates, they are required to adopt automated certificate management services and accept the revocation timelines that comply with the BR.
To further strengthen this alignment, SHECA is also developing a targeted communication plan: SHECA will issue formal announcements to all subscribers and partners, clarifying what qualifies as critical infrastructure and explaining why publicly trusted webPKI certificates are not recommended for such scenarios. Concurrently, SHECA will re-emphasize the non-negotiable requirements around revocation for webPKI certificates, ensuring all stakeholders understand their obligations.
|
||
Comment 14•9 months ago
|
||
It is very hard for the Mozilla community to take any of SHECA's statements at face value, when there are very straightforward contradictions like this (from Comment 5):
SHECA has never intended to apply for an extension of certificate revocation
as compared to this (from Comment 1):
SHECA hereby applies to the community for an extension of the certificate revocation deadline
It is entirely possible (and, indeed, likely) that these contradictions are caused by language barriers or cultural differences, and if that is the case, while I'm sympathetic, it doesn't change the core fact that SHECA's statements cannot be relied upon, and that is a very risky situation. One cannot help but wonder what other statements SHECA has made -- to the Mozilla community, to auditors, to subscribers, and other stakeholders -- that might not be correct, which haven't been questioned and clarified.
While it is beyond the strict scope of this incident, I call upon SHECA to improve their practices around communication with the Mozilla community to avoid these fairly obvious contradictions in their future communications -- whether that is hiring one or more people with suitable English communication skills, training existing staff to improve their English communication skills, or convincing everyone to adopt Esperanto as a common language.
As it stands at the moment, SHECA is not a trustworthy entity for no reason other than their statements cannot be trusted, let alone the trustworthiness of the systems that are being discussed and described.
|
|
Assignee | |
Comment 15•9 months ago
|
||
(In reply to mpalmer from comment #14)
We initially used an inaccurate term ("applying for an extension"), and when you questioned this, we corrected it in Comment 5 to reflect our true intent—this was a correction of the wording error rather than a contradictory claim.
It is very hard for the Mozilla community to take any of SHECA's statements at face value, when there are very straightforward contradictions like this (from Comment 5):
SHECA has never intended to apply for an extension of certificate revocation
as compared to this (from Comment 1):
SHECA hereby applies to the community for an extension of the certificate revocation deadline
We are investing more resources to address this. We do appreciate your understanding on "It is entirely possible (and, indeed, likely) that these contradictions are caused by language barriers or cultural differences". As it is indeed inevitable for non-native speakers, we would appreciate the opportunity to explain more until it is clear. Crucially, every statement we have made to the community and stakeholders is fact-based and verifiable (backed by logs, records, and documentation), as we uphold transparency and accuracy in substantive communications.
It is entirely possible (and, indeed, likely) that these contradictions are caused by language barriers or cultural differences, and if that is the case, while I'm sympathetic, it doesn't change the core fact that SHECA's statements cannot be relied upon, and that is a very risky situation. One cannot help but wonder what other statements SHECA has made -- to the Mozilla community, to auditors, to subscribers, and other stakeholders -- that might not be correct, which haven't been questioned and clarified.
While it is beyond the strict scope of this incident, I call upon SHECA to improve their practices around communication with the Mozilla community to avoid these fairly obvious contradictions in their future communications -- whether that is hiring one or more people with suitable English communication skills, training existing staff to improve their English communication skills, or convincing everyone to adopt Esperanto as a common language.
As it stands at the moment, SHECA is not a trustworthy entity for no reason other than their statements cannot be trusted, let alone the trustworthiness of the systems that are being discussed and described.
|
|
||
Comment 16•9 months ago
|
||
We initially used an inaccurate term ("applying for an extension"), and when you questioned this, we corrected it in Comment 5 to reflect our true intent—this was a correction of the wording error rather than a contradictory claim.
Which then raises the question: what other inaccurate terms has SHECA used in this report (or elsewhere) which have not been corrected because nobody has questioned them?
The Mozilla community needs to be able to rely on the communications from CAs being accurate, because if someone needs to individually question every statement by every CA, nothing useful will ever be done. Therefore, CAs must be held to account for "inaccuracies" when they are detected, rather than just being hand-wavingly dismissed as unimportant.
we would appreciate the opportunity to explain more until it is clear
I don't think that it is practical to allow a CA to endlessly "explain more", for several reasons. Most of the people engaging on incident reports are volunteering their valuable time and expertise, analysing and responding to the statements of CAs. If CAs are given unlimited licence to "explain more", it's inevitable that those volunteers will burn out and no longer contribute, leaving the CA to claim victory essentially by default.
every statement we have made to the community and stakeholders is fact-based and verifiable
That is an incredibly bold claim, given the number of things that I and others have already identified as being incorrect in SHECA's past statements.
|
|
Assignee | |
Comment 17•9 months ago
|
||
(In reply to Colin from comment #7)
I think it is worth bringing into the conversation SHECA's Incident from almost exactly 2 years ago. SHECA: Failure to revoke within 5 days
In which they committed to:
we will emphasize our right of active revocation
If they had done so, I would think it would prevent
we encountered special and unavoidable practical challenges in this case—particularly with China Mobile Cloud
Can I get more explanation around how SHECA's commitment from their last incident on this topic was enacted with China Mobile Cloud? How was it unavoidable if they were emphasizing their right to do so?
Since the mentioned incident, SHECA has established robust documentation: Clauses are added in the TLS CP/CPS, Mass Revocation Incident Preparation and Testing Plan (MRIP&TP), and subscriber agreements to clearly inform subscribers of their obligations of fulfilling the revocation timelines in certificate revocation events.
SHECA recognizes that paper-based documentation alone cannot eliminate execution gaps.
To bridge this gap and ensure consistent adherence to revocation timelines, SHECA is going to implement targeted, action-oriented measures: For subscribers with critical infrastructure use cases: SHECA will advise them to use private PKI or short-lived certificates. If they apply for publicly trusted certificates, they are required to adopt automated certificate management services and accept the revocation timelines that comply with the BR.
To further strengthen this alignment, SHECA is also developing a targeted communication plan: SHECA will issue formal announcements to all subscribers and partners, clarifying what qualifies as critical infrastructure and explaining why publicly trusted webPKI certificates are not recommended for such scenarios. Concurrently, SHECA will re-emphasize the non-negotiable requirements around revocation for webPKI certificates, ensuring all stakeholders understand their obligations.
These steps aim to translate the commitments outlined in our documentation into reliable execution, thereby preventing disruptions to revocation timelines. Updated action items will be published in the revised report.
|
|
||
Comment 18•9 months ago
|
||
If they apply for publicly trusted certificates, they are required to adopt automated certificate management services
How does SHECA intend to enforce such a requirement?
and accept the revocation timelines that comply with the BR.
Subscribers were already required to accept the revocation timelines that comply with the BRs. What has actually changed here?
|
|
Assignee | |
Comment 19•9 months ago
|
||
(In reply to incident-reporting from comment #8)
Please be aware that because this incident is tagged with a whiteboard label of “revocation-delay,” SHECA is required to report on certificates revoked/unrevoked/expired, and provide an estimated revocation completion date. See https://www.ccadb.org/cas/incident-report#when-are-reports-updated. Also, the “Analysis” section of the incident report must also include "the factors and rationales behind the decision to delay revocation (including detailed and substantiated explanations of how extensive harm would result to third parties–such as essential public services or widely relied-upon systems–and why the situation is exceptionally rare and unavoidable)."
The revocation delay attributed to China Mobile Cloud is most concerning. SHECA should have explicitly warned the subscriber against using publicly-trusted TLS server certificates on systems that could not tolerate timely revocation.
A more detailed and comprehensive report regarding this revocation delay will be issued by October 31, 2025.
|
||
Comment 20•9 months ago
|
||
A more detailed and comprehensive report regarding this revocation delay will be issued by October 31, 2025.
Even if the incident start date was today, which is isn't, then this commitment falls well beyond the SHOULD and also beyond the MUST deadlines of 3 days and 7 days respectively given in https://www.ccadb.org/cas/incident-report#when-are-reports-updated.
SHECA needs to report on certificates revoked/unrevoked/expired, and provide an estimated revocation completion date ASAP.
|
|
Assignee | |
Comment 21•9 months ago
|
||
(In reply to Malcolm D from comment #20)
A more detailed and comprehensive report regarding this revocation delay will be issued by October 31, 2025.
Even if the incident start date was today, which is isn't, then this commitment falls well beyond the SHOULD and also beyond the MUST deadlines of 3 days and 7 days respectively given in https://www.ccadb.org/cas/incident-report#when-are-reports-updated.
SHECA needs to report on certificates revoked/unrevoked/expired, and provide an estimated revocation completion date ASAP.
Hi , Malcolm
The “detailed and comprehensive report ” mentioned in the comments is a report that fully explains the incident at SHECA and the subsequent corrective measures.As SHECA has already reported in comment #11 that all certificates related to this case have been revoked, we would like to confirm whether further reporting on the certificate revocation status is required.
Best Regards!
|
||
Comment 22•9 months ago
|
||
While we may post additional questions, we have two observations we would like SHECA to address first:
-
We note several certificates are revoked, but lack a reasonCode of “keyCompromise” as required by the TLS BRs. Examples include certificates with the following hashes:
-
6940477a380c4406aeb3861123a4bff6fc9a98ab885cccaa1f35bf3263d46af8
-
5006a796d4226354bd578456b5323ac094893b1c60e9d495e009a299e5c41b9c
-
8dd67133c3cb3d1cc3f5660c596a20a320c174d6d0d38519713b7fd342c38d93
We can corroborate this via the attached CRLs, served at the time this issue was identified.
-
-
We note many instances of time-valid and unrevoked SHECA-issued certificates that contain the compromised keys. Examples include the following SPKIHashes:
- a5061f9f17fc81d633e1e1cff923909a0386d221890e3aa54bd5d726b4f3ace2 (revoked / not revoked)
- ca13734171aa915f195f37aa7d404ed545858b3b1d8c445b724481a636314328 (revoked / not revoked)
- 5079b7045a00a3a033b9c664f683603269514f72ee321ffc0a989fd2bd377927 (revoked / not revoked)
A complete list of our findings is located here.
Question: Can SHECA please help us understand these observations considering security best practices and expectations defined in the TLS BRs and SHECA’s own policies?
|
|
||
Comment 23•9 months ago
|
||
|
|
||
Comment 24•9 months ago
|
||
|
|
||
Comment 25•9 months ago
|
||
|
|
Assignee | |
Comment 26•9 months ago
|
||
(In reply to chrome-root-program from comment #22)
While we may post additional questions, we have two observations we would like SHECA to address first:
We note several certificates are revoked, but lack a reasonCode of “keyCompromise” as required by the TLS BRs. Examples include certificates with the following hashes:
6940477a380c4406aeb3861123a4bff6fc9a98ab885cccaa1f35bf3263d46af8
5006a796d4226354bd578456b5323ac094893b1c60e9d495e009a299e5c41b9c
8dd67133c3cb3d1cc3f5660c596a20a320c174d6d0d38519713b7fd342c38d93
We can corroborate this via the attached CRLs, served at the time this issue was identified.
We note many instances of time-valid and unrevoked SHECA-issued certificates that contain the compromised keys. Examples include the following SPKIHashes:
- a5061f9f17fc81d633e1e1cff923909a0386d221890e3aa54bd5d726b4f3ace2 (revoked / not revoked)
- ca13734171aa915f195f37aa7d404ed545858b3b1d8c445b724481a636314328 (revoked / not revoked)
- 5079b7045a00a3a033b9c664f683603269514f72ee321ffc0a989fd2bd377927 (revoked / not revoked)
A complete list of our findings is located here.
Question: Can SHECA please help us understand these observations considering security best practices and expectations defined in the TLS BRs and SHECA’s own policies?
Hi!
Sorry for not getting back to you sooner. We spent some time identifying the causes, and here are the responses to your two questions.
Cause of Problem 1: The revocation of the 7 certificates in question was not executed via SHECA's internal revocation script. Instead, the revocations were initiated by the customer through a partner's API, and the revocation reason selected by the customer was the default value: "unspecified (0)".
SHECA processes revocation requests originating from partners via an automated scheduled task. During the processing of this incident, this automated task was also used for these certificates, which ultimately resulted in the revocation reason for all 7 certificates being recorded as "unspecified (0)".
According to BR 7.2.2, in this situation, the CA should update the reason code to key compromise. SHECA is currently following internal procedures to change the CRL reason code for these certificates to "keyCompromise(1)".
Cause of Problem 2: SHECA's CA backend is configured with multiple CT Log addresses. SHECA's certificate issuance policy requires that at least 3 SCTs (Signed Certificate Timestamps) be obtained from CT Logs, with at least 1 of those SCTs coming from a Google CT Log, before a subscriber certificate will be issued.
During the actual issuance process, due to issues such as network timeouts, there were cases where the required 3 SCTs were not obtained. This resulted in the subscriber certificates failing to issue. However, the precertificates had already been submitted to 1 or 2 CT Logs. This ultimately led to the situation for the 55 certificates listed in the table, where only precertificates exist in CT Logs, with no corresponding issued subscriber certificates.SHECA provides an error log in the attachment "error.log".
SHECA's query scope in the mass revocation event was limited to the affected subscriber certificates; it did not include the pre-certificates data, resulting in these 55 entries of pre-certificates being missed.
SHECA has now instructed the R&D team to perform revocation operations for the 55 precertificates listed in the table. Referencing the document you provided, I have re-submitted the data with additional columns to map these missed precertificates to the serial numbers of the certificates that were actually revoked. Their public keys are identical.
data:https://docs.google.com/spreadsheets/d/1XGREc64-jvFeaAFUOcfgYUuuuwGtvTXxU_AaXvT31vA/edit?gid=0#gid=0
Remediation for Problem 2
When the CA program fails to obtain three pieces of SCT data (resulting in the final subscriber certificate not being issued), the pre-certificates that have been successfully reported to any CT Log must trigger an alert and be revoked.
|
|
Assignee | |
Comment 27•9 months ago
|
||
|
|
Assignee | |
Comment 28•9 months ago
|
||
SHECA has released a revised report to provide necessary supplementary explanations and address the key concerns raised in previous comments.
All timestamps are Beijing time (UTC+8)
Summary
CA Owner CCADB unique ID:
SHECA（A000261）
Incident description:
According to the Baseline Requirements (BR) 4.9.1.1, SHECA was required to complete the revocation of certificates involved in Bug #1993357 within 24 hours starting from 2025-10-08 20:35. However, due to some reasons, the revocation process was delayed. As of 2025-10-17 10:00, all subscriber certificates were revoked. As of 2025-10-24 20:15, the pre-certificates generated due to a CT reporting bug were revoked. The details are as follows:
Subscriber certificates: A total of 6920 certificates were revoked (including 5566 from Xinnet and 1354 from China Mobile Cloud), with 0 unrevoked.
Pre-certificates generated due to the CT reporting bug: A total of 55 were revoked, with 0 unrevoked.
Note: The public keys of the 55 pre-certificates were included in the scope of the revocation for Xinnet and China Mobile Cloud, but there were no final subscriber certificates issued for them.
Timeline summary:
- Non-compliance start date: 2025-10-09 20:35 (BR-required revocation deadline)
- Non-compliance identified date: 2025-10-13 20:35
- Non-compliance end date: 2025-10-24 20:15 (All subscriber certificates and pre-certificates (including those generated by the CT reporting bug) revocation completed)
**Relevant policies: **
BR 4.9.1.1（3）
The CA obtains evidence that the Subscriber's Private Key corresponding to the Public Key in the Certificate suffered a Key Compromise (CRLReason #1, keyCompromise);
Source of incident disclosure:
Self Reported
Impact
Total number of certificates: 6920+55
Total number of "remaining valid" certificates: 0
Affected certificate types: OV-TLS (OID:2.23.140.1.2.2) and DV-TLS (OID:2.23.140.1.2.1）and EV-TLS(OID:2.23.140.1.1)
Incident heuristic:
Not applicable.
Was issuance stopped in response to this incident?
The API that violated BR requirements has been taken offline. Newly issued certificates are not affected by this incident.
Analysis:
A partner (China Mobile Cloud) failed to deploy the automated replacement script as mandated by SHECA and was simultaneously in a "Network Freeze" period—during which system changes were prohibited. Given the significant societal impact of disrupting critical infrastructure services, SHECA did not enforce revocation within the timeframe specified in the BR. The root causes of this delayed revocation are detailed in Factor 1 and Factor 2.
Supplementary Explanation of China Mobile Cloud’s Delayed Revocation Circumstances:
As a major telecommunications service provider in mainland China, China Mobile Cloud relies on TLS certificates to secure communications at the application layer of its core network infrastructure. Mandatory revocation of the affected certificates would have directly disrupted its services: for instance, users might have been unable to top up accounts or purchase data plans, with some communication functions impaired. More severely, it could have triggered failures in communication component interactions, leading to large-scale service outages and subsequent disruptions to essential public services and critical sectors such as healthcare and transportation.
The "Network Freeze" mechanism is a common practice among mainland Chinese internet companies. It temporarily prohibits nearly all system updates during major events (e.g., significant sporting events, national conferences) to ensure service stability, minimizing the risk of unpredictable failures from code changes.
Due to SHECA’s failure to conduct advance drills with this partner to verify readiness for mass revocation, the revocation plan could not proceed as intended. While SHECA and China Mobile Cloud’s emergency response teams worked overtime to rapidly develop the automated script, the Network Freeze blocked code deployment. SHECA also attempted to assist China Mobile Cloud to seek approval from its leadership to lift the freeze, but the overly complex and cumbersome approval process prevented timely updates.
Additionally, SHECA had not previously identified the risk of revocation delays arising from the use of publicly trusted certificates in critical infrastructure—a gap that further contributed to the delay.
Additional considerations:
Not applicable.
Timeline
All timestamps are Beijing time (UTC+8)
2025-10-08 20:35 SHECA confirmed the need to revoke certificates involved in case #1993357.
2025-10-09 09:00 SHECA initiated the mass revocation plan, and the emergency response team began advancing the notification and processing procedures for revocation.
2025-10-11 10:00 Batch revocation was performed for Xinnet users with automated deployment, revoking 5315 certificates.
2025-10-14 03:00 China Mobile Cloud completed the code update for the automatic reissue script, making technical preparations for subsequent certificate processing.
2025-10-14 12:00 China Mobile Cloud has initiated the re-issuance application process for certificates that require revocation.
2025-10-14 14:23 Created Bugzilla report for the delayed revocation incident.
2025-10-16 14:30 Revoked the remaining Xinnet certificates; all Xinnet certificates revocation completed.
2025-10-16 23:00 Revoked 1301 certificates for China Mobile Cloud.
2025-10-17 10:00 Revoked the remaining China Mobile Cloud certificates; all subscriber certificates involved in this case revocation completed.
2025-10-24 13:46 According to a data report from Google, SHECA was alerted that some pre-certificates were not revoked.
2025-10-24 20:15 After investigation, SHECA found that 55 pre-certificates generated due to a CT reporting bug had not been revoked, and immediately revoked these pre-certificates. The public keys of these pre-certificates were included in the already revoked subscriber certificates.
Related Incidents
| Bug | Date | Description |
|---|---|---|
| 1885568 | 2024-03-05 | The incident revolved around Viking Cloud's failure to issue TLS certificates in compliance with standards, resulting in a large number of certificates not being revoked in a timely manner as required by the industry. It involved multiple dimensions such as certificate type, timeline, root cause, affected groups, corrective measures, and subsequent communication and follow-up. The core issue was to resolve the "certificate revocation delay" and the compliance and trust issues caused by it. |
| 1886788 | 2024-03-21 | The incident revolved around ACCV's failure to revoke relevant certificates in a timely manner in accordance with the "TLS Baseline Requirements" due to certificate issuance compliance issues. The core issue was to resolve the "revocation expiration due to delayed notification processing" issue. Ultimately, compliance repairs were completed through rectification and the incident was closed. |
| 1872738 | 2024-01-02 | The incident revolved around Buypass (a Norwegian certification authority) using an external DNS resolver for domain name verification, resulting in the non-compliant issuance of a large number of TLS certificates and failure to complete revocation within 24 hours as required by the "TLS Baseline Requirements". The core issue was a problem with AMCE's ARI, which led to the failure to revoke the certificates in a timely manner. |
Root Cause Analysis
Contributing Factor #1: Incomplete coverage of partner readiness in MRIP&TP verification and drills
-
Description:
-
In August 2025, SHECA released the "Mass Revocation Incident Preparation and Testing Plan" (MRIP&TP), mandating that API-connected partners deploy an "automatic replacement script" to ensure timely certificate revocation, with the script required to run at least once daily. However, the subsequent feasibility verification and drills for MRIP&TP, conducted in August 2025, failed to cover all partners.
This oversight directly led to delays: China Mobile Cloud only initiated urgent script development after receiving SHECA’s notification on October 9, 2025 (failing to complete it in advance as required by MRIP&TP). Further, due to a "Network Freeze" period (please refer to above "Analysis" Section for details), the developed script could not be deployed until October 16, 2025, exacerbating the revocation delay.
-
Timeline:
2025-08-09 SHECA released the first version of SHECA's MRIP&TP(Mass Revocation Incident Preparation and Testing Plan) in accordance with Mozilla's requirements. Partners connected through the API are required to implement an automated script for mass re-issuance of certificates (as described in the implementation solution above) in accordance with SHECA's requirements. This script is to be run at least once a day.
2025-10-09 SHECA notified China Mobile Cloud to run the script, but was informed that China Mobile Cloud had not yet scheduled the relevant feature implementation.
2025-10-10 China Mobile Cloud urgently implemented the relevant feature, but due to the Network Freeze issue described above, the code update could not be released.
-
Detection:
The contributing factor was identified when, during the course of this revocation incident, SHECA recognized that the feasibility verification for MRIP&TP had failed to assess partners' readiness to complete revocations within 24 hours. This gap was underscored by the revelation that SHECA had not conducted drills involving all partners after establishing the MRIP&TP—including a failure to verify the progress of script development across all partners—ultimately exposing delays in China Mobile Cloud’s script implementation and deployment.
This oversight persisted due to inadequate safeguards in the MRIP&TP verification process. Specifically, SHECA did not establish a requirement to conduct comprehensive drills with all partners after releasing the plan, nor did it implement systematic checks to verify partners’ progress in developing and deploying the mandatory automatic replacement scripts. Missed signals included the lack of proactive follow-up on whether partners (such as China Mobile Cloud) had met the requirements for script implementation. Without formalized mechanisms to track partner compliance or test end-to-end readiness, the gaps in partner preparedness remained undetected until the revocation incident happened.
-
Interaction with other factors:
Contributed to the extension of the revocation time along with Factor 2
-
Root Cause Analysis:
Contributing Factor #2: Lack of effective risk management for certificates used in critical infrastructure
-
Description:
SHECA did not realize that customers involved in critical infrastructure might have special circumstances preventing them from cooperating with certificate revocation, and instead applied a uniform onboarding process without differentiation.SHECA had not established clear guidance documents to define this part of the requirement for customers and employees to reference; nor did it differentiate and screen these use cases during the application review process to guide customers to suitable PKI solutions. The fundamental reason is that SHECA failed to conduct effective risk management for this situation, including accurately identifying risks, assessing impacts, and formulating corresponding mitigation measures (such as private PKI and short-term certificates).
-
Timeline:
Not applicable.
-
Detection:
The contributing factor was identified when a third party made comments regarding certificates used in critical infrastructure under this incident. SHECA acknowledged that it had not previously developed specific risk response measures for certificates used in critical infrastructure. confirming the gap.
-
Despite monitoring new cases on Bugzilla and discussions in forums, SHECA did not establish a structured process to translate these inputs into actionable risk assessments for critical infrastructure scenarios—allowing this gap to persist. Missed signals included related incidents that highlighted unique revocation challenges for critical infrastructure customers. Additionally, the absence of dedicated guidance documents meant there were no formal mechanisms to flag or mitigate these risks, allowing the gap to remain unaddressed.
-
Interaction with other factors
Contributed to the extension of the revocation time along with Factor 1
-
Root Cause Analysis:
Lessons Learned
-
What went well:
This revocation was carried out in accordance with SHECA's MRIP&TP, and the overall execution was relatively smooth.
-
What didn’t go well:
There are still some customers who do not use automation, and the certificate replacement speed is relatively slow.
-
**Where we got lucky:**Not applicable.
-
**Additional:**Not applicable.
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. | Prevent | Factor #1 | Each partner has been notified of the relevant requirements. | 2025-11-10 | Ongoing |
| Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. | Prevent | Factor #1 | The emergency plan document for each partner has been inspected and confirmed as complete. | 2025-11-31 | Ongoing |
| Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. | Prevent | Factor #1 | At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. | 2025-11-31 | Ongoing |
| Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. | Prevent | Factor #2 | The Guidance document has been written and finalized. | 2025-11-10 | Ongoing |
| Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. | Prevent | Factor #2 | The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. | 2025-11-31 | Ongoing |
| Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. | Prevent | Factor #2 | The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. | 2025-11-20 | Ongoing |
| Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. | Prevent | Factor #2 | / | 2025-11-31 | Ongoing |
| Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. | Mitigate | / | The bug has been successfully fixed and verified. | 2025-11-10 | Ongoing |
Appendix
Xinnet cert:（num:5566）
https://raw.githubusercontent.com/SHECA-Alvin/CASE/refs/heads/main/xinet-cert.md
China Mobile Cloud cert（num:1354）
https://github.com/SHECA-Alvin/CASE/blob/main/ecloud-cert.md
Pr-ecertificates generated due to the CT reporting BUG（num:55). The public key of the pre-certificate and their corresponding revoked subscriber certificate is identical.
https://docs.google.com/spreadsheets/d/1XGREc64-jvFeaAFUOcfgYUuuuwGtvTXxU_AaXvT31vA/edit?gid=0#gid=0
|
|
Assignee | |
Comment 29•9 months ago
|
||
Action Items Update
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. | Prevent | Factor #1 | Each partner has been notified of the relevant requirements. | 2025-11-10 | Completed |
| Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. | Prevent | Factor #1 | The emergency plan document for each partner has been inspected and confirmed as complete. | 2025-11-31 | Ongoing |
| Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. | Prevent | Factor #1 | At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. | 2025-11-31 | Ongoing |
| Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. | Prevent | Factor #2 | The Guidance document has been written and finalized. | 2025-11-10 | Completed |
| Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. | Prevent | Factor #2 | The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. | 2025-11-31 | Ongoing |
| Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. | Prevent | Factor #2 | The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. | 2025-11-20 | Ongoing |
| Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. | Prevent | Factor #2 | / | 2025-11-31 | Ongoing |
| Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. | Mitigate | / | The bug has been successfully fixed and verified. | 2025-11-10 | Delayed |
|
|
Assignee | |
Comment 30•9 months ago
|
||
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. | Prevent | Factor #1 | Each partner has been notified of the relevant requirements. | 2025-11-10 | Completed |
| Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. | Prevent | Factor #1 | The emergency plan document for each partner has been inspected and confirmed as complete. | 2025-11-31 | Ongoing |
| Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. | Prevent | Factor #1 | At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. | 2025-11-31 | Ongoing |
| Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. | Prevent | Factor #2 | The Guidance document has been written and finalized. | 2025-11-10 | Completed |
| Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. | Prevent | Factor #2 | The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. | 2025-11-31 | Ongoing |
| Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. | Prevent | Factor #2 | The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. | 2025-11-20 | Ongoing |
| Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. | Prevent | Factor #2 | / | 2025-11-31 | Ongoing |
| Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. | Mitigate | / | The bug has been successfully fixed and verified. | 2025-11-10 | Completed |
|
|
Assignee | |
Comment 31•8 months ago
|
||
Action Items Update
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. | Prevent | Factor #1 | Each partner has been notified of the relevant requirements. | 2025-11-10 | Completed |
| Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. | Prevent | Factor #1 | The emergency plan document for each partner has been inspected and confirmed as complete. | 2025-11-31 | Ongoing |
| Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. | Prevent | Factor #1 | At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. | 2025-11-31 | Ongoing |
| Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. | Prevent | Factor #2 | The Guidance document has been written and finalized. | 2025-11-10 | Completed |
| Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. | Prevent | Factor #2 | The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. | 2025-11-31 | Ongoing |
| Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. | Prevent | Factor #2 | The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. | 2025-11-20 | Ongoing |
| Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. | Prevent | Factor #2 | / | 2025-11-31 | Ongoing |
| Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. | Mitigate | / | The bug has been successfully fixed and verified. | 2025-11-10 | Completed |
| Establish an emergency communication mechanism for suspending TLS certificate issuance and revocation services during mass revocation events: Develop a dedicated notification template for partners, specifying the notification timeframe (e.g., within 1 hour of trigger), designated communication channels (e.g., email + WeChat Work), and core information to be communicated (suspension period, recovery time, inquiry method). | Prevent | / | The notification template has been formulated, and the MRIP&TP has been updated. | 2025-11-30 | Ongoing |
| Launch a function of separately issuing TLS-specific CRLs: Support manual on-demand immediate triggering of updates without relying on preset issuance frequency; control the CRL issuance time within 2 hours based on the existing certificate inventory. | Prevent | / | The function has been launched. | 2025-11-30 | Ongoing |
Background regarding 2 new action items:
To ensure the compliance and timeliness of the mass revocation process, SHECA has added the two action items above.
Background 1:
SHECA identified that after a mass revocation event (MRE) is triggered, continuing to provide issuance services will lead to errors in revocation quantity statistics due to dynamic data), and ontinuing to provide revocation services (including those requested by customers) may result in incorrect reasonCode for certificates within the scope of the MRE. Therefore, SHECA decides to immediately suspend all TLS certificate issuance and revocation services upon the triggering of an MRE. After confirming the scope of affected certificates, batch revocation processing will be conducted to ensure the completeness and accuracy of the revocation.And formulate a timely and effective customer notification plan.
Background 2:
Currently, SHECA’s CRL covers all types of subscriber certificates, resulting in long processing time (approximately 7 hours) for TLS certificate CRL issuance with the full list. To improve the efficiency of CRL updates in mass revocation scenarios, SHECA decides to issue TLS-specific CRLs separately.
|
|
Assignee | |
Comment 32•8 months ago
|
||
Action Items Update
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. | Prevent | Factor #1 | Each partner has been notified of the relevant requirements. | 2025-11-10 | Completed |
| Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. | Prevent | Factor #1 | The emergency plan document for each partner has been inspected and confirmed as complete. | 2025-11-31 | Ongoing |
| Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. | Prevent | Factor #1 | At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. | 2025-11-31 | Ongoing |
| Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. | Prevent | Factor #2 | The Guidance document has been written and finalized. | 2025-11-10 | Completed |
| Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. | Prevent | Factor #2 | The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. | 2025-11-31 | Ongoing |
| Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. | Prevent | Factor #2 | The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. | 2025-11-20 | Completed |
| Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. | Prevent | Factor #2 | / | 2025-11-31 | Ongoing |
| Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. | Mitigate | / | The bug has been successfully fixed and verified. | 2025-11-10 | Completed |
| Establish an emergency communication mechanism for suspending TLS certificate issuance and revocation services during mass revocation events: Develop a dedicated notification template for partners, specifying the notification timeframe (e.g., within 1 hour of trigger), designated communication channels (e.g., email + WeChat Work), and core information to be communicated (suspension period, recovery time, inquiry method). | Prevent | / | The notification template has been formulated, and the MRIP&TP has been updated. | 2025-11-30 | Ongoing |
| Launch a function of separately issuing TLS-specific CRLs: Support manual on-demand immediate triggering of updates without relying on preset issuance frequency; control the CRL issuance time within 2 hours based on the existing certificate inventory. | Prevent | / | The function has been launched. | 2025-11-30 | Ongoing |
|
|
Assignee | |
Comment 33•8 months ago
|
||
Action Items Update
Action Items
Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. Prevent Factor #1 Each partner has been notified of the relevant requirements. 2025-11-10 Completed Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. Prevent Factor #1 The emergency plan document for each partner has been inspected and confirmed as complete. 2025-11-30 Ongoing Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. Prevent Factor #1 At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. 2025-11-30 Ongoing Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. Prevent Factor #2 The Guidance document has been written and finalized. 2025-11-10 Completed Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. Prevent Factor #2 The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. 2025-11-30 Ongoing Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. Prevent Factor #2 The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. 2025-11-20 Completed Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. Prevent Factor #2 / 2025-11-30 Ongoing Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. Mitigate / The bug has been successfully fixed and verified. 2025-11-10 Completed Establish an emergency communication mechanism for suspending TLS certificate issuance and revocation services during mass revocation events: Develop a dedicated notification template for partners, specifying the notification timeframe (e.g., within 1 hour of trigger), designated communication channels (e.g., email + WeChat Work), and core information to be communicated (suspension period, recovery time, inquiry method). Prevent / The notification template has been formulated, and the MRIP&TP has been updated. 2025-11-30 Completed Launch a function of separately issuing TLS-specific CRLs: Support manual on-demand immediate triggering of updates without relying on preset issuance frequency; control the CRL issuance time within 2 hours based on the existing certificate inventory. Prevent / The function has been launched. 2025-11-30 Ongoing
|
|
Assignee | |
Comment 34•8 months ago
|
||
Action Items Update
Action Items
Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. Prevent Factor #1 Each partner has been notified of the relevant requirements. 2025-11-10 Completed Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. Prevent Factor #1 The emergency plan document for each partner has been inspected and confirmed as complete. 2025-11-30 Delayed Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. Prevent Factor #1 At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. 2025-11-30 Delayed Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. Prevent Factor #2 The Guidance document has been written and finalized. 2025-11-10 Completed Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. Prevent Factor #2 The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. 2025-11-30 Completed Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. Prevent Factor #2 The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. 2025-11-20 Completed Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. Prevent Factor #2 / 2025-11-30 Completed Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. Mitigate / The bug has been successfully fixed and verified. 2025-11-10 Completed Establish an emergency communication mechanism for suspending TLS certificate issuance and revocation services during mass revocation events: Develop a dedicated notification template for partners, specifying the notification timeframe (e.g., within 1 hour of trigger), designated communication channels (e.g., email + WeChat Work), and core information to be communicated (suspension period, recovery time, inquiry method). Prevent / The notification template has been formulated, and the MRIP&TP has been updated. 2025-11-30 Completed Launch a function of separately issuing TLS-specific CRLs: Support manual on-demand immediate triggering of updates without relying on preset issuance frequency; control the CRL issuance time within 2 hours based on the existing certificate inventory. Prevent / The function has been launched. 2025-11-30 Completed
|
||
Comment 35•8 months ago
|
||
(In reply to SHECA from comment #34)
Action Items Update
Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. Prevent Factor #1 The emergency plan document for each partner has been inspected and confirmed as complete. 2025-11-30 Delayed Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. Prevent Factor #1 At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. 2025-11-30 Delayed
Is there an explanation for these action items now being delayed?
|
|
Assignee | |
Comment 36•8 months ago
|
||
(In reply to Wayne from comment #35)
Is there an explanation for these action items now being delayed?
Hi Wayne,
SHECA has implemented these two action items but has not yet met the evaluation criteria.
By reviewing partners' emergency plans and conducting joint MRIP&TP drills, SHECA has identified that the document completeness, response processes, and technical capabilities of some partners fail to meet the latest requirements of SHECA MRIP&TP. The issues are as follows:
- Partners need to improve the efficiency and completeness of certificate revocation notifications to ensure that all affected subscribers receive the revocation notices within 24 hours.
- The automated revocation scripts of partners lack a robust exception alert mechanism —— Failures in revocation execution caused by network outages or system response timeouts cannot be detected in a timely manner, leading to inconsistent revocation data.
- Partners lack professional emergency response teams to handle revocation tasks on time during non-working hours.
SHECA is currently guiding partners to conduct targeted rectifications, after which MRIP&TP review drills will be organized. As necessary cycles should be reserved for rectification and verification, these two action items are expected to be completed by December 31st.
|
|
Assignee | |
Comment 37•8 months ago
|
||
The outstanding action items are progressing in an orderly manner and expected to be completed by December 31st.
|
|
||
Updated•8 months ago
|
|
|
Assignee | |
Comment 38•7 months ago
|
||
Action Items Update
Action Items
Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status Require partners to formulate MRIP&TP emergency plans, including setting up response teams, automatic re-signing scripts, network change procedures (such as network freeze exception processes), critical infrastructure emergency plans, etc. SHECA will provide plan templates and necessary guidance. Prevent Factor #1 Each partner has been notified of the relevant requirements. 2025-11-10 Completed Check all partners' revocation emergency plans to confirm they have established plans, the document content is complete, and the automated script meets the 24-hour revocation time limit requirements. Prevent Factor #1 The emergency plan document for each partner has been inspected and confirmed as complete. 2025-11-30 Completed by 2025-12-31 Require each of SHECA's partners to cooperate with SHECA in a complete MRIP&TP drill to ensure the entire process can be coordinated by all parties and is feasible at the operational level. Prevent Factor #1 At least one MRIP&TP drill has been completed with each partner, and all problems identified during execution have been required to be fixed by both parties. 2025-11-30 Completed by 2025-12-31 Formulate specific Guidance for customers involved in critical infrastructure services: clarify the definition of critical infrastructure and explain why it is not recommended to use public trusted webPKI certificates in such scenarios; for users involving critical infrastructure, recommend deploying private PKI or short-term certificates. Prevent Factor #2 The Guidance document has been written and finalized. 2025-11-10 Completed Issue a formal announcement to all users and partners, providing critical infrastructure Guidance while re-emphasizing the non-negotiable requirements for webPKI certificate revocation, ensuring all stakeholders understand their respective obligations. Prevent Factor #2 The formal announcement has been issued, and confirmation has been received that partners have understood its meaning. 2025-11-30 Completed Add detailed definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements. Prevent Factor #2 The updates to the CPS, Subscriber Agreement, and MRIP&TP have been published. 2025-11-20 Completed Further promote automated certificate frameworks such as ACME/ACME ARI to reduce the impact of certificate revocation on customer business. Prevent Factor #2 / 2025-11-30 Completed Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities. Mitigate / The bug has been successfully fixed and verified. 2025-11-10 Completed Establish an emergency communication mechanism for suspending TLS certificate issuance and revocation services during mass revocation events: Develop a dedicated notification template for partners, specifying the notification timeframe (e.g., within 1 hour of trigger), designated communication channels (e.g., email + WeChat Work), and core information to be communicated (suspension period, recovery time, inquiry method). Prevent / The notification template has been formulated, and the MRIP&TP has been updated. 2025-11-30 Completed Launch a function of separately issuing TLS-specific CRLs: Support manual on-demand immediate triggering of updates without relying on preset issuance frequency; control the CRL issuance time within 2 hours based on the existing certificate inventory. Prevent / The function has been launched. 2025-11-30 Completed
The outstanding action items have been completed. Up to now, all action items related to the problematic API and the delayed revocation (Bug 1993357, Bug 1994051) have been completed. SHECA has launched a supplementary assessment project specific to the two incidents, and the third-party auditor is expected to enter the site next week. (For detailed information on the assessment project, please refer to Comment 23 of Bug 1993357. )
- Schedule of the Supplementary Assessment
- The audit firm has been identified, with engagement background and demand fully aligned. Procurement procedures are in progress at both parties, and formal commissioning will be finalized upon completion of internal approvals.
- Standards based on: Baseline Requirements and WebTrust Guidelines.
- SHECA is progressively addressing the action items disclosed in Bugzilla, with full completion of all remediations targeted for November 2025. The audit team plans to commence assessment in December 2025 after remediations completed, and issue the report by January 2026.
- Scope of the assessment
The assessment will cover:
- processes and systems related to incidents 1993357 and 1994051
- all private key generation processes
- all certificate revocation processes (including mass revocations)
- mechanisms for verifying compliance with Baseline Requirements
3. Availability and form of the assessment
- A non-public assessment report will be issued by the auditor. Due to the auditor’s internal risk management policies, the detailed report will only be provided to SHECA management and cannot be disclosed to any third party.
- Should Browsers have any inquiries regarding the remediation of these two incidents during the engagement, the auditor is willing to provide feedbacks via email based on the testing results.
- A public summary of the key findings related to both incidents and the effectiveness of corrective actions will be provided by SHECA.
SHECA requests that the next update time be set as January 31, 2026.
|
|
||
Updated•7 months ago
|
|
|
Assignee | |
Comment 39•6 months ago
|
||
The audit team is conducting tests as planned. SHECA requests that the next update time be set to Feb 28, 2026.
|
|
||
Updated•6 months ago
|
|
|
Assignee | |
Comment 40•5 months ago
|
||
Owing to the impact of China’s Spring Festival holiday, the supplementary audit work has been postponed and is currently in the concluding phase. SHECA will update the status on a weekly basis commencing immediately.
|
|
Assignee | |
Comment 41•5 months ago
|
||
For the action items related to this incident, the auditors have completed the walkthrough tests. In addition, aligned with the main objectives of this audit, SHECA has implemented a series of additional measures to strengthen internal control management, which will be updated under the case.
|
|
||
Updated•5 months ago
|
|
|
Assignee | |
Comment 42•5 months ago
|
||
There are no updates available this week.
|
|
||
Comment 43•4 months ago
|
||
Please note that following the CCADB Incident Reporting Guidelines updates should be provided weekly, unless a 'next update' date has been set in advance. That is a separate incident to be raised.
When are reports updated?
CA Owners SHOULD respond promptly to comments and questions, and MUST respond within 7 days, even if only to acknowledge the request and provide a timeline for a full response.
|
|
Assignee | |
Comment 44•4 months ago
|
||
There are no updates available this week.(In reply to Wayne from comment #43)
Please note that following the CCADB Incident Reporting Guidelines updates should be provided weekly, unless a 'next update' date has been set in advance. That is a separate incident to be raised.
When are reports updated?
CA Owners SHOULD respond promptly to comments and questions, and MUST respond within 7 days, even if only to acknowledge the request and provide a timeline for a full response.
Hi,Wayne
Thank you for pointing this out.
Regarding this case, SHECA has no updates required this week. Regarding the delayed reporting issue, SHECA will open a new case to report this problem.
Thanks!
|
|
Assignee | |
Comment 45•4 months ago
|
||
Action item update
The assessment report has been finalized, which concludes the evaluation of control design effectiveness in meeting the six control objectives defined by SHECA as of March 23, 2026. The assessment procedures included walkthroughs, inquiries, documentation review, and other relevant testing procedures to validate the design of in-scope controls. No material deficiencies were identified, indicating that all controls are appropriately designed and ready for implementation. The six control objectives are as follows:
- CO1: Standardize system development process controls to ensure relevant compliance requirements are effectively identified and fulfilled
- CO2: Ensure security and compliance throughout the subscriber private key generation process, reducing the risks of unauthorized access and leakage
- CO3: Improve certificate revocation process to meet required timelines and relevant compliance requirements
- CO4: Improve partner revocation governance to mitigate external collaboration risks.
- CO5: Enhance BR compliance monitoring mechanisms
- CO6: Improve compliance training and compliance awareness(In reply to SHECA from comment #28)
|
|
Assignee | |
Comment 46•4 months ago
|
||
No update.
|
|
Assignee | |
Comment 47•4 months ago
|
||
Weekly update
There are no updates this week!
|
|
Assignee | |
Comment 48•4 months ago
|
||
We will send a closure report this week.
|
|
Assignee | |
Comment 49•3 months ago
|
||
Report Closure Summary
-
Incident description:
According to the Baseline Requirements (BR) 4.9.1.1, SHECA was required to complete the revocation of certificates involved in Bug #1993357 within 24 hours starting from 2025-10-08 20:35 (UTC+8). However, due to some reasons, the revocation process was delayed. As of 2025-10-17 10:00 (UTC+8), all the subscriber certificates (6920 in total) were revoked. As of 2025-10-24 20:15 (UTC+8), the pre-certificates (55 in total) generated due to a CT reporting bug were revoked.
-
Incident Root Cause(s):
Incomplete coverage of partner readiness in MRIP&TP verification and drills (failed to check the launch of auto-replacement script for all partners); Lack of effective risk management for certificates used in critical infrastructure (no clear guidance or differentiated review process in place, resulting in inappropriate use of public webPKI and increased revocation risks).
-
Remediation description:
SHECA has taken a series of remediation measures, including:
Require partners to develop MRIP&TP emergency plans (response teams, auto-reissue scripts, network change procedures, critical infrastructure plans);
Check partners’ revocation emergency plans for completeness and compliance with 24-hour revocation requirements;
Require partners to conduct full MRIP&TP drills with SHECA to validate operational feasibility;
Formulate specific guidance for critical infrastructure customers: define critical infrastructure, advise against public TLS certificates in such scenarios, and recommend private PKI or short-lived certificates;
Issue a formal announcement to users and partners to clarify critical infrastructure guidance and reinforce mandatory revocation requirements;
Add definitions of critical infrastructure to the CPS, Subscriber Agreement, and MRIP&TP, reaffirming the BR's revocation requirements;
Promote ACME/ACME ARI automation to reduce business impact from certificate revocation;
Fix the CT reporting bug where pre-certificates were issued with no subscriber certificates due to network abnormalities;
Establish an emergency communication mechanism for mass revocation events, including notification templates, channels, and timelines;
Launch a function of separately issuing TLS-specific CRLs within 2 hours.Additionally, SHECA has completed a supplementary independent third-party assessment covering incidents 1993357/1994051, private key generation, revocation processes, and BR compliance validation.
-
Commitment summary:
SHECA commits to maintain continuous improvement and inspection against six control objectives of the supplementary assessment.
CO1: Standardize system development process controls to ensure relevant compliance requirements are effectively identified and fulfilled
CO2: Ensure security and compliance throughout the subscriber private key generation process, reducing the risks of unauthorized access and leakage
CO3: Improve certificate revocation process to meet required timelines and relevant compliance requirements
CO4: Improve partner revocation governance to mitigate external collaboration risks
CO5: Enhance BR compliance monitoring mechanisms
CO6: Improve compliance training and compliance awareness
All Action Items disclosed in this report have been completed as described, and SHECA has passed the independent supplementary assessment. We request its closure.
|
|
||
Comment 50•3 months ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, it will be closed on approximately 2026-04-28.
|
|
Assignee | |
Comment 51•3 months ago
|
||
No updates for the case. We request its closure, Thanks.
|
|
||
Updated•3 months ago
|
Description
•