1959278 - Chunghwa Telecom: Delayed revocation for bug 1951415

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Tsung-Min Kuo]

Preliminary Incident Report

Summary

This is a preliminary report, a full incident report will be posted no later than 2025-04-23.

• Incident description:
  During the discussion in bug 1951415, we agree with the perspective that TLS BR 4.9.1.1 Item 5 is an appropriate reference, there was indeed a delayed revocation by CHT in this incident.

We also noted that this conversation related to bug 19511415, I agree with Aaron and Jeremy. CAA failures should be treated as requiring a 24 hour revocation deadline. We will take this case as lesson learned and keep watch on similar cases to ensure that future situations meet the requirement of TLS BR, especially Item 5 of TLS BR 4.9.1.1, and can be addressed promptly.

TLS BR 4.9.1.1 (5) states:
"The CA obtains evidence that the validation of domain authorization or control for any Fully‐Qualified Domain Name or IP address in the Certificate should not be relied upon (CRLReason #4, superseded)."

• Relevant policies: TLS BR 4.9.1.1
• Source of incident disclosure: Third Party Reported

Comment 1

[Tsung-Min Kuo]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A010946
• Incident description: This report explains an incident involving a delay in certificate revocation. The incident was caused by a batch of 11,860 TLS certificates issued by CHT, where CAA validation was not properly executed(bug 1951415). Although most of the affected certificates have been revoked according to procedure, 4 of them were not revoked within the 24-hour period specified in TLS BR 4.9.1.1 Item 5, thereby violating relevant regulations.

During the discussion in bug 1951415, we agree with the perspective that TLS BR 4.9.1.1 Item 5 is an appropriate reference, there was indeed a delayed revocation by CHT in this incident.

• Timeline summary:
  • Non-compliance start date: 2025-03-02
  • Non-compliance identified date: 2025-03-27
  • Non-compliance end date: 2025-03-03
• Relevant policies: Section 4.2.1 of our CPS (Version 1.0), TLS BR 4.9.1.1 Item 5 (Version 2.1.4)

TLS BR 4.9.1.1 states:
"With the exception of Short‐lived Subscriber Certificates, the CA SHALL revoke a Certificate within 24 hours and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:
...
Item 5. The CA obtains evidence that the validation of domain authorization or control for any
Fully‐Qualified Domain Name or IP address in the Certificate should not be relied upon
(CRLReason #4, superseded).

• Source of incident disclosure: Third Party Reported

Impact

• Total number of certificates: 4
• Total number of "remaining valid" certificates: 0
• Affected certificate types: This incident affects OV certificates (with OID 2.23.140.1.2.2).
• Incident heuristic: 4, the full corpus of affected certificates are disclosed in the Appendix.
• Was issuance stopped in response to this incident, and why or why not?: Not applicable, when this incident was identified as delayed revocation, all affected certificates were revoked, so it does not affect the issuance of new certificates.
• Analysis: Although all certificates involved in this incident have been revoked, there was a non-compliance with the timeliness of certificate revocation because 4 certificates were not revoked within the 24-hour period specified by TLS BR Section 4.9.1.1(5), where we originally believed that TLS BR 4.9.1.1(12) was more appropriate.
• Additional considerations:

Timeline

All times are UTC+8.

2025-03-01:

• 11:10 CHT became aware of the incident (bug 1951415).
• 11:27 CHT holds a preliminary incident discussion meeting.
• 12:28 Convene the CA team members and personnels of computer room.
• 17:47 Begin revoking the first problematic certificate.
• 18:27 Categorize the root cause of the incident as related to BR 4.9.1.1 (12) and begin large-scale revocation process.
• 21:30 Aware of several more certificates should also be revoked. However, because the computer room staff have already left work, we decided to revoke these certificates on 2025-03-03.

2025-03-03:

• 13:30 After a detailed investigation and inventory, an additional 4 affected certificates were identified.
• 14:59 Check the OCSP status of these four recently revoked certificates and confirm that the CRL status has been updated.

2025-03-04

• 22:44 Incident report bug1951415 posted.

2025-03-19

• Chrome root program mentioned in comment#15 of bug1951415 that 4.9.1.1 Item 5 should be the correct benchmark.

2025-03-27

• After internal discussion, we agreed that TLS BR 4.9.1.1 Item 5 is an appropriate reference, there was indeed a delay in revocation by CHT in the incident. The incident led to the creation of this bug (1959278), where 4 certificates were not revoked within the 24-hour period.

2025-04-09

• Submit the Preliminary Incident Report for bug 1959278.

Related Incidents

Bug	Date	Description
1951415	2025-03-04	Failure to check restrictive CAA record during Migration.

Root Cause Analysis

** Contributing Factor 1: Inappropriate reference of relevant policies**

• Description: The root cause of this incident was a misunderstanding of TLS BR 4.9.1.1(5) and inappropriate internal handling decisions. Specifically, when CHT received a notification from the Chrome Root Program regarding issues with certain certificates where CAA checks were not correctly executed, the revocation process was initiated immediately. However, in our preliminary investigation meeting, we should categorize the root cause of the incident as related to BR 4.9.1.1 (5) rather than BR 4.9.1.1 (12). Item #5 obligates CAs to revoke affected certificates within 24 hours, which led to some certificates being revoked beyond the specified 24-hour timeframe.

• Timeline:
  2025-03-01 18:27 Categorize the root cause of the incident as related to BR 4.9.1.1 (12) and begin large-scale revocation process.
  2025-03-01 21:30 Aware of several more certificates should also be revoked. However, because the computer room staff have already left work, we decided to revoke these certificates on 2025-03-03.
  2025-03-03 13:30 After a detailed investigation and inventory, an additional 4 affected certificates were identified.
  2025-03-19 Chrome root program mentioned in comment#15 of bug1951415 that 4.9.1.1 Item 5 should be the correct benchmark.
  2025-03-27 After internal discussion, we agreed that TLS BR 4.9.1.1 Item 5 is an appropriate reference

• Detection: During the discussion on 2025-03-27, we agreed that TLS BR 4.9.1.1 Item 5 is an appropriate reference, there was indeed a delay in revocation by CHT in this incident.

• Interaction with other factors:
  (1) The CHT team's misunderstanding of BR 4.9.1.1 led to an incorrect determination of the revocation timeframe, resulting in the revocation of 4 certificates exceeding the specified 24-hour limit.
  (2) The root cause of the delay was that 24 hours was extremely short to replace certificates. However, we actually had the opportunity to achieve it, but since the incident occurred during a holiday, it was more difficult to contact the affected customers. In the future, we should strengthen our notification channels.

Lessons Learned

• What went well: CHT revoked 11,860 certificates in 3 days.

• What didn’t go well:
  (1) Our team lacks an understanding of compliance against to TLS BRs and root program expectations this time.
  (2) Many customers were unaware of the revocation.

• Where we got lucky: Upon notification on 2025-03-03, the customers of the 4 affected certificates agreed to immediately replace them, allowing our revocation process to proceed smoothly.

• Additional: The industry needs a better notification process. Some customers thought the email notification was phishing. Others claimed not to receive it or had no habit of receiving mail on holidays.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Personnel retraining as to BR compliance	Prevent	Root Cause # 1	We will conduct retraining actions, including BR policies.	2025-04-10	Completed
Customers communications as to BR compliance	Prevent	What didn’t go well	Highlights our user agreement terms to inform customers that if the CA issues a certificate without complete/proper verification, there may be situations where immediate revocation is necessary.	2025-04-15	Completed

Appendix

See Attached.

Comment 2

[Tsung-Min Kuo]

Attachment: certsn_delayrev_crtsh_v1_4.csv

Comment 3

[Tsung-Min Kuo]

Update:

I found there's a typo.

• Relevant policies: Section 4.2.1 of our CPS (Version 1.0), TLS BR 4.9.1.1 Item 5 (Version 2.1.4)

should be

• Relevant policies: Section 4.9.1.1 of our CPS (Version 1.0), TLS BR 4.9.1.1 Item 5 (Version 2.1.4)

Comment 4

[Tsung-Min Kuo]

We are continuing to monitor this issue.

Comment 5

[chrome-root-program]

If CHT feels this bug is ready for closure, it should submit a closure report.

Comment 6

[Tsung-Min Kuo]

Thanks for the reminder, I will submit it today.

Comment 7

[Tsung-Min Kuo]

Report Closure Summary

• Incident description: This incident stemmed from CHT's certificate revocation process, which was initiated in response to a prior issue where CAA checks were not fully performed during a large-scale migration (Bug 1951415). Although most of the affected certificates have been revoked according to procedure, 4 of them were not revoked within the 24-hour period specified in TLS BR 4.9.1.1 Item 5, leading to a compliance violation.

• Incident Root Cause(s):
  (a) Inappropriate reference of relevant policies: The CHT team's misunderstanding of BR 4.9.1.1 led to an incorrect determination of the revocation timeframe, where TLS BR 4.9.1.1 (5) is an appropriate reference rather than BR 4.9.1.1 (12), resulting in the revocation of 4 certificates exceeding the specified 24-hour limit.
  (b) Another root cause of the delay was that 24 hours was extremely short to replace certificates. However, we actually had the opportunity to achieve it. However, since the incident occurred during a holiday period, it was more difficult to contact the affected customers.

• Remediation description: We completed the revocation of the remaining 4 certificates on March 3, 2025, and on March 27, we confirmed that TLS BR 4.9.1.1 (5) is an appropriate reference. And we have conducted the following actions:
  (a) Retraining our CA team members on TLS BR compliance. In particular, the correct interpretation and application of relevant regulations were reinforced through case-based discussions, enabling team members to gain a deeper understanding of the requirements.
  (b) Strengthened communication with subscribers and emphasized the importance of adhering to our Subscriber Agreement. This ensures that subscribers clearly understand the CA’s obligation to revoke certificates within a very short timeframe when validation data fails to meet compliance requirements.

• Commitment summary: All improvement actions evaluated in this incident, including the Internal Training Measures and Subscriber Communication Enhancements, have been completed on schedule. We continue to strengthen our understanding of the TLS Baseline Requirements to prevent similar incidents from occurring in the future.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 8

[Tsung-Min Kuo]

We are continuing to monitor this issue.

Comment 9

[Tsung-Min Kuo]

We are continuing to monitor this issue.

Comment 10

[Tsung-Min Kuo]

We are continuing to monitor this issue.

Comment 11

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-06-10.