Open Bug 1884568 Opened 7 months ago Updated 1 day ago

TWCA: Revocation delay for EV TLS certificates with invalid subject attribute order

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: hcli, Assigned: hcli)

References

(Blocks 1 open bug)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-10-31)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Steps to reproduce:

Incident Report

Summary

TWCA mis-issued 90 EV TLS certificates with subject attribute order nonconforming to BR Section 7.1.4.2. All affected certificates should be revoked within 5 days. However, because some of the customers cannot complete replacing their certificates which are used in critical applications, not all affected certificates were revoked in time.

Impact

13 out of the 90 mis-issued EV TLS certificates were not revoked in time. We expect to complete the revocation of these certificates no later than 2024-03-23.

Timeline

All times are UTC+8.

2023-09-15:

  • 08:00 BR for TLS 2.0.0 has become effective.

2024-03-04:

  • 21:00 Email reporting the issue received.

2024-03-05:

  • 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped the issuance of EV TLS certificates.
  • 17:30 Patch for the issuing system is completed.
  • 20:28 Preliminary report for the mis-issuance is posted (Bug 1883620).

2024-03-06:

  • 07:46 All affected certificates are identified.
  • 08:55 TWCA confirmed the patch is working as intended and resumed EV TLS certificates issuance.
  • 13:24 TWCA started contacting customers for replacing the certificate and collecting detailed reasons if the customer cannot replace the certificate in time.

2024-03-09:

  • 15:52 After risk assessment, we concluded there are 13 certificates that cannot be revoked in time.
  • 19:30 77 of the affected certificates have been revoked or expired.

Root Cause Analysis

The 13 unrevoked certificates belong to 3 banks. They all requested a delay in revocation because the certificates are installed on the servers for Internet banking/payment apps, and the certificates are pinned in those apps.

The apps need to be updated and published before they can replace the certificates on the servers, but they were unable to complete the publish processes in time. Revoking these certificates will interrupt these services and impact a great number of users. Trying to replace the certificates in 5 days using some emergency processes, skipping normal steps like QA or security testing, may impose higher risks.

Lessons Learned

What went well

  • Most customers are willing to cooperate and completed replacing the certificate in time.

What didn't go well

  • Some customers use the certificate in critical applications. Replacing certificates using emergency processes may impose higher risks.

Where we got lucky

N/A

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent 2024-03-31
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent 2024-03-31

Incident Report

Updated final report with details for each customer.

Summary

TWCA mis-issued 90 EV TLS certificates with subject attribute order nonconforming to BR Section 7.1.4.2. All affected certificates should be revoked within 5 days. However, because some of the customers cannot complete replacing their certificates which are used in critical applications, not all affected certificates were revoked in time.

Impact

13 out of the 90 mis-issued EV TLS certificates were not revoked in time. We expect to complete the revocation of these certificates no later than 2024-03-23.

Timeline

All times are UTC+8.

2023-09-15:

  • 08:00 BR for TLS 2.0.0 has become effective.

2024-03-04:

  • 21:00 Email reporting the issue received.

2024-03-05:

  • 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped the issuance of EV TLS certificates.
  • 17:30 Patch for the issuing system is completed.
  • 20:28 Preliminary report for the incident is posted (Bug 1883620).

2024-03-06:

  • 07:46 All affected certificates are identified.
  • 08:55 TWCA confirmed the patch is working as intended and resumed EV TLS certificates issuance.
  • 13:24 TWCA started contacting customers for replacing the certificate and collecting detailed reasons if the customer cannot replace the certificate in time.

2024-03-09:

  • 15:52 After risk assessment, we concluded there are 13 certificates that cannot be revoked in time.
  • 19:30 77 of the affected certificates have been revoked or expired.

2024-03-10+

  • 20:40 Posting this report.

Root Cause Analysis

The 13 unrevoked certificates belong to 3 banks. (See below) They all requested a delay in revocation because the certificates are installed on the servers for Internet banking/payment apps, and the certificates are pinned in those apps.

The apps need to be updated and published before they can replace the certificates on the servers, but they were unable to complete the publish processes in time. Revoking these certificates will interrupt these services and impact a great number of users. Trying to replace the certificates in 5 days using some emergency processes, skipping normal steps like QA or security testing, may impose higher risks.

Bank SinoPac Co., Ltd.

7 certificates total

Each certificate is used and pinned for a banking/financial app.

CTBC Bank Co., Ltd.

1 certificate total

The certificate is pinned in a payment app.

Hua Nan Commercial Bank, Ltd.

5 certificates total

3 of them are used and pinned for banking/financial apps.
1 is used and pinned for a payment app.
1 is used for communication with Financial Information Service Co., Ltd. (FISC, the national interbank processing center) and requires system update on FISC side.

Lessons Learned

What went well

  • Most customers are willing to cooperate and completed replacing the certificate in time.

What didn't go well

  • Some customers use the certificate in critical applications. Replacing certificates using emergency processes may impose higher risks.

Where we got lucky

N/A

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent 2024-03-31
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent 2024-03-31

Appendix

Details of affected certificates

https://crt.sh/?id=10800045622
https://crt.sh/?id=10800514489
https://crt.sh/?id=10829597911
https://crt.sh/?id=11236340596
https://crt.sh/?id=11403770564
https://crt.sh/?id=11705778194
https://crt.sh/?id=11846781240
https://crt.sh/?id=12227499561
https://crt.sh/?id=10569099309
https://crt.sh/?id=10891451209
https://crt.sh/?id=11032236585
https://crt.sh/?id=11092594508
https://crt.sh/?id=11403675054

Assignee: nobody → hcli
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [leaf-revocation-delay]

Following certificates has been revoked at 2024-03-12 10:53
https://crt.sh/?id=10569099309
https://crt.sh/?id=10891451209

The certificate below was revoked at 2024-03-13 17:38
https://crt.sh/?id=11403770564

https://crt.sh/?id=10829597911 revoked at 2024-03-20 16:15
https://crt.sh/?id=11705778194 revoked at 2024-03-20 16:16
https://crt.sh/?id=11846781240 revoked at 2024-03-20 16:16

The following 3 certificates have not yet been revoked.
https://crt.sh/?id=11032236585
https://crt.sh/?id=11092594508
https://crt.sh/?id=11403675054

All affected certificates in this incident have been revoked.

We continue to monitor the incident and complete action items as planned.

Thank you for this report and for explaining the perceived challenges for each customer. We’re hopeful that you can share a bit more related to the two Action Items you’ve listed. Specifically:

  • What feedback mechanisms are in place to ensure customers have understood these risks and the importance of quick action?
  • Can you share how customers are being advised to explore alternate methods to certificate pinning and what those methods might be?
  • How are you planning to monitor and evaluate the overall effectiveness of these two actions?

(In reply to Chris Clements from comment #9)

Thank you for this report and for explaining the perceived challenges for each customer. We’re hopeful that you can share a bit more related to the two Action Items you’ve listed. Specifically:

  • What feedback mechanisms are in place to ensure customers have understood these risks and the importance of quick action?

We will enhance the relevant descriptions in user contracts and notices, and educate our sales staff to explain the risks to customers before selling certificates.

  • Can you share how customers are being advised to explore alternate methods to certificate pinning and what those methods might be?

We currently suggest three possible methods:
1. Binding the public key instead of the certificate fingerprint.
2. Binding CA certificates, such as UCA certificates or RCA certificates.
3. Automatically updating the list of bound certificates upon app launch, allowing the binding of multiple certificates. When establishing a TLS connection, if it matches any one of the bound certificates, it is considered valid. This method requires the server side to provide a certificate retrieval interface.

  • How are you planning to monitor and evaluate the overall effectiveness of these two actions?
  • We plan to distribute a survey to our customers, which will include questions on the feasibility of automation and gather information on how customers use certificates.
  • Additionally, we will use this incident as a case study, setting up a scenario for revoking certificates on a large scale, and use it as a reference for Business Continuity Plan (BCP) drills.

We are currently updating the user agreement and preparing materials for internal educational training to ensure that our sales can explain the relevant risks and possible alternatives to our users. We will delay the completion time to synchronize with Bug 1886110.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent 2024-04-30
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent 2024-04-30

We've designed a preliminary version of a questionnaire that covers issues related to risk perception and certificate automation management. Once we've completed our internal discussions, we'll send it out to our customers to complete, allowing us to better understand them. We request to set the next update for April 30, 2024. Thank you.

Whiteboard: [ca-compliance] [leaf-revocation-delay] → [ca-compliance] [leaf-revocation-delay] Next update 2024-04-30

All items have been completed (synchronized with Bug 1886110):

  • Updated the correct customer contact information through this incident.
  • Updated user contracts, adding descriptions related to the termination of validity.
  • Completed sales staff education and training:
    • Explanation of requirements according to BR standards.
    • Alternative solutions for certificate binding.
    • Explanation of risks associated with wildcard certificates.
  • Feedback has been progressively received from the surveys previously sent to customers; internally, we will continue to evaluate this as a reference for future improvements in the certificate application process.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent Done
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent Done

We are continuously monitoring this incident.

We are continuously monitoring this incident.

Can you please revisit your delayed revocation action items and add some additional tasks to improve technical solutions that will speed up certificate replacement when needed? This could be based on what other CAs are doing in this area. If you need help identifying what I mean, I can point you to these other bugs, but just one example might be to implement Automated Certificate Management Environment (ACME) Renewal Information (ARI) and make it available to subscribers who might be able to implement that capability. Thanks.

Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-04-30 → [ca-compliance] [leaf-revocation-delay]

(In reply to Ben Wilson from comment #16)

Can you please revisit your delayed revocation action items and add some additional tasks to improve technical solutions that will speed up certificate replacement when needed? This could be based on what other CAs are doing in this area. If you need help identifying what I mean, I can point you to these other bugs, but just one example might be to implement Automated Certificate Management Environment (ACME) Renewal Information (ARI) and make it available to subscribers who might be able to implement that capability. Thanks.

Our system currently supports ACME but does not yet support ARI. As we are integrating the pkilint tool until the end of June, following evaluation by the development team, we commit to completing ARI support by the end of September.

We request to set the next update for June 30, 2024. Thank you.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent Done
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent Done
The CA system supports ACME Renewal Information (ARI) Mitigate 2024-09-30
Whiteboard: [ca-compliance] [leaf-revocation-delay] → [ca-compliance] [leaf-revocation-delay] Next update 2024-06-30

Currently, ACME ARI is developing as expected, we request setting the nextUpdate to Aug 30, 2024. Thank you.

Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-06-30 → [ca-compliance] [leaf-revocation-delay] Next update 2024-08-30
Blocks: 1911183

The ACME ARI feature has been successfully launched this week, and all the action items we committed to have been completed.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent Done
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent Done
The CA system supports ACME Renewal Information (ARI) Mitigate Done

We are continuously monitoring this issue.

We are continuously monitoring this issue.

Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-08-30 → [ca-compliance] [leaf-revocation-delay] Next update 2024-10-01
Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-10-01 → [ca-compliance] [leaf-revocation-delay] Next update 2024-10-31
You need to log in before you can comment on or make changes to this bug.