Closed Bug 1884568 Opened 1 year ago Closed 10 months ago

TWCA: Revocation delay for EV TLS certificates with invalid subject attribute order

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: hcli, Assigned: hcli)

References

(Blocks 1 open bug)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2025-03-03)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Steps to reproduce:

Incident Report

Summary

TWCA mis-issued 90 EV TLS certificates with subject attribute order nonconforming to BR Section 7.1.4.2. All affected certificates should be revoked within 5 days. However, because some of the customers cannot complete replacing their certificates which are used in critical applications, not all affected certificates were revoked in time.

Impact

13 out of the 90 mis-issued EV TLS certificates were not revoked in time. We expect to complete the revocation of these certificates no later than 2024-03-23.

Timeline

All times are UTC+8.

2023-09-15:

  • 08:00 BR for TLS 2.0.0 has become effective.

2024-03-04:

  • 21:00 Email reporting the issue received.

2024-03-05:

  • 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped the issuance of EV TLS certificates.
  • 17:30 Patch for the issuing system is completed.
  • 20:28 Preliminary report for the mis-issuance is posted (Bug 1883620).

2024-03-06:

  • 07:46 All affected certificates are identified.
  • 08:55 TWCA confirmed the patch is working as intended and resumed EV TLS certificates issuance.
  • 13:24 TWCA started contacting customers for replacing the certificate and collecting detailed reasons if the customer cannot replace the certificate in time.

2024-03-09:

  • 15:52 After risk assessment, we concluded there are 13 certificates that cannot be revoked in time.
  • 19:30 77 of the affected certificates have been revoked or expired.

Root Cause Analysis

The 13 unrevoked certificates belong to 3 banks. They all requested a delay in revocation because the certificates are installed on the servers for Internet banking/payment apps, and the certificates are pinned in those apps.

The apps need to be updated and published before they can replace the certificates on the servers, but they were unable to complete the publish processes in time. Revoking these certificates will interrupt these services and impact a great number of users. Trying to replace the certificates in 5 days using some emergency processes, skipping normal steps like QA or security testing, may impose higher risks.

Lessons Learned

What went well

  • Most customers are willing to cooperate and completed replacing the certificate in time.

What didn't go well

  • Some customers use the certificate in critical applications. Replacing certificates using emergency processes may impose higher risks.

Where we got lucky

N/A

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent 2024-03-31
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent 2024-03-31

Incident Report

Updated final report with details for each customer.

Summary

TWCA mis-issued 90 EV TLS certificates with subject attribute order nonconforming to BR Section 7.1.4.2. All affected certificates should be revoked within 5 days. However, because some of the customers cannot complete replacing their certificates which are used in critical applications, not all affected certificates were revoked in time.

Impact

13 out of the 90 mis-issued EV TLS certificates were not revoked in time. We expect to complete the revocation of these certificates no later than 2024-03-23.

Timeline

All times are UTC+8.

2023-09-15:

  • 08:00 BR for TLS 2.0.0 has become effective.

2024-03-04:

  • 21:00 Email reporting the issue received.

2024-03-05:

  • 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped the issuance of EV TLS certificates.
  • 17:30 Patch for the issuing system is completed.
  • 20:28 Preliminary report for the incident is posted (Bug 1883620).

2024-03-06:

  • 07:46 All affected certificates are identified.
  • 08:55 TWCA confirmed the patch is working as intended and resumed EV TLS certificates issuance.
  • 13:24 TWCA started contacting customers for replacing the certificate and collecting detailed reasons if the customer cannot replace the certificate in time.

2024-03-09:

  • 15:52 After risk assessment, we concluded there are 13 certificates that cannot be revoked in time.
  • 19:30 77 of the affected certificates have been revoked or expired.

2024-03-10+

  • 20:40 Posting this report.

Root Cause Analysis

The 13 unrevoked certificates belong to 3 banks. (See below) They all requested a delay in revocation because the certificates are installed on the servers for Internet banking/payment apps, and the certificates are pinned in those apps.

The apps need to be updated and published before they can replace the certificates on the servers, but they were unable to complete the publish processes in time. Revoking these certificates will interrupt these services and impact a great number of users. Trying to replace the certificates in 5 days using some emergency processes, skipping normal steps like QA or security testing, may impose higher risks.

Bank SinoPac Co., Ltd.

7 certificates total

Each certificate is used and pinned for a banking/financial app.

CTBC Bank Co., Ltd.

1 certificate total

The certificate is pinned in a payment app.

Hua Nan Commercial Bank, Ltd.

5 certificates total

3 of them are used and pinned for banking/financial apps.
1 is used and pinned for a payment app.
1 is used for communication with Financial Information Service Co., Ltd. (FISC, the national interbank processing center) and requires system update on FISC side.

Lessons Learned

What went well

  • Most customers are willing to cooperate and completed replacing the certificate in time.

What didn't go well

  • Some customers use the certificate in critical applications. Replacing certificates using emergency processes may impose higher risks.

Where we got lucky

N/A

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent 2024-03-31
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent 2024-03-31

Appendix

Details of affected certificates

https://crt.sh/?id=10800045622
https://crt.sh/?id=10800514489
https://crt.sh/?id=10829597911
https://crt.sh/?id=11236340596
https://crt.sh/?id=11403770564
https://crt.sh/?id=11705778194
https://crt.sh/?id=11846781240
https://crt.sh/?id=12227499561
https://crt.sh/?id=10569099309
https://crt.sh/?id=10891451209
https://crt.sh/?id=11032236585
https://crt.sh/?id=11092594508
https://crt.sh/?id=11403675054

Assignee: nobody → hcli
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [leaf-revocation-delay]

Following certificates has been revoked at 2024-03-12 10:53
https://crt.sh/?id=10569099309
https://crt.sh/?id=10891451209

The certificate below was revoked at 2024-03-13 17:38
https://crt.sh/?id=11403770564

https://crt.sh/?id=10829597911 revoked at 2024-03-20 16:15
https://crt.sh/?id=11705778194 revoked at 2024-03-20 16:16
https://crt.sh/?id=11846781240 revoked at 2024-03-20 16:16

The following 3 certificates have not yet been revoked.
https://crt.sh/?id=11032236585
https://crt.sh/?id=11092594508
https://crt.sh/?id=11403675054

All affected certificates in this incident have been revoked.

We continue to monitor the incident and complete action items as planned.

Thank you for this report and for explaining the perceived challenges for each customer. We’re hopeful that you can share a bit more related to the two Action Items you’ve listed. Specifically:

  • What feedback mechanisms are in place to ensure customers have understood these risks and the importance of quick action?
  • Can you share how customers are being advised to explore alternate methods to certificate pinning and what those methods might be?
  • How are you planning to monitor and evaluate the overall effectiveness of these two actions?

(In reply to Chris Clements from comment #9)

Thank you for this report and for explaining the perceived challenges for each customer. We’re hopeful that you can share a bit more related to the two Action Items you’ve listed. Specifically:

  • What feedback mechanisms are in place to ensure customers have understood these risks and the importance of quick action?

We will enhance the relevant descriptions in user contracts and notices, and educate our sales staff to explain the risks to customers before selling certificates.

  • Can you share how customers are being advised to explore alternate methods to certificate pinning and what those methods might be?

We currently suggest three possible methods:
1. Binding the public key instead of the certificate fingerprint.
2. Binding CA certificates, such as UCA certificates or RCA certificates.
3. Automatically updating the list of bound certificates upon app launch, allowing the binding of multiple certificates. When establishing a TLS connection, if it matches any one of the bound certificates, it is considered valid. This method requires the server side to provide a certificate retrieval interface.

  • How are you planning to monitor and evaluate the overall effectiveness of these two actions?
  • We plan to distribute a survey to our customers, which will include questions on the feasibility of automation and gather information on how customers use certificates.
  • Additionally, we will use this incident as a case study, setting up a scenario for revoking certificates on a large scale, and use it as a reference for Business Continuity Plan (BCP) drills.

We are currently updating the user agreement and preparing materials for internal educational training to ensure that our sales can explain the relevant risks and possible alternatives to our users. We will delay the completion time to synchronize with Bug 1886110.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent 2024-04-30
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent 2024-04-30

We've designed a preliminary version of a questionnaire that covers issues related to risk perception and certificate automation management. Once we've completed our internal discussions, we'll send it out to our customers to complete, allowing us to better understand them. We request to set the next update for April 30, 2024. Thank you.

Whiteboard: [ca-compliance] [leaf-revocation-delay] → [ca-compliance] [leaf-revocation-delay] Next update 2024-04-30

All items have been completed (synchronized with Bug 1886110):

  • Updated the correct customer contact information through this incident.
  • Updated user contracts, adding descriptions related to the termination of validity.
  • Completed sales staff education and training:
    • Explanation of requirements according to BR standards.
    • Alternative solutions for certificate binding.
    • Explanation of risks associated with wildcard certificates.
  • Feedback has been progressively received from the surveys previously sent to customers; internally, we will continue to evaluate this as a reference for future improvements in the certificate application process.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent Done
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent Done

We are continuously monitoring this incident.

We are continuously monitoring this incident.

Can you please revisit your delayed revocation action items and add some additional tasks to improve technical solutions that will speed up certificate replacement when needed? This could be based on what other CAs are doing in this area. If you need help identifying what I mean, I can point you to these other bugs, but just one example might be to implement Automated Certificate Management Environment (ACME) Renewal Information (ARI) and make it available to subscribers who might be able to implement that capability. Thanks.

Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-04-30 → [ca-compliance] [leaf-revocation-delay]

(In reply to Ben Wilson from comment #16)

Can you please revisit your delayed revocation action items and add some additional tasks to improve technical solutions that will speed up certificate replacement when needed? This could be based on what other CAs are doing in this area. If you need help identifying what I mean, I can point you to these other bugs, but just one example might be to implement Automated Certificate Management Environment (ACME) Renewal Information (ARI) and make it available to subscribers who might be able to implement that capability. Thanks.

Our system currently supports ACME but does not yet support ARI. As we are integrating the pkilint tool until the end of June, following evaluation by the development team, we commit to completing ARI support by the end of September.

We request to set the next update for June 30, 2024. Thank you.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent Done
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent Done
The CA system supports ACME Renewal Information (ARI) Mitigate 2024-09-30
Whiteboard: [ca-compliance] [leaf-revocation-delay] → [ca-compliance] [leaf-revocation-delay] Next update 2024-06-30

Currently, ACME ARI is developing as expected, we request setting the nextUpdate to Aug 30, 2024. Thank you.

Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-06-30 → [ca-compliance] [leaf-revocation-delay] Next update 2024-08-30
Blocks: 1911183

The ACME ARI feature has been successfully launched this week, and all the action items we committed to have been completed.

Action Items

Action Item Kind Due Date
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons Prevent Done
Ensure customers are aware of the risk of certificate pinning and alternative methods Prevent Done
The CA system supports ACME Renewal Information (ARI) Mitigate Done

We are continuously monitoring this issue.

We are continuously monitoring this issue.

Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-08-30 → [ca-compliance] [leaf-revocation-delay] Next update 2024-10-01
Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-10-01 → [ca-compliance] [leaf-revocation-delay] Next update 2024-10-31
Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-10-31 → [ca-compliance] [leaf-revocation-delay] Next update 2024-11-30

We continue work on incident-reporting and compliance requirements aimed at reducing delayed revocation, so this bug will remain open until at least February 1, 2025. Meanwhile, CAs should review https://github.com/mozilla/www.ccadb.org/pull/186.

Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2024-11-30 → [ca-compliance] [leaf-revocation-delay] Next update 2025-02-01
Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2025-02-01 → [ca-compliance] [leaf-revocation-delay] Next update 2025-03-03

Report Closure Summary

  • Incident description:
    • In Bug 1883620, TWCA mis-issued 90 EV TLS certificates with subject attribute order nonconforming to BR Section 7.1.4.2. All affected certificates must be revoked within 5 days.
      • However, 13 of these certificates were not timely revoked.
  • Incident Root Cause(s):
    • Customers were not made aware of the importance of immediate revocation.
    • Customers were not made aware of the risks associated with certificate binding.
    • The CA was too lenient in recognizing reasons for delayed revocation and did not enforce policies strictly with customers.
    • ACME automation has not been widely adopted and ARI has not yet been implemented.
  • Remediation description:
    • Complete the revocation of remaining certificates and update the progress weekly.
    • Revise the user agreement to explicitly prohibit delayed revocations, and grant the CA the right to revoke immediately as the expiration approaches without further notification to the customer.
    • Advise customers to bind to intermediate CA certificates if binding is necessary.
    • The CA system supports ACME ARI.
  • Commitment summary:
    We commit to adhering to the BRs and browser-related regulations, and through the user agreement, we will impose stricter requirements on users to prevent similar incidents from occurring again.

All Action Items disclosed in this report have been completed as described, and we request its closure.

I'll close this tomorrow, Friday, 7-Feb-2025.

Flags: needinfo?(bwilson)

In my review of this bug and the other currently open delrev bug from TWCA, I do not see evidence that the CA agrees it was at fault or has taken action to change. Statements such as “The CA was too lenient” and “We commit to adhering to the BRs and browser-related regulations” are not the same as a firm commitment not to delay revocation in the future.

“We commit to adhering to the BRs and browser-related regulations” to my ear sounds a lot like the non-statements we repeatedly got in 2024 from CAs who were trying to look like they were promising not to delay revocation while not, in fact, making that promise. Likewise, when I look at the Remediation Description in comment 37, I see a change in user agreement language but no actual policy change in TWCA and its decision-making process or criteria around BR-mandated revocation.

I suggest that TWCA should commit to a policy change defaulting to No on any requested revocation delay and commit this policy to the Bugzilla community. If this is simply a matter of the wording in the post, we recently saw a pledge of this sort from Chunghwa Telecom, which could be a good model for TWCA to look at. A clarifying post in the same spirit as Chunghwa Telecom’s pledge would in my mind serve as a firm commitment to the community that TWCA has revised its policy. At that point Sectigo would be prepared to support closing this bug.

(In reply to Tim Callan from comment #25)

In my review of this bug and the other currently open delrev bug from TWCA, I do not see evidence that the CA agrees it was at fault or has taken action to change. Statements such as “The CA was too lenient” and “We commit to adhering to the BRs and browser-related regulations” are not the same as a firm commitment not to delay revocation in the future.

“We commit to adhering to the BRs and browser-related regulations” to my ear sounds a lot like the non-statements we repeatedly got in 2024 from CAs who were trying to look like they were promising not to delay revocation while not, in fact, making that promise. Likewise, when I look at the Remediation Description in comment 37, I see a change in user agreement language but no actual policy change in TWCA and its decision-making process or criteria around BR-mandated revocation.

I suggest that TWCA should commit to a policy change defaulting to No on any requested revocation delay and commit this policy to the Bugzilla community. If this is simply a matter of the wording in the post, we recently saw a pledge of this sort from Chunghwa Telecom, which could be a good model for TWCA to look at. A clarifying post in the same spirit as Chunghwa Telecom’s pledge would in my mind serve as a firm commitment to the community that TWCA has revised its policy. At that point Sectigo would be prepared to support closing this bug.

We completely agree with your statement—it was our lack of clarity.

Policy: We acknowledge that in the past we made erroneous decisions that did not meet the WebPKI community’s expectations for timely certificate revocation.

Technical: We are committed to employing various technological measures to prevent future mistakes and have already established automation as our primary focus.

Customer: We have revised our terms of use to emphasize that delayed revocation is unacceptable, building consensus with both new and existing customers.

We promise to comply with the BR and all related regulations, and we will not delay revocation in the future.

I will close this on Friday, 14-Feb-2025.

Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.