Open Bug 1887096 Opened 1 month ago Updated 8 days ago

Chunghwa Telecom: Wrong Extended Key Usage setting by GTLSCA

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: tmkuo, Assigned: tmkuo, NeedInfo)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(1 file)

Delayed Revocation List (6,450 Records)
359.03 KB, text/csv
Details

Incident Report (on behalf of GTLSCA)

Summary

GTLSCA received the problem report on March 19, 2024, via CHT root CA team, and conducted preliminary processing within 24 hours of receiving the report. First, we revoked the three certs listed on the report and notify the users to re-apply. After a one-day detail investigation, we found there are about 6,450 certs, including the revoked ones, with wrong EKU setting. After impact assessment, we decide to make the certificate renewal for all the certs impacted automatically to reduce the manpower and time costs required for manual verification, and an announcement will be issued to notify the users to download the updated certs.

How we first became aware of the problem.

On 19 March 2024 about 00:41, GTLSCA received the certificate mis-issuance notification from CHT root CA team.

Impact

There are 6,450 were affected, all these problematic certs will be revoked and renewed.

Timeline of incident and actions taken in response.

All times are UTC.

2024-03-19

  • 01:35 RAO counter informs the users of 3 listed certs informed by Chrome Root Program this incident, and ask them to re-apply asap, and GTLSCA revoked the problematic certs after reissued the new ones in hours.
  • 01:41 Ask RAO to suspend review and issuance of new application cases and starts an action plan and initial investigation.
  • 02:55 Confirmed the impact scope and checked the problem has been fixed on 2024-03-11. The preliminary investigation is completed.
  • 03:15 Certificate issuing continued.
  • 03:21 Revoked the first two problematic certificates on the report.
  • 03:27 Hold the first investigation report meeting, confirm the problem, and impact scope.
  • 04:39 Revoked the third problematic certificates on the report.
  • 07:00 Hold the second report meeting, propose solutions with minimal impact on users and how to avoid recurrence.
  • 08:32 Provide incident report to CHT root CA team.

Root Cause Analysis

The mistake was made due to the misunderstanding of the profile as to the EKU in the new version TLS BR. GTLSCA ignored the part where the EKU itself cannot be marked as critical. We have corrected this problem on March 11, 2024.

Action Items

GTLSCA will complete these remediation actions by Thursday, April 4, and will reissue the affected certificates on or before that date.

Remediation Status Due Date
announce the incident and the reissuance plan Not yet started 2024-03-25
reissue new certificates started 2024-03-19
revoke all problematic certs started 2024-04-05

[!NOTE] Update the Action item part and the corresponding due dates as below:

Action Items

GTLSCA will complete these remediation actions by Thursday, April 17, and will reissue the affected certificates on or before that date.

Remediation Status Due Date
announce the incident and the reissuance plan Not yet started 2024-04-01
reissue new certificates started 2024-03-29
revoke all problematic certs started 2024-04-17
Assignee: nobody → tmkuo
Status: UNCONFIRMED → ASSIGNED
Type: enhancement → task
Ever confirmed: true
Summary: GTLSCA: Wrong Extended Key Usage setting → Chunghwa Telecom: Wrong Extended Key Usage setting by GTLSCA
Whiteboard: [ca-compliance] [ov-misissuance]

Hi GTLSCA,

The CCADB incident reporting requirements state:

The timeline must include not just the actual discovery of the incident and subsequent events, but also relevant events occuring beforehand (e.g. something changed or was introduced).

Can you please update the timeline to indicate when the first relevant profile settings were first put into place, when the first misissued certificate was issued, etc? Thanks.

(In reply to Aaron Gable from comment #2)

Hi GTLSCA,

The CCADB incident reporting requirements state:

The timeline must include not just the actual discovery of the incident and subsequent events, but also relevant events occuring beforehand (e.g. something changed or was introduced).

Can you please update the timeline to indicate when the first relevant profile settings were first put into place, when the first misissued certificate was issued, etc? Thanks.

Hi Aaron

This incident should be traced back to 2023/9/15, the effective date of Baseline Requirement v2.0.0.

The first wrong certificate issuanced time should start from 2023/9/15

We found this problem when self-checking the specifications on 2024/3/5, repaired it, and corrected the problem on 3/11.

So the certificates issued after 3/11 will be correct.

In conjunction with this incident, the previously mistakenly issued certificate will be reissued(with same keypair) to the user.

The reason for this incident mainly due to the certificate EKU cannot be marked as critical after effective date of BR v2.0.0 on 2023/09/15, where the problem was discoved and further fixed at 14:30, 2023/03/11.

We are working on reissuing all the problematic certificates issued between 2023/09/15 to 2023/03/11.

Leo, Tsung-Min,

https://www.ccadb.org/cas/incident-report says "the Appendix must include a listing of the complete certificate details of all affected certificates. The recommended format is to ensure that all affected certificates are logged to CT, then to attach a text file where each line is of the form https://crt.sh/?sha256=[sha256 fingerprint of the certificate]."

There is no list of affected certificates attached to this bug. Are you able to attach "a listing of the complete certificate details of all affected certificates" using "the recommended format"?

https://www.ccadb.org/cas/incident-report also says "There should be a single incident report for each distinct matter, and CA Owners should submit an additional, separate incident report when: Policy requires the revocation of one or more certificates by a certain deadline, such as those in BR section 4.9, but that deadline will not be or has not been met by the CA Owner."

Per TLSBR 4.9.1.1, these certificates were "12...not issued in accordance with these Requirements" and should have been revoked within 5 days. Please open a new bug and provide an incident report for this delayed revocation.

(In reply to Rob Stradling from comment #7)

Hi Rob,

  1. Regarding your question, we listed the list of affected certificates on April 10 and uploaded it to the appendix.
  2. Regarding the revoking of all credentials within 5 days, due to the large impact, we will open another bug to explain this matter.
    Thank you for your advice

This incident still has no real timeline. Here is an example incident that has done an excellent job with reporting it per the guidelines:

https://bugzilla.mozilla.org/show_bug.cgi?id=1886876

Beyond the IR being necessary for delayed revocation, there is an additional question on why there is a delay in making the delayed revocation IR.

The initial IR was also delayed, being opened ~3 days after receiving the report.

TimeLine

All times are UTC+8.

2023-09-15

  • BR for TLS 2.0.0 has become effective.

2024-02-26

  • When GTLSCA PMA reviewed CPS v1.0.3 and BR, it was found that the extKeyUsage Boolean Flag was "MUST" to be marked as non-critical in version v2.0.0, and it was proposed to adjust the certificate profile.
  • Update GTLSCA CPS v1.0.4 draft.

2024-03-05

  • Provide certificate profile and update documentation based on BR.
  • Submit GTLSCA CPS v1.0.4 to RootCA for review.

2024-03-11

  • 14:16 Updated the certificate profile setting to mark the certificate field extKeyUsage as non-critical.
  • 14:34 Certificate issued after this point in time comply with BR standards.

2024-03-19

  • 09:35 RAO counter informs the users of 3 listed certs informed by Chrome Root Program this incident, and ask them to re-apply asap, and GTLSCA revoked the problematic certs after reissued the new ones in hours.
  • 09:41 Ask RAO to suspend review and issuance of new application cases and starts an action plan and initial investigation.
  • 10:55 Confirmed the impact scope and checked the problem has been fixed on 2024-03-11. The preliminary investigation is completed.
  • 11:15 Certificate issuing continued.
  • 11:21 Revoked the first two problematic certificates on the report.
  • 11:27 Hold the first investigation report meeting, confirm the problem, and impact scope.
  • 12:39 Revoked the third problematic certificates on the report.
  • 15:00 Hold the second report meeting, propose solutions with minimal impact on users and how to avoid recurrence.
  • 16:32 Provide incident report to CHT root CA team.

2024-03-22

  • Posted this incident report.

2024-03-24

  • According to BR, all problematic certificates should be revoked at this time.

(In reply to amir from comment #10)

This incident still has no real timeline. Here is an example incident that has done an excellent job with reporting it per the guidelines:

=>Supplementary TimeLine as mentioned in post #11.

https://bugzilla.mozilla.org/show_bug.cgi?id=1886876

Beyond the IR being necessary for delayed revocation, there is an additional question on why there is a delay in making the delayed revocation IR.

==>In response to this incident of mistaken issuance, the verification targets are all government units and government agency websites. We have assessed that the cause of this mis-issuance does not involve a key issue, but only a certificate field issue, which will not affect the customer's information security. In addition, in accordance with the administrative efficiency of government agencies, from notification to the start of processing, it requires agency supervisors at all levels. Signing and approval, and some public agencies need to find information vendors for processing, so it is difficult to complete the replacement within 5 days. Therefore, the certificate is postponed and revoked within a time limit so that the certificates of all websites can be updated smoothly.

We understand Mozilla's position that CA should comply with BR's requirements. However, considering different circumstances, the harm caused may exceed the choice to meet this requirement, and the risk cannot be transferred to users who use TLS certificates.

In addition, we have not encountered such a large number of requests for certificate abolition and renewal. The original program basically provides for revocation, but the package does not preset a large number of certificate renewals, so this incident has a large part The time lies in developing the program for renewing certificates.

The initial IR was also delayed, being opened ~3 days after receiving the report.

==>We received the notice on March 19 and are not familiar with the reporting mechanism. After listening to the opinions of experienced colleagues, we first asked experienced colleagues to forward the post on our behalf on March 22. Unfortunately, it is still not possible to present the bug immediately.

(In reply to Leo Fang from comment #9)

(In reply to Rob Stradling from comment #7)

Hi Rob,

  1. Regarding your question, we listed the list of affected certificates on April 10 and uploaded it to the appendix.
  2. Regarding the revoking of all credentials within 5 days, due to the large impact, we will open another bug to explain this matter.
    Thank you for your advice

==>We create a new bug 1892419 for Delayed Revocation.

Another question: What is the purpose of the extension with the oid of 2.5.29.9 in your certificates?

Flags: needinfo?(tmkuo)

(In reply to amir from comment #14)

Another question: What is the purpose of the extension with the oid of 2.5.29.9 in your certificates?

About the contents of 2.5.29.9, please refer to CPS section 7.1.2.2.(page74-75)
https://gtlsca.nat.gov.tw/download/GTLSCA_CPS_v1.0.3_eng.pdf
We put some government related attributes such as goverment agency identifiers at this extension.

Can you give an example of what these might be? How are they used by the relaying parties?

I'm looking at this for example: https://crt.sh/?id=12589745529

Beyond that, when looking at that section of your CPS I noticed something odd. In that same table you specify Subject AltName as a Prohibited extension. But then below that you specify Subject Alternative Name as Necessary.

I think this may be another bug?

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: