1916392 - Chunghwa Telecom: TLS Certificates Contains two LocalityName Values in SubjectDN by GTLSCA

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Leo Fang]

Attachment: affectedCerts247.csv

Incident Report

TLS Certificates Contains two LocalityName Values in SubjectDN.

Summary

Since the Extension Key Usage field was found to be incorrect last time, the GTLSCA team will regularly conduct cross-checks on the certificate Profile and BR with CHT RootCA. During the regular inspection on the morning of September 2, 2024, we randomly checked and found that there were two localityName values ​​in the SubjectDN field of some certificates.

Impact

A total of 247 certificates are affected.

Root Cause

This matter goes back to the GCA era, the predecessor of GTLSCA. The DN issued to the user is automatically compiled by the program according to the user's location. If it is an agency or unit of the central government, there will only be one L, which is no problem. The classification of local government agencies or units is more detailed. If it is only a county or city, there will only be one L value. However, if it is a small area in a township, this type of two L will appear, expressed by two L. Counties, cities and towns.

TimeLine

All times are UTC+8.

2024-09-02

• 09:50 RootCA Cross-Check regular meeting.
• 10:18 Found a problem and notified GTLSCA to stop issuing certificates.
• 11:00 CrossCA meeting and check root cause.
• 11:41 Notify CA owners.
• 14:00 Inventory of impact scope.
• 15:32 Update certificate profile.(fixed problem)
• 15:40 Confirm that the problem has been corrected and certicicates issuance resumed.
• 17:52 CA re-issues 247 new certificates. (started)
• 18:18 CA re-issued 247 new certificates. (completed)
• 19:55 GTLSCA website update announcement.

2024-09-03

• 02:01 Send email to notify users to download and install new certificates
• 03:25 Notify external auditors.
• 17:35 bugzilla report.

2024-09-06

• All affected certificates should be revoked.

Lessons Learned

What went well

With the experience of revoking certificates in large batches in the past, we quickly formulated a revocation plan within a time limit, such as stopping the issuance of certificates, holding a meeting to confirm the problem, quickly assessing the scope of the impact, and then reporting according to the steps. Including notifying owners, subscribers, external auditors, issuing announcements, etc.

What didn't go well

N/A

Where we got lucky

This matter was discovered through self-inspection, which means that this implementation method is indeed effective, allowing more researchers to discuss the BR regulations together, and allowing us to regularly review whether CA has actually completed all aspects of the BR requirements. . It also allows all research colleagues to learn and grow together.

Action Items

Action Item	Kind	Due Date (UTC+8)	Status
Stop issuance and remove the user certificate profile that contains two localityNames in SubjectDN	Mitigate	2024-09-02	Completed
Scope of Impact Survey		2024-09-02	Completed
Consulting meeting with the Root CA team		2024-09-02	Completed
Multi-party discussion with Root CA team and CA owner		2024-09-02	Completed
Report to the supervisor, CA owner and auditor		2024-09-02, 2024-09-03	Completed
Reissue the certificate and issue an announcement of certificate replacement	Mitigate	2024-09-02	Completed
Contact subscribers to replace certificates	Mitigate	2024-09-03	Completed
Bugzilla report		2024-09-03	Completed
Follow the issue and respond to questions		2024-09-03	On-Schedule
Revoke all affected certificates		2024-09-06	On-Schedule
Increase collaboration and establish cross-checking processes with the Root CA team to assist with certificate profile review and BR compliance	Prevent	Keep Going	On-Schedule

Appendix

List of 247 affected certificates to be revoked.

Comment 1

[Leo Fang]

Update processing progress at 2024-09-06. All affected certificates are revoked, and we are keep paying attention to this bug.

Action Items

Action Item	Kind	Due Date (UTC+8)	Status
Revoke all affected certificates		2024-09-06	Completed
Follow the issue and respond to questions		Keep Going	On-Schedule
Increase collaboration and establish cross-checking processes with the Root CA team to assist with certificate profile review and BR compliance	Prevent	Keep Going	On-Schedule

Comment 2

[Leo Fang]

We are continuing to monitor this issue.

Comment 3

[Ryan Dickson]

Hi Leo,

Can you help us understand why pre/post issuance linting was not considered as an action item in either this bug or https://bugzilla.mozilla.org/show_bug.cgi?id=1887096?

The issues described in both incident reports could have been detected, and possibly prevented, if linting tools like pkilint, zlint, or pkimetal were in use.

Thanks,
Ryan

Comment 4

[Leo Fang]

(In reply to Ryan Dickson from comment #3)
Hi Ryan,

When we mentioned "cross-check", we also included the use of zlint, although we did not explicitly mention it at the time.

Additionally, we would like to clarify that in the prior incident, Extended Key Usage setting incident, zlint was also utilized. However, the version of zlint used at that time did not detect the related issue.

Thank you.

Leo

Comment 5

[Tsung-Min Kuo]

(In reply to Ryan Dickson from comment #3)

Hi Leo,

Can you help us understand why pre/post issuance linting was not considered as an action item in either this bug or https://bugzilla.mozilla.org/show_bug.cgi?id=1887096?

The issues described in both incident reports could have been detected, and possibly prevented, if linting tools like pkilint, zlint, or pkimetal were in use.

Thanks,
Ryan

Hi Ryan,

This incident was found by the Root CA team when we were conducting regular cross-check on Sep. 02, and we found that there may cause a two LocalityName Values situation in SubjectDN in some rare case. GTLSCA use zlint as linting tool, where its newest version v3.6.3 is implemented around Aug. 23 in GTLSCA system and it does not detect this bug for most cases.

Comment 6

[Leo Fang]

We are continuing to monitor this issue.

Comment 7

[Leo Fang]

We are continuing to monitor this issue.

Comment 8

[Leo Fang]

We are continuing to monitor this issue.

Comment 9

[Leo Fang]

We are continuing to monitor this issue.

Comment 10

[Leo Fang]

We are continuing to monitor this issue.

Comment 11

[Leo Fang]

We are continuing to monitor this issue.

Comment 12

[Leo Fang]

We are continuing to monitor this issue.

Comment 13

[Leo Fang]

For this bug, all action items had been completed. There are no further action items related to this bug. We kindly request that it be closed.

Comment 14

[Ben Wilson]

Greetings,
Even though this has not yet been officially formalized as a bug-closure requirement, could you please provide a closing summary?
Thanks,
Ben

A closing summary should briefly:

• describe the incident, its root cause(s), and remediation;
• summarize any ongoing commitments made in response to the incident; and
• attest that all Action Items have been completed.

Here is a markdown template if needed:

Incident Report Closure Summary

• Incident Description: [Two or three sentences summarizing the incident.]
• Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
• Remediation Description: [Two or three sentences summarizing the incident's remediation.]
• Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 15

[Leo Fang]

OK, since the holiday has just ended, we will put the closure summary up in the next 2 days. Thank you for your reminder.

Comment 16

[Leo Fang]

Incident Report Closure Summary

• Incident Description:
  During the cross-check meeting on 2024/9/2, we checked the profile formats of some certificates and found that there were 2 localityNames in the SubjectDN field of some certificates.

• Incident Root Cause(s):
  Tracing back to GTLSCA’s predecessor, GCA, when applying for TLS certificates on small unit websites under local governments, the program would use two Ls to indicate the two levels of local names (counties, cities, and towns).

• Remediation Description:
  These measures include stopping the issuance of certificates, holding an emergency meeting to confirm the root cause, deleting certificate format settings containing more than 2 Ls, investigating the scope of impact and reissuing certificates, notifying users to replace certificates, and revoking all incorrectly issued certificates.

• Commitment Summary:
  We will hold regular TLS certificate profile format cross-check meetings, corresponding to the last item of the action items, currently once a quarter.

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 17

[Ben Wilson]

I will close this on or about Friday 7-Feb-2025.