Hongkong Post: TLS certificates with basicConstraints not marked as critical
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: manho, Assigned: manho)
Details
(Whiteboard: [ca-compliance] [ov-misissuance])
Attachments
(1 file)
|
15.08 KB,
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
|
Details |
Incident Report
This is a preliminary report.
Summary
During our investigation of the bug at https://bugzilla.mozilla.org/show_bug.cgi?id=1886406 , we were notified that some TLS certificates were issued with the basicConstraints extension present, but without the critical flag set. This is a violation of BR 7.1.2.7.6.
As per the adoption of Ballot SC62 on 22 April 2023, the Subscriber Certificate Extensions have been defined that basicConstraints MAY be present, and if included, they MUST be set as critical. This requirement has been effective since 15 September 2023.
Impact
We’re still investigating the list of affected certificates.
Timeline
All times are UTC+8.
2023-09-15:
- 08:00 BR for TLS 2.0.0 has become effective.
2024-03-20:
- 20:41 As part of the bug reported at https://bugzilla.mozilla.org/show_bug.cgi?id=1886406, we are notified about TLS certificates issued with the basicConstraint extension present, but without the critical flag set.
2024-03-21:
- 08:58 We have been made aware of this error and started an examination of the matter with the compliance team.
2024-03-22:
- 13:30 The problem seems to stem from a system bug in the certificate issuance system, along with an older version of zlint that failed to identify this error. The investigation is currently underway.
Root Cause Analysis
To be provided.
Lessons Learned
What went well
What didn't go well
Where we got lucky
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Example | Prevent | 2038-01-19 |
Appendix
Details of affected certificates
Based on Incident Reporting Template v. 2.0
|
||
Updated•1 year ago
|
Incident Report
Summary
During our investigation of the bug at https://bugzilla.mozilla.org/show_bug.cgi?id=1886406 , we were notified that some TLS certificates were issued with the basicConstraints extension present, but without the critical flag set. This is a violation of BR 7.1.2.7.6.
Following the adoption of Ballot SC62, the Subscriber Certificate Extensions have been defined that basicConstraints MAY be present, and if included, they MUST be set as critical. This requirement has been effective since 2023-09-15.
Impact
46 TLS certificates were issued since 2023-09-15 with basicConstraints extension present but without the critical flag set. They are a subset of the affected TLS certificates as stated in the original bug report.
The affected TLS certificates were mainly issued to government bureaus or departments in Hong Kong SAR. It is important to note that there have been no actual disruptions to their websites or online services due to the presence of the basicConstraints extension. We will collaborate closely with our affected customers to facilitate the replacement of their certificates, and follow the outlined plan in the original bug report to revoke all affected certificates.
Timeline
All times are UTC+8.
2023-09-15:
- 08:00 BR for TLS 2.0.0 has become effective.
2024-03-20:
- 15:30 Already stopped issuance due to the bug reported at https://bugzilla.mozilla.org/show_bug.cgi?id=1886406.
- 20:41 As part of that bug, we are notified about TLS certificates issued with the basicConstraints extension present, but without the critical flag set.
2024-03-21:
- 08:58 We have been made aware of this error and started examination of the matter with compliance team.
2024-03-22:
- 13:30 The problem seems to stem from a system bug in the certificate issuance system, along with an older version of zlint that failed to identify this error.
- 18:00 The cause of the problem has been identified. Work on developing a patch for the certificate issuance system to incorporate the latest version (3.6.1) of zlint.
- 18:54 A total of 46 affected TLS certificates has been identified, all of which are linked to 48 CT logs uncovered in crt.sh. Out of these logs, 46 are pre-certificate logs and 2 are final certificate logs.
2024-03-25:
- 19:53 The system patch has been successfully implemented in the production system. Any Certificate Signing Request (CSR) that contains a basicConstraints extension failing the zlint linting will be rejected.
- 20:12 Resumed the issuance of TLS certificates to our customers, allowing them to receive new TLS certificates from our platform.
Root Cause Analysis
The basicConstraints extension has been set as critical in the certificate issuance system since establishment of Hongkong Post CA in the year 2000. The criticality of this extension is expected to be mandatory, but the certificate issuance system allow the Certificate Signing Request provided by customer to override the criticality flag of the issued certificate. This system bug has been reported to the certificate issuance system vendor to follow up.
Despite having zlint deployed in our system to perform pre-issuance linting, we were unable to detect the error because the testing and implementation of the latest zlint version (3.6.0), which was released on 7 January 2024 with the inclusion of this new linting, was underway in our systems.
Lessons Learned
What went well
- N/A
What didn't go well
- Due to a bug in our certificate issuance system, the criticality flag of the basicConstraints extension was inadvertently overridden by the CSR provided by the customer.
- The update cycle for linting tools occurs more frequently compared to the update cycle of our certificate issuance system.
Where we got lucky
- There have been no actual disruptions to websites or online services of our customers due to the presence of basicConstraints extension.
- We have conducted pkilint linting on every TLS certificate that has been issued, and we have not come across any new errors in the process.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Continue to revoke the certificates that have been delayed in revocation. | Mitigate | 2024-05-20 |
| Upgrade the zlint to version (3.6.1) in the production environment. | Prevent | DONE |
| Conduct pkilint linting on all the issued TLS certificates to identify any potential additional errors. | Detect | DONE |
Appendix
Details of affected certificates
See in attachment.
Based on Incident Reporting Template v. 2.0
1 affected certificate was revoked without delay.
2024-03-20:
- 19:00 (UTC+8) https://crt.sh/?id=12442627980
This is to share an interim update regarding the ongoing revocation process of the affected TLS certificates.
2024-04-26:
- 23:00 In the process of re-issuing TLS certificates, a total of 43 new certificates have been provided to our customers. Consequently, there are currently 3 outstanding certificates remaining, which are either awaiting the customers' generation of a Certificate Signing Request (CSR) or have already confirmed as no longer required. With confirmation from our customers, we have already revoked 26 TLS certificates that were affected. It is anticipated that this number will continue to increase as their certificates are replaced.
2024-05-03:
- 18:15 The re-issuance of TLS certificates is underway. We have additionally re-issued 2 new certificates, bringing the total number of new certificates provided to our customers to 45. Additionally, 1 certificate has been verified as no longer required and has been revoked. With confirmation from our customers, we have already revoked 41 TLS certificates that were affected.
2024-05-10
- 20:45 With confirmation from our customers, we have already revoked 44 certificates (accounting for 95.7% of the affected certificates), and 2 certificates (4.3%) are pending confirmation from customers regarding the successful replacement of their affected certificates.
2024-05-17
- 20:30 All affected certificates were revoked.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Continue to revoke the certificates that have been delayed in revocation. | Mitigate | DONE |
| Upgrade the zlint to version (3.6.1) in the production environment. | Prevent | DONE |
| Conduct pkilint linting on all the issued TLS certificates to identify any potential additional errors. | Detect | DONE |
We identified the following new action items that aim to prevent reoccurrence of this incident.
| Action Item | Kind | Due Date |
|---|---|---|
| 1. Upgrade zlint to the latest version (3.6.2). | Prevent | 2024-05-31 |
| 2. Include “pkilint” as a pre-issuance linting tool in the certificate issuance process. Both zlint and pkilint will be used in parallel. Every certificate must pass both linting before issuance of certificate. | Prevent | 2024-06-07 |
| 3. Plan for upgrade of the certificate issuance system to the latest version that enforce all mandated configuration to the certificate, including the basicConstraint extension. | Prevent | 2024-06-30 |
|
|
||
Updated•1 year ago
|
2024-05-30
- 19:00 The “pkilint” has been included as a pre-issuance linting tool in the certificate issuance process. An upgrade to version 3.6.2 has been made for the existing "zlint" tool. Both zlint and pkilint will be used in parallel. Every certificate must pass both linting before issuance of certificate.
Here below a status update of the action items:
| Action Item | Kind | Due Date |
|---|---|---|
| 1. Upgrade zlint to the latest version (3.6.2). | Prevent | DONE |
| 2. Include “pkilint” as a pre-issuance linting tool in the certificate issuance process. Both zlint and pkilint will be used in parallel. Every certificate must pass both linting before issuance of certificate. | Prevent | DONE |
| 3. Plan for upgrade of the certificate issuance system to the latest version that enforce all mandated configuration to the certificate, including the basicConstraint extension.. | Prevent | 2024-06-30 |
|
Assignee | |
Comment 10•1 year ago
|
||
Here below a status update of the outstanding action items:
| Action Item | Kind | Due Date |
|---|---|---|
| 3. Plan for upgrade of the certificate issuance system to the latest version that enforce all mandated configuration to the certificate, including the basicConstraint extension. | Prevent | DONE |
|
|
||
Updated•1 year ago
|
|
|
Assignee | |
Comment 11•1 year ago
|
||
While we are going through the action items in the original bug 1886406, we continue monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 12•1 year ago
|
||
I would like to drop a note to say that we continue monitoring this bug for further comments or questions. We will follow up within this bug itself.
|
|
Assignee | |
Comment 13•1 year ago
|
||
I would like to drop a note to say that we continue monitoring this bug for further comments or questions.
|
|
||
Comment 14•1 year ago
|
||
What other action items exist? Otherwise, please request that I close this bug with a "Need Info" / "Request information from triage owner".
|
|
Assignee | |
Comment 15•1 year ago
|
||
We have finalized all action items outlined in this bug report and will ensure that these actions are consistently maintained. There are no further action items related to this incident. Kindly request the closure of this bug.
|
|
||
Comment 16•1 year ago
|
||
I will close this bug tomorrow, Wed., 28-Aug-2024, unless there are issues or items still to discuss.
|
|
||
Updated•1 year ago
|
Description
•