1886406 - Hongkong Post: TLS certificates with Certificate Policies extension that does not assert http scheme

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[Man Ho]

Attachment: Affected TLS certificates.xlsx

Incident Report

This is a preliminary report.

Summary

Hongkong Post CA received a report concerning a potential issue that URIs included in policyQualifiers attribute of Certificate Policies extension does not comply with BR Section 7.1.2.7.9.
The policyQualifiers attribute has been a certificate profile setting that we have used since establishment of Hongkong Post CA in the year 2000. However, as per the adoption of Ballot SC62 on 22 April 2023, the certificate profile requirements in the BR have been updated. It is now NOT RECOMMENDED to use the policyQualifiers attribute. Alternatively, if it is present, it MUST only contain permitted policyQualifiers, as defined in BR Section 7.1.2.7.9. This requirement has been effective since 15 September 2023.
Unfortunately, the discrepancy in this attribute was not detected during our existing compliance checking process against the BR. However, please be assured that Hongkong Post CA is fully committed to adhering to the BR. As part of our plan, we will remove the policyQualifiers attribute from the Certificate Policies extension for non-EV TLS certificates. For EV TLS certificates, the Certificate Policies extension will contain the permitted policyQualifiers attribute.
We will work with affected customers to replace their certificates, and create another bug to address the delayed revocation of the affected TLS certificates.

Impact

1,176 TLS certificates were issued since 2023-09-15 with Certificate Policies extension that does not comply with BR Section 7.1.2.7.9.
The majority of these certificates are actually issued to government bureaus or departments in Hong Kong SAR. These affected certificates cover over 70% of government websites and online services in Hong Kong, mainly serving the entire local population of over 8 million people.
We have duly notified our major customers regarding the recommended change on certificate policies extension and engaged in discussions regarding its potential impact on their websites and online services. It’s worth noting that there have been no actual disruptions to websites or online services as a result of the certificate policies extension. However, there are significant concerns surrounding the possibility of the affected TLS certificates being revoked within a short timeframe, which could lead to a substantial cumulative impact on government websites and online services.
We are fully aware of the expectations from the WebPKI community regarding the prompt revocation of the affected TLS certificates, as demonstrated by some CA owners. In order to minimize the impact on the broader web, we will collaborate closely with our affected customers to facilitate the replacement of their certificates.
Starting from 22 March 2024, new issuance of our TLS certificates will comply with the updated requirements on policyQualifiers attribute.
6 mis-issued TLS certificates were revoked in time. We expect to complete the revocation of all affected certificates no later than 2024-05-20.

Timeline

All times are UTC+8.
2023-09-15:

• 08:00 BR for TLS 2.0.0 has become effective.
  2024-03-04:
• 21:00 Email reporting the issue received in our obsoleted Certificate Problem Reporting email box, [e-mail] .
  2024-03-18:
• 21:01 Chrome Root Program sent email to the contact person of Hongkong Post CA with the original email included, pointing to the Certificate Policies extension issue and the delayed revocation of the affected certificates, and the failure to respond to a certificate problem report in a complete and/or timely manner.
• 22:44 Hongkong Post CA’s contact person acknowledged receipt of the email, and start examine the matter with compliance team.
  2024-03-19:
• 15:10 Compliance team confirmed the issue and started planning for remediation.
• 15:30 Hongkong Post CA stopped approval of new TLS certificate application.
• 15:57 Replied to Chrome Root Program, confirming that we will submit this incident report in Bugzilla to deliberate the timeline of the incident, root cause analysis, lesson learnt and next actions.
  2024-03-20:
• 09:00 Prepare the related system and CPS change
• 15:30 Stopped issuing TLS certificate application upon Certificate Signing Request submitted from customer.
• 16:00 The issue is confirmed as a mis-issuance. All affected certificates identified as at 15:30.
• 19:00 Revoke 6 affected certificates as listed below
  https://crt.sh/?id=12442627980
  https://crt.sh/?id=12434590783
  https://crt.sh/?id=12434251054
  https://crt.sh/?id=12433819246
  https://crt.sh/?id=12431788321
  https://crt.sh/?id=12431445466
• 19:10 Determine to create a separate bug focusing on the delayed revocation of the affected certificates, and another bug focusing on the failure to respond to a certificate problem report in a complete and/or timely manner.
• 19:25 This first preliminary report is posted.

Root Cause Analysis

The introduction of TLS BR version 2.0.0 marked a complete overhaul of section 7, primarily aimed at providing clarifications and encoding canonicalizations. Although we carefully assessed the modifications introduced in TLS BR version 2.0.0, we unintentionally failed to address Certificate Policies extension issue by the deadline on 15 September 2023.
The method we have employed to detect these problems is linting, but unfortunately, we did not uncover this issue until now. This is mainly because our primary reliance was on zlint for pre-issuance linting, as well as crt.sh cablint & x509lint for post-issuance linting. No errors were reported up until this point in time.
Furthermore, we could have potentially identified this requirement earlier if we had been diligently monitoring compliance issues on Bugzilla.

Lessons Learned

What went well

• N/A

What didn't go well

• The required change to the certificate profile was undetected during review of BR update.
• The discrepancy remains undetected during self-assessment for BR compliance.

Where we got lucky

• There have been no actual disruptions to websites or online services of our customers as a result of the certificate policies extension.

Action Items

Action Item	Kind	Due Date
Allocate additional personnel to review the BR and verify the Hongkong Post CA certificate issuance process for double-checking.	Prevent	2024-04-01
Add pkilint to the certificate issuance process.	Detect

Appendix

Details of affected certificates

As attached in this bug.

Based on Incident Reporting Template v. 2.0

Comment 1

[Martijn Katerbarg]

Thank you for this report.

I could not help notice that for at least one of the certificates you mentioned in this incident report, the basicConstraint extension is present, but is not set to critical. This should be considered as an additional incident.

Since you mentioned you rely on zlint for pre-issuance linting: Given that zlint does report this as an error, could you please explain in that incident report why pre-issuance linting failed on this particular certificate?

Comment 2

[Man Ho]

(In reply to Martijn Katerbarg from comment #1)

Thank you for this report.

I could not help notice that for at least one of the certificates you mentioned in this incident report, the basicConstraint extension is present, but is not set to critical. This should be considered as an additional incident.

Since you mentioned you rely on zlint for pre-issuance linting: Given that zlint does report this as an error, could you please explain in that incident report why pre-issuance linting failed on this particular certificate?

Thank you for reporting this error, which caught me by surprise as well. I shall examine the problem with my team immediately and provide a response ASAP.

Comment 3

[amir]

Starting from 22 March 2024, new issuance of our TLS certificates will comply with the updated requirements on policyQualifiers attribute.

Have you stopped issuance until then, or are you currently still issuing certificates?

Comment 4

[Man Ho]

We have stopped issuance since 2024-03-20 15:30 (UTC+8) until we have successfully applied patch to our system. It is targeted on 22 March 2024.

Comment 5

[Man Ho]

We have successfully implemented a patch to our system today. However, as we continue to investigate the issue related to the basicConstraint extension error, we have decided to continue stopping issuance until our system is also patched to prevent the recurrence of this error. Then I'll update this bug again.

Comment 6

[Man Ho]

2024-03-25:

• 20:12 Resumed the issuance of TLS certificates to our customers, allowing them to receive new TLS certificates from our platform. This has been made possible by patching the certificate issuance system to prevent the recurrence of the basicConstraints extension error.

Comment 7

[Man Ho]

This is to share an interim update regarding the ongoing revocation process of the affected TLS certificates.

2024-04-26:

• 22:40 Upon thorough examination of the 1,176 CT logs found in crt.sh, it has been determined that a total of 1,111 distinct TLS certificates must be revoked. This determination takes into consideration the removal of duplicated pre-certificate logs and final certificate logs. In the process of re-issuing TLS certificates, a total of 1,038 new certificates have been provided to our customers. Consequently, there are currently 73 outstanding certificates remaining, which are either awaiting the customers' generation of a Certificate Signing Request (CSR) or have already confirmed as no longer required. With confirmation from our customers, we have already revoked 372 TLS certificates that were affected. It is anticipated that this number will continue to increase as their certificates are replaced.

Comment 8

[Man Ho]

2024-05-03:

• 18:15 The re-issuance of TLS certificates is underway. We have additionally re-issued 32 new certificates, bringing the total number of new certificates provided to our customers to 1,070. Consequently, there are still 41 outstanding certificates that need to be addressed. Out of these, 23 are awaiting either the customer’s generation of a Certificate Signing Request (CSR) or the confirmation that they are no longer required, while 18 certificates that have already been confirmed as no longer required. With confirmation from our customers, we have already revoked 841 TLS certificates that were affected.

Comment 9

[Man Ho]

2024-05-10

• 20:45 With confirmation from our customers, we have already revoked 950 certificates (accounting for 85.5% of the affected certificates), and we are currently in the process of revoking 92 certificates (8.3%), and 64 certificates (5.8%) are pending confirmation from customers regarding the successful replacement of their affected certificates. Consequently, there are still 5 outstanding certificates (0.4%) that await the customer's generation of a Certificate Signing Request (CSR) or confirmation that they are no longer required.

Comment 10

[Man Ho]

2024-05-17

• 22:20 We have already revoked 1,079 certificates (accounting for 97.2% of the affected certificates), and 32 certificates (2.8%) are pending confirmation from customers regarding the successful replacement of their affected certificates.

In light of several difficulties and problems encountered by our customers during the certificate replacement process, we propose extending the revocation deadline from 2024-05-20 to 2024-05-31. The details regarding this request can be found in the bug report at https://bugzilla.mozilla.org/show_bug.cgi?id=1886665. We value the community's input and would appreciate any feedback on the bug regarding this revised deadline.

Comment 11

[Man Ho]

In line with the delayed revocation bug report, this is to repeat the update here regarding the ongoing revocation process of the affected TLS certificates.

2024-05-20

• 21:00 We have additionally revoked 11 certificates, bringing the total number of revoked certificates to 1,090 (accounting for 98.1% of the affected certificates), and 21 certificates (1.9%) are pending confirmation from customers regarding the successful replacement of their affected certificates.

Comment 12

[Man Ho]

2024-05-29

• 14:15 We have already revoked all affected certificates.

2024-05-30

• 19:00 The “pkilint” has been included as a pre-issuance linting tool in the certificate issuance process. An upgrade to version 3.6.2 has been made for the existing "zlint" tool. Both zlint and pkilint will be used in parallel. Every certificate must pass both linting before issuance of certificate.

Here below a status update of the action items:

Action Item	Kind	Due Date
Allocate additional personnel to review the BR and verify the Hongkong Post CA certificate issuance process for double-checking.	Prevent	DONE
Add pkilint to the certificate issuance process.	Detect	DONE
Upgrade zlint to the latest version (3.6.2).	Detect	DONE

Comment 13

[Man Ho]

We continue monitoring this bug for further comments or questions.

Comment 14

[Man Ho]

I would like to drop a note to say that we continue monitoring this bug for further comments or questions. We will follow up within this bug itself.