Closed Bug 1804843 Opened 1 year ago Closed 11 months ago

Hongkong Post: Subject CN converted to Unicode representation incident

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: manho, Assigned: manho)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(1 file)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36

Steps to reproduce:

  1. How your CA first became aware of the problem.
    We became aware of the problem when we received a report by email at 2022-12-07 17:01 HKT that one of the TLS certificates issued had got zlint error.

  2. A timeline of the actions your CA took in response.
    2022-12-07 17:12 HKT : Start investigating the problem
    2022-12-07 17:56 HKT : Identified the problem related to the subject CN that contained the subscriber’s domain name in Chinese characters encoded as ISO/IEC 10646, but not encoded as P-Labels.
    2022-12-07 18:55 HKT : Provided initial feedback to the inquirer that we were following up the remediation.
    2022-12-08 09:53 HKT : Searched for similar cases in the certificate issuance system. There were a total 8 certificates with the same problem and no pending certificate for issuance.
    2022-12-08 16:19 HKT : Confirmed the problem as an incident of HKPost CA system and triggered an incident reporting process.
    2022-12-08 17:00 HKT : Developed and tested a fix to the problem and began the system change procedure for production.
    2022-12-08 17:26 HKT : Informed the subscribers of the 8 certificates that we were going to re-issue the certificates.
    2022-12-08 18:44 HKT : Provided an update to the inquirer that we would fix the problem and re-issue the 8 certificates to the concerned subscribers.
    2022-12-09 14:17 HKT : Deployed the fix to the problem in production and started re-issuing the 8 certificates to subscribers.

  3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.
    Every application for TLS certificate required manual approval by our CA officers before applicant could submit CSR to our certificate issuance system to generate TLS server certificate. Since we became aware of the problem, our CA officers had stopped approval of certificate applications with Chinese domain name.

  4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.)
    There are 8 TLS server certificates involved in the problem. The earliest certificate was issued on 2022/01/11 and the latest certificate was issued on 2022/12/06.

  5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates.
    The concerned TLS server certificates are:
    https://crt.sh/?id=6661012732&opt=zlint
    https://crt.sh/?id=7891749303&opt=zlint
    https://crt.sh/?id=6653853146&opt=zlint
    https://crt.sh/?id=6653957416&opt=zlint
    https://crt.sh/?id=6653933701&opt=zlint
    https://crt.sh/?id=5955615622&opt=zlint
    https://crt.sh/?id=6653890037&opt=zlint
    https://crt.sh/?id=8134988253&opt=zlint

  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    Our TLS server certificate supports the subscriber’s website using Chinese FQDN. The Chinese FQDN was encoded as P-Labels in the subjectAltName extension. However, there was a system bug that the Chinese FQDN was encoded as ISO/IEC 10646 but not the same as the subjectAltName. The bug was not detected as no subscribers had reported any issues on their websites with Chinese FQDN. On the other hand, due to the limitation of our certificate issuance system that does not support any external linting programs such as zlint, we currently conduct post-issuance linting process manually, although we see the effectiveness and efficiency of post-issuance linting need to be improved.

  7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
    We had immediately notified the concerned subscribers of the 8 certificates and arranged to re-issue their certificates for replacement. The subscribers were reminded to replace their certificates immediately without delay. We will revoke the concerned TLS certificates when the subscribers have replaced their certificates.
    By the way, we are looking at some measures to automate the post-issuance linting process before delivery of our TLS server certificates with Chinese FQDN in order to prevent the problem from happening again. We will share more information on the timeline of the remediation steps as soon as possible.

Assignee: nobody → manho
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Summary: Subject CN converted to Unicode representation incident → Hongkong Post: Subject CN converted to Unicode representation incident

A quick update on timeline of our actions:

2022-12-13 10:22 HKT : All concerned TLS certificates were revoked.

In my next update soon, I'll explain our plan to improve our linting process before the delivery of our TLS server certificates with Chinese FQDN to subscribers.

Whiteboard: [ca-compliance] [ov-misissuance]

At the moment, our certificate issuance system does not support any external linting programs such as zlint before issuing TLS certificates. Apart from the existing manual checking, our engineer is studying and working on an enhancement that can retrieve the pre-certificate from the certificate issuance system. The pre-certificate is then automatically taken to zlint linting process before sending to CT logs.

The enhancement is planned for completion of testing by end of February 2023. I shall update again on the progress of our plan.

After testing our pre-certificate linting process successfully, we have already implemented this enhancement to production environment. Thank you for bringing up this bug for our attention. If you have any suggestions, we're happy to listen.

I will close this on or about Wed. 19-Apr-2023 if there are no other comments or questions.

Flags: needinfo?(bwilson)

Having checked our linting process log for TLS certificates issued so far, the linting process runs smoothly without finding any new error or exception.

Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: