1886665 - Hongkong Post: Delayed revocation of TLS certificates with Certificate Policies extension problem

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Man Ho]

Incident Report

This is a preliminary report.

Summary

Hongkong Post CA has issued a total of 1,176 certificates that are affected by the problem with Certificate Policies extension as described in the bug report at https://bugzilla.mozilla.org/show_bug.cgi?id=1886406.

All certificates should have been revoked within 5 days when CA is made aware of the incident. As described in the above mentioned bug, we have been working with affected customers to replace their certificates, but not all of them could replace their certificates within the given period. Since the continuity of those customers' critical infrastructures depends on the affected certificates, not all affected certificates were revoked in time.

The affected TLS certificates form part of the security infrastructure of the government of Hong Kong SAR. They covered over 70% of government websites and online services in Hong Kong, mainly serving the entire local population of over 8 million people. Given the wide spectrum of essential community e-services being offered by diversified subscribers as our major customers, such as Education, Medical and Health, Transport, Social Services, Public Order and Inland Revenue, there are significant concerns from them and therefore difficulties in coordinating the completion for revocation of the affected TLS certificates within the given period of 5 days. If these certificates are revoked without proper coordination and assured completion of replacement TLS certificates by the subscriber organisations, it could lead to substantial cumulative impacts on the sustainable delivery of critical e-services by our government subscribers.

While our TLS certificates are mainly targeted for subscription by local customers for e-services provision to serve local community per se, we are fully aware of the expectations from the overall WebPKI community regarding the prompt revocation of the affected TLS certificates, as demonstrated by some CA owners. In order to minimize the impact on the broader web, we take full responsibility for collaborating closely with our affected customers to facilitate and prioritize the replacement of their certificates.

We expect to complete the revocation of all affected certificates no later than 2024-05-20, subject to no special concern or constraint from any individual subscribers.

Impact

1,170 out of the 1,176 affected certificates were not revoked in time, i.e. these certificates are subject of this bug.

Timeline

All times are UTC+8.
The provided timeline focuses solely on the events related to the delayed revocation of the affected TLS certificates. For a comprehensive understanding of the original bug, please refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1886406.

2024-03-18:

• 21:01 The original problem report was received.
• 22:44 Hongkong Post CA’s contact person acknowledged receipt of the report, and started examining the matter with compliance team.

2024-03-19:

• 15:10 Compliance team confirmed the issue and started planning for remediation.
• 15:24 Replied to Chrome Root Program, confirming that we would submit this incident report in Bugzilla to deliberate the timeline of the incident, root cause analysis, lesson learnt and next actions.

2024-03-20:

• 16:00 The issue was confirmed as a mis-issuance. All affected certificates were identified as at 15:30.
• 19:10 The total list of affected TLS certificates was attached in the original bug.
• 20:10 Determine to create a separate bug focusing on the delayed revocation of the affected certificates, and another bug focusing on the failure to respond to a certificate problem report in a complete and/or timely manner.

2024-03-21:

• 12:30 Posting this preliminary report.

Root Cause Analysis

In response to this certificate problem, our top priority is to promptly reach out to our major customers, followed by the other affected customers, to discuss the potential impact on their websites and online services. We clearly confirm and it is important to note that there have been and will be no actual disruptions to websites or online services adopting our TLS certificates due to the certificate policies extension. However, all affected customers have expressed grave and genuine concerns if the revocation of the impacted TLS certificates to be conducted within the given period of 5 days.

The main reason is that all affected TLS certificates are being managed manually by the customers themselves. This process typically involves multiple personnel, including the authorized representative who handles the certificate application, the manager who grants authorization, and the system administrator, especially for government websites and online services. These individuals must go through several manual procedures in order to install the TLS certificates on their systems. It is important to recognize that certificate replacement, even though under urgent requests as to resolve the current incident, must take ample time for the customers concerned to complete through manual process.

Furthermore, it is important to note that our major customers, which include government bureaus and departments, do not utilize a unified solution for managing and deploying their TLS certificates. Consequently, a mass revocation of certificates as being called for would significantly take time in ensuring each individual customer for completion of installing the new TLS certificates on their respective servers. Revoking impacted certificates without proper coordination and assured completion of replacement TLS certificates could have a significant cumulative impact on government websites and online services.

Lessons Learned

What went well

• Major customers are willing to cooperate.

What didn't go well

• The affected TLS certificates are being utilized by our major customers in critical applications. Implementing emergency processes to replace these certificates may introduce higher risks, potentially impacting the stability and security of their applications.
• It is challenging to explain to some customers the reasons for the urgent revocation and replacement of certificates, as well as the differences in appearance between the certificates before and after replacement.

Where we got lucky

• N/A.

Action Items

Action Item	Kind	Due Date
Continue to revoke the certificates that have been delayed in revocation.	Mitigate	2024-05-20
Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons.	Prevent	2024-05-20

Appendix

Details of affected certificates

Based on Incident Reporting Template v. 2.0

Comment 1

[Ryan Dickson]

This incident report has gone "stale" and violates the expectations described on CCADB.org.

Comment 2

[Man Ho]

Thanks for bringing this to my attention. I'm sorry for the unintentional oversight in not providing an update here while addressing the original bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1886406, on 2024-04-26. We have already notified all affected customers about the incident and it is imperative that their certificates must be replaced quickly. The revocation process for the affected certificates is currently underway.

Comment 3

[Ryan Dickson]

@Man Ho, can you please share a status update related to how revocation is progressing (minimally describing the number of certificates that have not yet been revoked in accordance with the TLS BRs)?

Comment 4

[Man Ho]

Sure, the original bug report included a status update on the ongoing revocation process for the affected TLS certificates, and that update is provided again here for easy reference.

2024-04-26:

• 22:40 Upon thorough examination of the 1,176 CT logs found in crt.sh, it has been determined that a total of 1,111 distinct TLS certificates must be revoked. This determination takes into consideration the removal of duplicated pre-certificate logs and final certificate logs. In the process of re-issuing TLS certificates, a total of 1,038 new certificates have been provided to our customers. Consequently, there are currently 73 outstanding certificates remaining, which are either awaiting the customers' generation of a Certificate Signing Request (CSR) or have already confirmed as no longer required. With confirmation from our customers, we have already revoked 372 TLS certificates that were affected. It is anticipated that this number will continue to increase as their certificates are replaced.

  It’s worth noting that out of the 1,111 distinct TLS certificates, 6 of them were actually revoked within 5 days when we were made aware of the incident.

2024-05-03:

• 18:15 The re-issuance of TLS certificates is underway. We have additionally re-issued 32 new certificates, bringing the total number of new certificates provided to our customers to 1,070. Consequently, there are still 41 outstanding certificates that need to be addressed. Out of these, 23 are awaiting either the customer’s generation of a Certificate Signing Request (CSR) or the confirmation that they are no longer required, while 18 certificates that have already been confirmed as no longer required. With confirmation from our customers, we have already revoked 841 TLS certificates that were affected.

Comment 5

[Man Ho]

2024-05-10

• 20:45 With confirmation from our customers, we have already revoked 950 certificates (accounting for 85.5% of the affected certificates), and we are currently in the process of revoking 92 certificates (8.3%), and 64 certificates (5.8%) are pending confirmation from customers regarding the successful replacement of their affected certificates. Consequently, there are still 5 outstanding certificates (0.4%) that await the customer's generation of a Certificate Signing Request (CSR) or confirmation that they are no longer required.

Comment 6

[Man Ho]

2024-05-17

• 22:20 We have already revoked 1,079 certificates (accounting for 97.2% of the affected certificates), and 32 certificates (2.8%) are pending confirmation from customers regarding the successful replacement of their affected certificates.

The progress we have made so far would not have been possible without the cooperation of all the affected customers, who have been urged to replace their certificates as a matter of urgency. Unfortunately, some of our major customers, particularly those who provide essential community e-services like healthcare, banking, and electronic payment to the local population of over 8 million people, are still encountering various difficulties and problems in the process of replacing their certificates. As a result, they have expressed serious concerns about the potential revocation of the affected certificates before the replacement TLS certificates are fully implemented by the customer.

After conducting further investigation, we have identified several difficulties and problems that are hindering the certificate replacement process. These include:

1. Some certificates being mostly adopted in the financial industry are pinned within the system interface to their banks and replacing the certificates requires additional manual process. Customers have indicated that the process from testing to production cannot be completed by the specified deadline.
2. Some major customers have critical ongoing business operations and are unable to adhere to the specified deadline for shutting down and replacing certificates (e.g. Our customer is currently providing 24/7 public healthcare services to the entire local population of over 8 million people).
3. The process of replacing certificates involves engaging external vendors and their availability significantly impacts work schedules. As a result, some customers need to ask for an extension on the revocation deadline.
4. Various obstacles that have impeded our ability to maintain communication with the affected customers (e.g. a recent personnel change of the customer representatives, which has resulted in a disruption in the responsible party for handling this matter).

Considering the potential negative impacts on the government's ability to provide critical e-services to the public, we propose extending the revocation deadline from 2024-05-20 to 2024-05-31. We believe this additional two-week period will allow for a smoother transition. We value the community's input and would appreciate any feedback regarding this revised deadline.

Comment 7

[amir]

How would have these entities handled this revocation case if this was an undeniably serious issues such as key compromise? Would you still delay revocation then? If you didn’t, would that mean the entity providing healthcare of 8 million people would just stop being able to do so?

Comment 8

[Man Ho]

Attachment: [e-mail]

Comment 9

[Wayne]

I would ask that a serious question be raised with your subscribers: is a public root is necessary for their operations? Given the sensitive nature of their operations would a private root suit them better for their use-case? If they wish to use a public root then they must change their business practices to make revocation in a timely manner possible.

Comment 10

[Man Ho]

(In reply to amir from comment #7)

How would have these entities handled this revocation case if this was an undeniably serious issues such as key compromise? Would you still delay revocation then? If you didn’t, would that mean the entity providing healthcare of 8 million people would just stop being able to do so?

The revocation process has been delayed due to the large number of certificates involved and the diverse range of customers affected. The healthcare service provider, for instance, has implemented nearly 100 certificates across multiple systems, which requires careful planning and scheduling for their replacement. However, in cases where there is clear evidence of an undeniably serious issue which poses an unbearable risk, such as a compromised key supported by specific details in the CSR or the private key itself, our system will expeditiously revoke the certificate without any delay.

Comment 11

[Man Ho]

(In reply to Wayne from comment #9)

I would ask that a serious question be raised with your subscribers: is a public root is necessary for their operations? Given the sensitive nature of their operations would a private root suit them better for their use-case? If they wish to use a public root then they must change their business practices to make revocation in a timely manner possible.

We have already communicated to our major customers about the significance of promptly replacing their certificates, as HKPost is fully committed to complying with the baseline requirement in cases where certificate revocation is necessary within either 24 hours or 5 days. This commitment stems from the fact that HKPost's existing root certificate is a public root. We have also provided our customers with information about new technology, like ACME, that can automate the replacement and renewal process for certificates in the future. I'm certain that they will explore the implementation of such technology, especially for the parts that interact with the general public.

Comment 12

[amir]

our system will expeditiously revoke the certificate without any delay.

So in that case you'd also still make it impossible for the healthcare to continue operating its services to the population, right?

What I'm trying to get across here is:

If you'd do the revocation in that condition, you should also do it in this condition. Otherwise, neither you as the CA, nor them as the subscriber get the "practice" of handling situations that require mass revocation.

I appreciate the work Hongkong Post is doing here, but I do worry that you will use the same justification the next time a revocation event rolls around.

Also, beyond that, I think it's irresponsible to put services like this behind public PKI.

Comment 13

[Man Ho]

(In reply to amir from comment #12)

What I'm trying to get across here is:

If you'd do the revocation in that condition, you should also do it in this condition. Otherwise, neither you as the CA, nor them as the subscriber get the "practice" of handling situations that require mass revocation.

Taking this as a valuable lesson, I acknowledge that it is crucial for both our major customers and us as a public CA to tackle the difficulties associated with mass revocations. To address this, we have begun strategizing the adoption of new technologies such as ACME, that can automate the replacement and renewal process for certificates in the future. Furthermore, we have provided our major customers with information about ACME for their exploration and planning. I believe that they will seriously consider implementing this technology, especially for the parts that interact with the general public. Moving forward, we are actively working towards putting this plan into “practice” as quickly as possible.

I appreciate the work Hongkong Post is doing here, but I do worry that you will use the same justification the next time a revocation event rolls around.

I am grateful for your understanding and concern. I want to emphasize that the purpose of proposing an extension to the revocation deadline is not meant to justify future revocation events. Our main goal remains the same - to establish trust within the WebPKI community and web browsers, and to gain the trust of our customers. In the event of a mass revocation incident, I do believe it is crucial that we take on the responsibility to promptly mitigate the impact, learn from other CAs and improve ourselves to prevent recurrence, and increase transparency throughout the entire process. This includes ensuring that our customers understand the baseline requirements for revoking non-compliant certificates within 5 days, and ensuring that the WebPKI community comprehends the specific impact of the revocation on the local web users in Hong Kong, particularly, if these certificates are revoked without proper coordination and the customer's replacement certificates are not assuredly completed.

Comment 14

[amir]

I want to emphasize that the purpose of proposing an extension to the revocation deadline is not meant to justify future revocation events.

Can you then commit to not delaying revocation in the future? Something that says from this point on Hongkong Post will not delay revocation when an incident requires it?

The issue is that you've not committed to that. Your action items do not really address making sure you won't delay revocation in the future.

Our main goal remains the same - to establish trust within the WebPKI community and web browsers, and to gain the trust of our customers.

By not committing to follow the BRs in the future, I think you're going to fail the first goal.

By reading all of this, I'm like - mostly certain that if some incident were to happen again that needed certificate revocation within 120 hours, you would push back the revocation timeline for that too. I do not see anything in this incident that assures me that won't happen.

Comment 15

[Man Ho]

2024-05-20

• 21:00 We have additionally revoked 11 certificates, bringing the total number of revoked certificates to 1,090 (accounting for 98.1% of the affected certificates), and 21 certificates (1.9%) are pending confirmation from customers regarding the successful replacement of their affected certificates.

We are committed to follow the Mozilla’s revocation guidelines (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation) in handling the revocation of the certificates. To this end, we have performed an analysis on factors that prevented timely revocation of the certificates and work out the following action items that aim to prevent reoccurrence of this incident (New action items 3 - 5) and future revocation delays (New action items 6 - 7).

Compliance with the BR and root program policies is of paramount importance to us. We genuinely value the community's input and would appreciate any feedback on these action items.

Action Items

Action Item	Kind	Due Date
Update
1. Continue to revoke the certificates that have been delayed in revocation, and complete by the proposed revocation deadline.	Mitigate	2024-05-31
2. Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons.	Prevent	DONE
New
3. Upgrade zlint to the latest version.	Prevent	2024-05-31
4. Include “pkilint” as a pre-issuance linting tool in the certificate issuance process. Both zlint and pkilint will be used in parallel. Every certificate must pass both linting before issuance of certificate.	Prevent	2024-06-07
5. Strengthen the team for regular monitoring of any updates to BR, establish a two-persons cross-checking procedure for individual configuration items against current system.	Prevent	2024-06-07
6. Educate our customers, offer training regarding the certificate revocation requirement and facilitate our customers for certificate replacement process.	Prevent	2024-06-30
7. Request our customers to provide secondary contact information for timely communication.	Prevent	2024-06-30

Comment 16

[Man Ho]

Attachment: [e-mail]

Comment 17

[Wayne]

I'm seeing all certificates are now revoked with the last 5 occurring on 2024-05-29.

I am not seeing a response to comment 14.

Comment 18

[Man Ho]

2024-05-29

• 14:15 We have already revoked all affected certificates.

2024-05-30

• 19:00 The “pkilint” has been included as a pre-issuance linting tool in the certificate issuance process. An upgrade to version 3.6.2 has been made for the existing "zlint" tool. Both zlint and pkilint will be used in parallel. Every certificate must pass both linting before issuance of certificate.

Here below a status update of the action items:

Action Item	Kind	Due Date
1. Continue to revoke the certificates that have been delayed in revocation.	Mitigate	DONE
2. Ensure customers are aware of possible situations where certificates must be replaced quickly for security reasons.	Prevent	DONE
3. Upgrade zlint to the latest version	Prevent	DONE
4. Include “pkilint” as a pre-issuance linting tool in the certificate issuance process. Both zlint and pkilint will be used in parallel. Every certificate must pass both linting before issuance of certificate.	Prevent	DONE
5. Strengthen the team for regular monitoring of any updates to BR, establish a two-persons cross-checking approach for individual configuration items against current system.	Prevent	2024-06-07
6. Educate our customers, offer training regarding the certificate revocation requirement and facilitate our customers for certificate replacement process.	Prevent	2024-06-30
7. Request our customers to provide secondary contact information for timely communication.	Prevent	2024-06-30

I'm seeing all certificates are now revoked with the last 5 occurring on 2024-05-29.

I am not seeing a response to comment 14.

We're currently going through the action items. I will strive to provide our responses to comment 14 as soon as possible, by next update date 2024-06-07.

Comment 19

[Man Ho]

Here below a status update of the outstanding action items.

Action Item	Kind	Due Date
5. Strengthen the team for regular monitoring of any updates to BR, establish a two-persons cross-checking approach for individual configuration items against current system.	Prevent	DONE
6. Educate our customers, offer training regarding the certificate revocation requirement and facilitate our customers for certificate replacement process.	Prevent	2024-06-30
7. Request our customers to provide secondary contact information for timely communication.	Prevent	2024-06-30

We keep going through the action items. I will strive to provide the latest status of the outstanding action items as soon as possible, by next update date 2024-06-17.

Comment 20

[Man Ho]

(In reply to amir from comment #14)

Can you then commit to not delaying revocation in the future? Something that says from this point on Hongkong Post will not delay revocation when an incident requires it?

I'm sorry for the late reply. Taking this as a valuable lesson, we have carefully reviewed several action items, including action item #6, aiming to minimize the time it takes for subscribers to replace certificates. We will educate the subscribers, enhance their understanding of immediate revocation requirements, and facilitate them in preparing for a swift certificate replacement process, to ensure that the certificates can be replaced within the revocation deadline in case needed. Additionally, subscribers will also be advised to consider utilizing private PKI, and prepare other contingency plans for enforced certificate revocation to minimize disruptions to their e-services. With the implementation of these actions, we are confident that we will be able to fulfil the BRs of timely revocation going forward.

Comment 21

[amir]

With the implementation of these actions, we are confident that we will be able to fulfil the BRs of timely revocation going forward.

I'm not confident in that at all. What your subscribers do, or not do, is irrelevant to what you will do as a CA.

Do you commit that if you are required to revoke certificates in the future, you will do it within the deadlines provided by the BRs?

Comment 22

[Man Ho]

(In reply to amir from comment #21)

Do you commit that if you are required to revoke certificates in the future, you will do it within the deadlines provided by the BRs?

As mentioned in my comment #11, we are fully committed to complying with the baseline requirement in cases where certificate revocation is necessary within either 24 hours or 5 days, the deadlines provided by the BRs. The action items #6 and #7 are intended to raise awareness among subscribers about the need to be prepared for these cases, but not to undermine our commitment.

We have been making progress on the action items #6 and #7. I will strive to promptly share the status of our outreach to subscribers, aiming to provide an update no later than 2024-06-30.

Comment 23

[amir]

Thanks for the answer!

It's great to hear that Hongkong Post will in the future, revoke certificates on a timeline manner without waiting for confirmation from subscribers. I am also glad to hear that in parallel to this, efforts will be made to educate subscribers of what will happen during these events.

I don't have any further questions here.

Comment 24

[Tim Callan]

(In reply to Man Ho from comment #22)

As mentioned in my comment #11, we are fully committed to complying with the baseline requirement in cases where certificate revocation is necessary within either 24 hours or 5 days, the deadlines provided by the BRs. The action items #6 and #7 are intended to raise awareness among subscribers about the need to be prepared for these cases, but not to undermine our commitment.

This is good to see. I know you made a statement to this effect in comment 11, but then later in comment 15 you appeared to be backing away from this commitment when you said,

We are committed to follow the Mozilla’s revocation guidelines (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation) in handling the revocation of the certificates.

This was further aggravated by the fact that the root cause analysis in comment 0 did not acknowledge that meeting revocation deadlines is ultimately the CA’s responsibility and not the Subscriber’s and by the lack of action items addressing this unstated root cause.

I am very pleased to see the statement quoted above as an unambiguous promise to the community not to repeat the willful failure to revoke on time as we see here and in bug 1887888. As this commitment will remain in the record and future backsliding would look very bad for Hongkong Post, it is encouraging to see such a clear commitment, which I hope to see you keep for the duration of your tenure as a public CA.

There are two more things needed to clean this matter up:

1. As this is a general commitment, it should also apply to bug 1887888. I will post a separate message there suggesting you confirm this commitment on that bug.
2. I see no action items on either incident that address the willful decision not to revoke on time. My suggestion is to add an action item by which you will establish a specific, written policy stating that your CA will not deliberately choose not to revoke on time, share specifics of this policy on this thread such as its wording and where it is published, and educate all present and future compliance employees on that policy.

Comment 25

[Man Ho]

(In reply to Tim Callan from comment #24)

1. As this is a general commitment, it should also apply to bug 1887888. I will post a separate message there suggesting you confirm this commitment on that bug.

Thanks for your suggestion. I have repeated our commitment on that bug.

2. I see no action items on either incident that address the willful decision not to revoke on time. My suggestion is to add an action item by which you will establish a specific, written policy stating that your CA will not deliberately choose not to revoke on time, share specifics of this policy on this thread such as its wording and where it is published, and educate all present and future compliance employees on that policy.

To align the action items for this incident and Bug 1887888, I have added new action items #8 and #9 below, copied from the action item #5 and #6 of the Bug 1887888 respectively.

Action Item	Kind	Due Date
New
8. Update the CA operation procedure for (i) the swift certificate replacement and (ii) the enforced revocation to ensure adherence to BR revocation requirement.	Prevent	2024-06-30
9. Incorporating this BR requirement into our risk management plan to effectively manage crises resulting from enforced revocation that could potentially cause significant harm to critical infrastructure or essential e-services.	Prevent	2024-06-30

As part of our action item #8 (ii), the update of CA operation procedure for enforced revocation to ensure adherence to BR revocation requirement has been introduced following the analysis of two incidents involving delayed revocation (bug 1886665 and bug 1887888). At Hongkong Post CA, our officers are required to strictly follow the operational procedure as stated in our policy, and they will receive regular training to ensure their compliance with the procedure.

Comment 26

[Man Ho]

2024-06-28:

• 15:42 Educational materials regarding the BR revocation requirements and the request for secondary contact information for timely communication have been distributed to all customers, including new customers as well as those affected and unaffected by this incident.

Additionally, we have also prepared new CA operation procedures for (i) the swift certificate replacement and (ii) the enforced revocation to ensure adherence to BR revocation requirement stated in action item #8. At Hongkong Post CA, our officers are required to strictly follow these operation procedures, and they will receive regular training to ensure their compliance with these procedures. Furthermore, our risk management plan has been updated to incorporate the BR revocation requirement mentioned in action item #9, enabling us to effectively manage crises resulting from enforced revocation.

Here below a status update of the outstanding action items.

Action Item	Kind	Due Date
6. Educate our customers, offer training regarding the certificate revocation requirement and facilitate our customers for certificate replacement process.	Prevent	DONE
7. Request our customers to provide secondary contact information for timely communication.	Prevent	DONE
8. Update the CA operation procedure for (i) the swift certificate replacement and (ii) the enforced revocation to ensure adherence to BR revocation requirement.	Prevent	DONE
9. Incorporating this BR requirement into our risk management plan to effectively manage crises resulting from enforced revocation that could potentially cause significant harm to critical infrastructure or essential e-services.	Prevent	DONE

By the way, we keep monitoring this bug for further comments or questions. We will follow up within this bug itself.

Comment 27

[Man Ho]

I would like to drop a note to say that we continue monitoring this bug for further comments or questions. We will follow up within this bug itself.

Comment 28

[Man Ho]

Acknowledged that this bug has been marked for next update 2024-10-31. We continue monitoring this bug for further comments or questions. We will follow up here.

Comment 29

[Ben Wilson]

We continue work on incident-reporting and compliance requirements aimed at reducing delayed revocation, so this bug will remain open until at least February 1, 2025. Meanwhile, CAs should review https://github.com/mozilla/www.ccadb.org/pull/186.

Comment 30

[Ben Wilson]

Before closing this incident, Hong Kong Post should repeat its commitment to revoke TLS certificates timely in accordance with section 4.9.1 of the TLS Baseline Requirements.

Mozilla acknowledges that many of Hong Kong Post's subscribers operate under complex regulatory or bureaucratic constraints. Still, Hong Kong Post will need to provide additional Action Items aimed at handling these government-managed and other subscribers (Education, Medical and Health, Transport, Social Services, Public Order and Inland Revenue) and ensuring that no external policies prevent timely revocation. Examples include: requiring these entities to provide written confirmation that they can comply with revocation timelines before issuance; ensuring that they have plans for replacing certificates within 24 hours of a misissuance or security incident; and streamlined approval processes so that TLS certificates can be replaced without problematic bureaucratic approval chains.

Finally, we will need a completed Closure Summary.

Thanks,

Ben

Comment 31

[Man Ho]

Through the experience of dealing with mass revocation, we have gained insights on how to handle such situations better. We would like to reiterate our firm commitment adhering to the baseline requirement for timely certificate revocation in accordance with section 4.9.1 of the TLS Baseline Requirements.

We wish to provide an update on the follow-up actions that were implemented last year and have since been established as part of our application procedure to prevent any external policies from obstructing timely revocation. The enumerated actions are outlined below:

Action Item	Kind	Due Date
10. All applicants, including government-related entities (Education, Medical and Health, Transport, Social Services, Public Order and Inland Revenue) , must sign the application form. The form states that HKPCA will revoke the TLS certificate within 24 hours or 5 days under the conditions stipulated in section 4.9.1 of the CPS, in alignment with section 4.9.1 of the TLS Baseline Requirements. This act signifies their agreement to the timely revocation of the TLS certificate as mandated by section 4.9.1 of the TLS Baseline Requirements.	Prevent	2024-06-30 DONE
11. A reminder on the effective management of TLS certificates was sent to all government-related entities last year, ensuring that they have plans for replacing certificates within 24 hours of a misissuance or security incident and streamlining their approval processes as well. Receive confirmation of understanding from all the government-related entities on this TLS Baseline Requirement.	Prevent	2025-01-10 DONE

We will prepare a Closure Summary if there are no further questions.

Comment 32

[Man Ho]

Incident Report Closure Summary

Incident Description:

• Hongkong Post CA issued a total of 1,111 certificates affected by the problem with the Certificate Policies extension that does not assert http scheme, which should have been revoked within 5 days as per TLS BR. Despite efforts to collaborate with affected subscribers for certificate replacement, 1,105 certificates were not revoked in time. If these certificates are revoked without proper coordination and assured completion of replacement TLS certificates by the affected subscribers, it could lead to substantial cumulative impacts on the sustainable delivery of critical e-services by the government.

Incident Root Cause(s):

• The delay in revocation was primarily due to the manual management of the affected TLS certificates by subscribers, involving multiple personnel procedures. Additionally, the lack of a unified solution for managing and deploying TLS certificates among major subscribers, such as government bureaus and departments, led to difficulties in getting subscribers to replace the certificates within the set deadline.

Remediation Description:

• To address the delayed revocation of certificates, we have taken actions to streamlining the revocation process and ensuring subscribers are aware of the importance of promptly replacing certificates for security reasons. Action items have been implemented, including upgrading linting tools, streamlining the certificate revocation and replacement procedures, educating subscribers on revocation requirements, and establishing procedures for timely communication and training.

Commitment Summary:

• We are committed to comply with the baseline requirement for timely certificate revocation in accordance with section 4.9.1 of the TLS Baseline Requirements. We are also committed to preventing recurrence of similar incidents and ensuring compliance with industry standards of CCADB, web browsers and CA/Browser Forum to maintain trust within the WebPKI community.

Since all the action items mentioned in this Incident Report have been taken care of, we kindly ask for it to be closed.

Comment 33

[Ben Wilson]

I will close this on Friday, 28-Feb-2025, unless there are questions or issues to discuss.