Hongkong Post: Delayed response to CPR
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: manho, Assigned: manho)
Details
(Whiteboard: [ca-compliance] [policy-failure])
Incident Report
This is a preliminary report.
Delayed response to a certificate problem report in a complete and/or timely manner.
Summary
Hongkong Post CA has received a problem report from the email address dickson.linting.experiment@gmail.com but failed to respond in a complete and/or timely manner, as it was treated as junk. This failure to comply is a violation of CABF BR #4.9.3 Procedure for revocation request: “The CA SHALL maintain a continuous 24x7 ability to accept and respond to revocation requests and Certificate Problem Reports.”.
Impact
The certificate problem report was verified as a fact. Delayed response to CPR failed the expectation of the reporter and the wider WebPKI community.
Timeline
All times are UTC+8.
The provided timeline focuses solely on the events related to the delayed response to CPR. For a comprehensive understanding of the original bug, please refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1886406 .
2024-03-04:
- 21:00 Email reporting the issue received in our obsoleted Certificate Problem Reporting email box, techsupp@hongkongpost.gov.hk . All emails directed to this address are automatically rerouted to a designated internal mailbox. Amongst the hundreds of legitimate technical support requests we receive daily, the email in question was inadvertently classified as junk.
2024-03-18:
- 21:01 Chrome Root Program sent email to the contact person of Hongkong Post CA with the original email included, pointing to the failure to respond to a certificate problem report in a complete and/or timely manner.
- 22:44 Hongkong Post CA’s contact person acknowledged receipt of the email, and immediately started the examination of the original incident with compliance team.
2024-03-19:
- 15:10 Compliance team confirmed the issue and started planning for remediation.
2024-03-20:
- 19:10 Determine to create a separate bug focusing on the delayed revocation of the affected certificates, and another bug focusing on the failure to respond to a certificate problem report in a complete and/or timely manner.
2024-03-21:
- 19:36 Posting this bug.
Root Cause Analysis
The email from the sender: dickson.linting.experiment@gmail.com was classified as junk, so the email was not read.
Lessons Learned
What went well
- Upon identifying the cause, the contact person from Hongkong Post CA promptly took immediate actions and responded to the issue without any further delay.
What didn't go well
- The technical support staff responsible for receiving the emails on the techsupp@hongkongpost.gov.hk did not verify the junk emails.
- The email box designated for reporting certificate problems is burdened with an extensive collection of junk emails, which are interspersed with the legitimate technical support requests.
Where we got lucky
- Upon thorough inspection of the techsupp@hongkongpost.gov.hk email box, it was determined that out of all the emails received, only this particular email turned out to be a genuine problem report. All other messages were unrelated to the reported issue.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Dedicate a new email address for certificate problem reporting. | Prevent | 2024-04-01 |
| Train the technical support staff regarding the Problem reporting mechanism | Prevent | 2024-04-01 |
Appendix
Details of affected certificates
Based on Incident Reporting Template v. 2.0
|
||
Updated•7 months ago
|
You may want to consider what other CAs do to not have the same problem.
Let’s Encrypt, last I checked, integrates with Zendesk as a CRM tool so these problems are less pronounced there. Google Trust Services uses a Google form based flow.
Could you please share what is your current email triage process? How often is it checked? Have there been any false negatives in the spam detection? Has there been any other false positives other than this one?
How far back did you look to see if any other legitimate CPR was missed?
(In reply to amir from comment #1)
You may want to consider what other CAs do to not have the same problem.
Let’s Encrypt, last I checked, integrates with Zendesk as a CRM tool so these problems are less pronounced there. Google Trust Services uses a Google form based flow.
Thank you for the information. Currently, we do not possess an advanced solution such as Zendesk or a web form to streamline the CPR process. However, we will investigate best practices implemented by other CAs for future improvement.
Could you please share what is your current email triage process? How often is it checked? Have there been any false negatives in the spam detection? Has there been any other false positives other than this one?
Our technical support staff consistently monitors emails during work hours and prioritizes them based on urgency, flagging important emails for follow-up, provided the email is not spam. One area for improvement is the verification of junk mail, which has not been adequately addressed.
How far back did you look to see if any other legitimate CPR was missed?
We did look for any other legitimate CPR in the past 24 months, but we have not identified any instances where one was missed.
2024-03-26
- 16:00 (UTC+8) Submitted a case to update the Certificate Problem Mechanism to the designated email address cpr@eCert.gov.hk . This mailbox is dedicated to managing certificate problem reports, and not a general technical support mailbox. Our compliance team will directly monitor emails received in this email address.
This serves as the final report.
Incident Report
Delayed response to a certificate problem report in a complete and/or timely manner.
Summary
Hongkong Post CA has received a problem report from the email address dickson.linting.experiment@gmail.com but failed to respond in a complete and/or timely manner, as it was treated as junk. This failure to comply is a violation of CABF BR #4.9.3 Procedure for revocation request: “The CA SHALL maintain a continuous 24x7 ability to accept and respond to revocation requests and Certificate Problem Reports.”.
Impact
The certificate problem report was verified as a fact. Delayed response to CPR failed the expectation of the reporter and the wider WebPKI community.
Timeline
All times are UTC+8.
The provided timeline focuses solely on the events related to the delayed response to CPR. For a comprehensive understanding of the original bug, please refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1886406 .
2024-03-04:
- 21:00 Email reporting the issue received in our obsoleted Certificate Problem Reporting email box, techsupp@hongkongpost.gov.hk . All emails directed to this address are automatically rerouted to a designated internal mailbox. Amongst the hundreds of legitimate technical support requests we receive daily, the email in question was inadvertently classified as junk.
2024-03-18: - 21:01 Chrome Root Program sent email to the contact person of Hongkong Post CA with the original email included, pointing to the failure to respond to a certificate problem report in a complete and/or timely manner.
- 22:44 Hongkong Post CA’s contact person acknowledged receipt of the email, and immediately started the examination of the original incident with compliance team.
2024-03-19: - 15:10 Compliance team confirmed the issue and started planning for remediation.
2024-03-20:
- 19:10 Determined to create a separate bug focusing on the delayed revocation of the affected certificates, and another bug focusing on the failure to respond to a certificate problem report in a complete and/or timely manner.
2024-03-21:
- 19:30 Posted this bug.
2024-03-26:
- 09:00 Open a new case to CCADB in order to update new email address for reporting certificate problems.
2024-04-04:
- 15:15 Confirmed update in CCADB.
Root Cause Analysis
The email from the sender: dickson.linting.experiment@gmail.com was classified as junk, so the email was not read.
Lessons Learned
What went well
- Upon identifying the cause, the contact person from Hongkong Post CA promptly took immediate actions and responded to the issue without any further delay.
What didn't go well
- The technical support staff responsible for receiving the emails on the techsupp@hongkongpost.gov.hk did not verify the junk emails.
- The email box designated for reporting certificate problems is burdened with an extensive collection of junk emails, which are interspersed with the legitimate technical support requests.
Where we got lucky
- Upon thorough inspection of the techsupp@hongkongpost.gov.hk email box, it was determined that out of all the emails received, only this particular email turned out to be a genuine problem report. All other messages were unrelated to the reported issue.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Dedicate a new email address for reporting certificate problem. | Prevent | DONE |
| Provide training to the technical support staff on the problem reporting mechanism, including regular verification of junk mail, and assign compliance team members to oversee the monitoring of the newly created email box. | Prevent | DONE |
Appendix
Details of affected certificates
|
|
||
Comment 5•1 month ago
|
||
Dear Man Ho,
Have the remediation items been working successfully or as anticipated? Have junk-mail filters been refined to work better?
Thanks,
Ben
Hi Ben, I confirm that the current junk-mail filters are functioning more effectively. Furthermore, we have rechecked for any valid CPR instances in the last 6 months, and we have not found any cases where one was overlooked. There are currently no additional action items related to this incident. Kindly request the closure of this bug.
|
|
||
Comment 7•1 month ago
|
||
I'll close this tomorrow, 28-Aug-2024, unless there are questions or issues that still need to be discussed.
|
|
||
Updated•1 month ago
|
Description
•