2032063 - Hongkong Post: Certificates with invalid embedded SCT signature

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Man Ho]

Preliminary Incident Report

Summary

• CA Owner CCADB unique ID: A000030

• Incident description:
  We received a certificate problem report email on 13 April 2026 about a certificate with an invalid embedded SCT signatures. Following an initial investigation, it was noted that the subscriber had previously reported a problem when using this certificate. At that time, it was determined that the problem was likely caused by a CT log (Let's Encrypt Sycamore 2027h1) that had been added during the recent update to the eligible CT log list, rather than by the certificate issuance system itself. To address the issue, the CT log was temporarily removed from the certificate issuance system. Subsequently, the certificate was re-issued to the subscriber, and the original certificate was revoked.

https://crt.sh/?id=23070144686

We have re-initiated the investigation and a full incident report is currently in progress.

• Relevant policies:
  TLS BR Section 7.1.2.11.3: Signed Certificate Timestamp List

• Source of incident disclosure:
  Notified via certificate problem reporting.

Comment 1

[Man Ho]

Attachment: affected_certificate_list_20260422_v2.csv

During the investigation, three additional certificates with the issue of invalid SCT were identified. All affected certificates were revoked. A list of all affected certificates is attached. The investigation is continuing. We shall submit a full incident report as soon as possible.

Comment 2

[Man Ho]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000030

• Incident description:
  We received a certificate problem report email on 13 April 2026 about a certificate with an invalid embedded SCT signature. Following an initial investigation, it was noted that the subscriber had previously reported a problem when using this certificate. At that time, it was determined that the problem was likely caused by a CT log (Let's Encrypt Sycamore 2027h1) that had been added during the recent update to the eligible CT log list on 27 November 2025, rather than by the certificate issuance system itself. To address the issue, the CT log was temporarily removed from the certificate issuance system. Subsequently, the certificate was re-issued to the subscriber, and the original certificate was revoked.

  Following further investigation, it was noted that on 27 November 2025, an update to the eligible CT log list was applied to specific OV and EV certificate profiles to include CT log servers. Subsequently, the issuance software failed to properly process the SCT data returned by one CT log (Let's Encrypt Sycamore 2027h1). Specifically, the API returned the SCT extensions data as a Base64-encoded string. The issuance software failed to Base64-decode this string into its binary representation. Instead, it embedded the raw Base64 string into the certificate's SCT extensions field. This resulted in the issuance of certificates containing malformed SCT extensions that did not match the SCT signatures returned from the CT log and non-compliance with RFC 6962.

• Timeline summary:

  • Non-compliance start date: 3 Dec 2025
  • Non-compliance identified date: 13 April 2026
  • Non-compliance end date: 22 April 2026

• Relevant policies:
  TLS BR Section 7.1.2.11.3: Signed Certificate Timestamp List

• Source of incident disclosure:
  Notified via certificate problem reporting

Impact

• Total number of certificates: 4
• Total number of "remaining valid" certificates: 0
• Affected certificate types: OV certificates
• Incident heuristic: (3) the full corpus of affected certificates are disclosed in the Appendix
• Was issuance stopped in response to this incident, and why or why not?:
  Certificate issuance was not stopped, as the issue was resolved by removing the CT log (Let's Encrypt Sycamore 2027h1) from the issuance system before a stop became necessary.

Timeline

All times are UTC+8.
2025-11-27 19:30 - An update to the eligible CT log list was applied to the certificate issuance system for the OV and EV certificate profiles.
2025-12-12 17:16 - A subscriber reports a problem when using the certificate https://crt.sh/?id=23070144686
2025-12-15 12:26 - An initial investigation indicated that the problem was likely caused by a CT log (Let's Encrypt Sycamore 2027h1) that had been added during the recent update to the eligible CT log list on 27 November 2025, rather than by the certificate issuance system itself.
2025-12-15 13:00 - Remove the CT log (Let's Encrypt Sycamore 2027h1) from the certificate issuance system.
2025-12-18 13:58 - The certificate was revoked and a new certificate re-issued to the subscriber. The case was closed.
2026-04-13 01:34 - Received a certificate problem report email regarding that certificate that has an invalid SCT.
2026-04-13 17:43 - Re-initiate investigation on that certificate.
2026-04-14 11:30 - Confirmed the CT log (Let's Encrypt Sycamore 2027h1) was not in use. All other CT log in use were functioning properly. Certificate issuance remains normal, as the issue was resolved by removing the CT log (Let's Encrypt Sycamore 2027h1) from the issuance system.
2026-04-14 14:43 - Submitted a support ticket to the certificate issuance software vendor.
2026-04-15 08:02 - The vendor responded with explanation of mechanism implemented by the software.
2026-04-15 18:16 - Provided additional information to the vendor and requested further investigation of the problem
2026-04-15 22:21 - Submitted the Preliminary Incident Report
2026-04-20 12:30 - Completed scanning and examination of all unexpired certificates and identified that three affected certificates were issued before the removal of the CT log (Let's Encrypt Sycamore 2027h1). Two have been revoked and one was still valid.
2026-04-20 14:18 – Informed the affected customer.
2026-04-22 11:46 – Revoked the last affected certificate.
2026-04-23 12:38 - The vendor confirmed the SCT extension values returned are put into the certificate without additional processing.

Related Incidents

Bug	Date	Description
https://bugzilla.mozilla.org/show_bug.cgi?id=2006711	2025-12-17	Encoding non-conformity in SCT extensions.

Root Cause Analysis

Contributing Factor #1: Certificate Issuance Software Error

• Description:
  Hongkong Post CA utilizes a certificate issuance software provided by third-party vendor. Although the certificate issuance software was featured to support CT log in compliance with RFC 6962, it did not account for the possibility of SCT extensions containing values or the necessity of decoding the extensions into binary bytes before embedding it.
  Consequently, the issuance software inserted the Base64-encoded string directly into the certificate. This created a mismatch between the data signed by the CT log and the data present in the certificate, resulting in invalid SCT signatures.

• Timeline:
  2025-11-27 19:30 The error occurred in the production environment when updating the eligible CT log list of the certificate issuance system.

• Detection:
  The error was detected after receiving the report via certificate problem reporting.

• Interaction with other factors:
  This software error created the malformed data. However, it would not have resulted in a mis-issuance incident if the contributing factor #2 had successfully identified the defect before updating the eligible CT log list in the certificate issuance system.

• Root Cause Analysis methodology used:
  5 Whys

Contributing Factor #2: Insufficient Testing when updating the eligible CT log list

• Description:
  The test cases included checks for the successful receipt of CT log responses but did not consider SCT signature validation.

• Timeline:
  2025-11-27 19:30 An update to the eligible CT log list was applied to the certificate issuance system for the OV and EV certificate profiles, but test scenarios did not cover SCT signature validation.

• Detection:
  The insufficiency of the test cases was confirmed during the post‑incident review of this incident.

• Interaction with other factors:
  This factor allowed the Certificate Issuance Software Error to remain undetected, resulting in the CT log (Let's Encrypt Sycamore 2027h1) being included in the eligible CT log list of the certificate issuance system.

• Root Cause Analysis methodology used:
  5 Whys

Lessons Learned

• What went well:
  • All 4 affected certificates were revoked.
  • The investigation identified a clear and reproducible root cause, enabling a targeted fix.
• What didn’t go well:
  • The issue was not detected internally and relied on certificate problem reporting.
  • Test scenarios specifically covering SCT signature validation during the update of the eligible CT log list in the certificate issuance system were insufficient.
• Where we got lucky:
  • The issue was identified before a larger volume of certificates was issued under the same conditions. Among the four affected certificates, three were revoked upon subscriber confirmation. One was successfully revoked within 5 days of awareness, ensuring full compliance with the timeline mandated by BR 4.9.1.1.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke all affected certificates	Correct	N/A (Immediate Mitigation)	100% revocation of 4 certificates	2025-04-22	Completed
Include testing on SCT signature validation during next update of eligible CT log list	Prevent	Root Cause #2	No error found by CT log linting	To be confirmed	Ongoing
Upgrade the certificate issuance software to support SCTs containing values in the extensions field	Prevent	Root Cause #1	No error found by CT log linting	To be confirmed	Ongoing
Enhance certificate problem investigation to include CT log linting whenever a user report is received	Detect	What didn’t go well: The issue was not detected internally	CT log linting tool deployed and responsible staff adequately trained	2026-04-30	Ongoing

Appendix

See the attached file https://bugzilla.mozilla.org/attachment.cgi?id=9572425

Comment 3

[Man Ho]

Action Items Update

We use “pkimetal” as a comprehensive tool for performing CT log linting, and the responsible staff have been trained in its use. The ongoing items require further clarification from the software vendor, and we will provide a progress update in the next report.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke all affected certificates	Correct	N/A (Immediate Mitigation)	100% revocation of 4 certificates	2025-04-22	Completed
Include testing on SCT signature validation during next update of eligible CT log list	Prevent	Root Cause #2	No error found by CT log linting	To be confirmed	Ongoing
Upgrade the certificate issuance software to support SCTs containing values in the extensions field	Prevent	Root Cause #1	No error found by CT log linting	To be confirmed	Ongoing
Enhance certificate problem investigation to include CT log linting whenever a user report is received	Detect	What didn’t go well: The issue was not detected internally	CT log linting tool deployed and responsible staff adequately trained	2026-04-30	Completed

Comment 4

[Man Ho]

Weekly Update:
The software vendor is still investigating, and it is anticipated that an update will be available by next week. We will report further progress on these items once additional information becomes available.

Comment 5

[Man Ho]

Weekly Update:
The software vendor has completed its investigation and is preparing a patch to address the issue. It is anticipated that the patch will be available by next week. We will confirm the target completion date for the ongoing items after the patch becomes available.

Comment 6

[Man Ho]

Weekly Update:
Having conducted further analysis on the system impact, the software vendor is working to prepare the software patch to address the issue. We will confirm the target completion date for the ongoing items after the patch becomes available.

Comment 7

[Man Ho]

Weekly Update:
The software vendor has provided a patch to support SCTs containing values in the extensions field. We are currently testing the patch to confirm that the issue has been resolved. Upon successful completion of testing, we will deploy to production environment ASAP. The target completion date for the ongoing items will be confirmed after testing.

Comment 8

[Man Ho]

Weekly Update:
We have successfully tested the patch provided by the software vendor. We will deploy it to production environment on or before 13 June 2026. After the deployment, we will review and re-test our CT log list to verify the changes and complete the ongoing items.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke all affected certificates	Correct	N/A (Immediate Mitigation)	100% revocation of 4 certificates	2025-04-22	Completed
Include testing on SCT signature validation during next update of eligible CT log list	Prevent	Root Cause #2	No error found by CT log linting	To be confirmed	Ongoing
Upgrade the certificate issuance software to support SCTs containing values in the extensions field	Prevent	Root Cause #1	No error found by CT log linting	2026-06-13	Ongoing
Enhance certificate problem investigation to include CT log linting whenever a user report is received	Detect	What didn’t go well: The issue was not detected internally	CT log linting tool deployed and responsible staff adequately trained	2026-04-30	Completed

Comment 9

[Man Ho]

Weekly Update:
We have deployed the patch to production environment and completed the review and testing on SCT signature validation across the latest updated CT log list. The upgraded certificate issuance software can now support SCTs containing values in the extensions field.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke all affected certificates	Correct	N/A (Immediate Mitigation)	100% revocation of 4 certificates	2025-04-22	Completed
Include testing on SCT signature validation during next update of eligible CT log list	Prevent	Root Cause #2	No error found by CT log linting	2026-06-13	Completed
Upgrade the certificate issuance software to support SCTs containing values in the extensions field	Prevent	Root Cause #1	No error found by CT log linting	2026-06-13	Completed
Enhance certificate problem investigation to include CT log linting whenever a user report is received	Detect	What didn’t go well: The issue was not detected internally	CT log linting tool deployed and responsible staff adequately trained	2026-04-30	Completed

Comment 10

[Man Ho]

Hi everyone,
All the action items have been completed. If no additional questions are received by 26 June, we will proceed to submit the closure report.

Comment 11

[Man Ho]

Report Closure Summary

• Incident description:
  Four OV TLS certificates with invalid embedded SCT signatures were issued between 3 and 12 December 2025. The issuance system embedded Base64‑encoded SCT data from a CT log (Let’s Encrypt Sycamore 2027h1) directly into the certificate’s SCT extension instead of decoding it to binary, causing malformed SCT extensions and non‑compliance with RFC 6962 and TLS BR Section 7.1.2.11.3. The issue was first reported via a certificate problem report on 13 April 2026. All four affected certificates were revoked by 22 April 2026.

• Incident Root Cause(s):
  #1 Certificate Issuance Software Error: SCT extension values returned by a CT log were inserted as Base64-encoded string rather than being decoded into binary bytes, producing invalid SCT signatures.
  #2 Insufficient Testing when updating the eligible CT log list: test scenarios verified SCT retrieval but did not validate SCT signature, allowing the defect to reach production after the CT log list update on 27 November 2025.

• Remediation description:
  #1 Immediate mitigation: Temporarily removed the Let’s Encrypt Sycamore 2027h1 CT log from the eligible CT log list as an interim measure. Revoked all affected certificates without delay upon identification of the issue (completed by 22 April 2026).
  #2 Certificate problem investigation process improvements: Deployed CT log linting (“pkimetal”) and trained responsible staff (completed by 30 April 2026).
  #3 Preventive fixes: Worked with the software vendor to correct SCT handling; patch successfully tested and deployed to production by 13 June 2026. Updated procedures to include SCT signature validation testing whenever the eligible CT log list is changed (completed by 13 June 2026).

• Commitment summary:
  The certificate issuance software has been patched in the production environment to resolve the problem. Hongkong Post will continue to monitor for the latest Certificate Transparency (CT) standards and maintain regular testing and validation of CT logs as part of the routine change management process.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 12

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-07-06.