2006711 - NAVER Cloud Trust Services: Encoding non-conformity in SCT extensions

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Hogeun Yoo]

Preliminary Incident Report

Summary

• Incident description:
  Incident description: A technical non-conformity was identified in the encoding of the Signed Certificate Timestamp (SCT) extension field. A logic error in the data transformation layer resulted in the extensions field being populated with its encoded string representation rather than its intended binary format.

• Issuance Status: 2025-12-17 17:43 UTC: Suspended all TLS certificate issuance to initiate remediation.

• Relevant policies:
  TLS BR Section 7.1.2.11.3: Specifies the required encoding format for the Signed Certificate Timestamp List.
  RFC 6962 Section 3.3: Defines the structural requirements and data types for SCT fields.

• Source of incident disclosure:
  2025-12-17 16:01 UTC: Received an inquiry from the Chrome Root Program regarding potential encoding non-conformities in specific certificates.

Comment 1

[sooyoung.eo]

We have identified the affected certificates, and revocation is scheduled to be performed in accordance with CABF BR Section 4.9.1.1. A complete list of the affected certificates will be provided in the full incident report.
Certificate issuance resumed at 2025-12-18 00:42 UTC.

Comment 2

[sooyoung.eo]

In accordance with our revocation plan, all affected certificates were successfully revoked within the timeframe of 08:00 to 11:00 on December 22, 2025 (UTC). A complete list of the revoked certificates will be provided in the upcoming full incident report.

Comment 3

[sooyoung.eo]

Attachment: affected_certificate_list_2006711.csv

Comment 4

[sooyoung.eo]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A005672

• Incident description:
  On December 11, 2025, a configuration change was applied to specific DV and OV certificate profiles to include CT log servers utilizing the Static CT API. It was subsequently identified that the issuance software failed to properly process the SCT data returned by these logs. Specifically, the API returned the SCT extensions data as a Base64-encoded string. The issuance system failed to Base64-decode this string into its binary representation. Instead, it embedded the raw Base64 string into the certificate's SCT extensions field. This resulted in the issuance of certificates containing malformed SCT extensions that did not match the binary data signed by the Log, leading to invalid SCT signatures and non-compliance with RFC 6962.

• Timeline summary:

  • Non-compliance start date: 2025-12-11 UTC
  • Non-compliance identified date: 2025-12-17 UTC
  • Non-compliance end date: 2025-12-22 UTC

• Relevant policies:

  • TLS Baseline Requirements - 7.1.2.11.3 Signed Certificate Timestamp List

• Source of incident disclosure: Third-party report via MDSP

Impact

• Total number of certificates: 82

• Total number of "remaining valid" certificates: 0

• Affected certificate types: 79 DV TLS server certificates / 3 OV TLS server certificates

• Incident heuristic:

  • Following the report on the Mozilla dev-security-policy mailing list and subsequent communication from the Google Chrome Root Program regarding the Static CT API, we immediately conducted an internal investigation.
  • On December 11, 2025, at 07:30 UTC, a configuration change was made to include CT log servers using the Static CT API for certain DV and OV certificate profiles. We confirmed that the SCT extensions received from these logs were not correctly encoded. Consequently, only certificates issued under these specific profiles following this configuration change were affected.
  • To address the issue, we suspended certificate issuance for the affected profiles on December 17, 2025, at 17:43 UTC. We then queried the CA database for all certificates issued under these specific profiles between the configuration change (December 11, 2025, at 07:30 UTC) and the time of suspension. We confirmed that 82 certificates were affected.
  • Following the remediation of the certificate profiles, Certificate issuance was resumed on December 18, 2025, at 00:42 UTC.

• Was issuance stopped in response to this incident, and why or why not?:
  To address the issue, we suspended certificate issuance for the affected profiles on December 17, 2025, at 17:43 UTC. Certificate issuance was resumed on December 18, 2025, at 00:42 UTC, after the remediation of the certificate profiles.

• Analysis:

  • The incident was caused by a failure in the issuance system to correctly parse and decode the SCT data structure returned by the Static CT API.
  • When the configuration was updated to include logs utilizing the Static CT API, the API response provided the extensions field as a Base64-encoded string.
  • The software logic responsible for embedding these SCTs into the certificate treated this Base64 string as raw data. It embedded the Base64 string directly into the SCT extensions field without performing the decoding.
  • Since the CT Log server generates the SCT signature based on the decoded binary data, but the certificate contained the undecoded ASCII text, the cryptographic digest of the SCT data calculated by clients did not match the signature. This discrepancy caused the SCT signature validation to fail and rendered the certificates non-compliant.

• Additional considerations:
  None. No additional impact types or different behaviors were observed beyond the scope described.

Timeline

All times are in UTC
2025-12-11 07:30 - Configuration change applied to include CT log servers using the Static CT API for certain DV and OV certificate profiles
2025-12-12 01:56 - Issue reported on MDSP
2025-12-17 16:01 - Received notice from Google Chrome Root Program regarding the MDSP report; Initial awareness and investigation initiated
2025-12-17 17:43 - Suspended all TLS certificate issuance to initiate remediation
2025-12-17 18:26 - Submitted the Preliminary Incident Report
2025-12-18 00:42 - Configuration remediation(Change target CT log server) completed; Certificate issuance resumed
2025-12-18 05:07 - Initial notification sent to affected customers
2025-12-22 06:20 - Final notification sent to affected customers
2025-12-22 08:00 to 11:00 - Completed the revocation of all 82 affected certificates

Related Incidents

Bug	Date	Description
https://bugzilla.mozilla.org/show_bug.cgi?id=2007132	2025-12-19	Certificates with invalid embedded SCT signature
https://bugzilla.mozilla.org/show_bug.cgi?id=2007070	2025-12-19	Non conformant SCT Encoding Due to SCT Modification
https://bugzilla.mozilla.org/show_bug.cgi?id=1852404	2023-09-09	Empty SCT extensions in certificates

Root Cause Analysis

Contributing Factor #1: Software Implementation Error

• Description:

  • Although the issuance system was built in compliance with RFC 6962, the initial design did not account for the possibility of SCT extensions containing values or the necessity of decoding processing. This oversight resulted in a flaw in the certificate generation logic regarding the handling of Base64-encoded fields from the Static CT API.
  • The system received the extensions field from the API as a Base64 string but lacked the logic to decode this string into binary bytes before embedding it.
  • Consequently, the system inserted the Base64-encoded string directly into the certificate. This created a mismatch between the data signed by the Log and the data present in the certificate, resulting in invalid SCT signatures.

• Timeline:
  This error was introduced into the production environment when the configuration change was applied on December 11, 2025, at 07:30 UTC.

• Detection:
  The error was not detected by internal systems. It was identified following a report on the dev-security-policy mailing list and subsequent communication from the Google Chrome Root Program on December 17, 2025.

• Interaction with other factors:
  This implementation error created the malformed data. However, it would not have resulted in a mis-issuance incident if the Insufficient Testing Scenarios or Limitations in Pre-issuance Linting had successfully identified the defect before issuance.

• Root Cause Analysis methodology used:
  5 Whys

Contributing Factor #2: Insufficient Testing Scenarios

• Description:
  The test scenarios included checks for the successful receipt and recording of SCT timestamps, but did not consider SCT signature validation.

• Timeline:
  The gaps in the testing scenarios existed during the development and QA phases prior to the deployment on December 11, 2025.

• Detection:
  The insufficiency of the test cases was confirmed during the post-incident review conducted on December 17, 2025, after the incident was reported.

• Interaction with other factors:
  This factor allowed the Software Implementation Error to pass through the QA process and be deployed to the production environment.

• Root Cause Analysis methodology used:
  5 Whys

Contributing Factor #3: Limitations in Certificate Linting

• Description:
  Our certificate linting tools did not include ctlint. Consequently, the issue was not detected during the linting phase and proceeded to issuance.

• Timeline:
  This limitation was present throughout the incident period, from December 11, 2025, at 07:30 UTC to December 17, 2025, at 17:43 UTC.

• Detection:
  This limitation was discovered during the post-incident investigation on December 17, 2025, when comparing the results of our existing linters against the pkimetal linter, which successfully flagged the issue.

• Interaction with other factors:
  This factor served as the final failed control. Had the linters detected the malformed extension caused by the Software Implementation Error, the issuance would have been blocked automatically despite the testing gaps.

• Root Cause Analysis methodology used:
  5 Whys

Lessons Learned

• What went well:

  • Rapid Response & Remediation: Upon receiving the report, the issue was identified, issuance was suspended, and the remediation was deployed within less than 24 hours.
  • Full Revocation: All 82 affected certificates were successfully revoked within 5 days of awareness, ensuring full compliance with the timeline mandated by BR 4.9.1.1.

• What didn’t go well:

  • Detection Latency: The issue was not detected internally and relied on external reports (MDSP), appearing in production for approximately 7 days.
  • Linting Failure: Existing certificate linting tool failed to flag the specific Base64 encoding error in the SCT extensions.
  • Testing Gap: Regression testing did not sufficiently validate the binary structure of extensions specifically for the Static CT API data path.

• Where we got lucky:

  • Low Volume: The configuration was applied to a limited subset of profiles, resulting in a manageable number of affected certificates (82).

• Additional:

  • This incident highlighted the necessity of diversifying linting tools (e.g., incorporating pkimetal) and the importance of deep-packet inspection for API responses that involve binary data encoding.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Update certificate profiles to remove Static CT API usage	Correct	N/A (Immediate Mitigation)	Profiles configured to use Non-static CT API only	2025-12-18	Completed
Revoke all affected certificates	Correct	N/A (Incident Response)	100% revocation of 82 certificates	2025-12-22	Completed
Implement correct SCT extensions encoding logic for Static CT API	Prevent	Root Cause #1 (Software Implementation Error)	SCT extensions contain properly decoded binary data when received from Static CT API logs	2026-01-31	Ongoing
Update regression test suite to include SCT signature validation	Prevent	Root Cause #2 (Insufficient Testing Scenarios)	Tests pass with valid SCT signatures	2026-01-31	Ongoing
Integrate pkimetal into certificate issuance linting	Detect	Root Cause #3 (Limitations in Certificate Linting)	Linter successfully flags invalid SCTs	2026-02-13	Ongoing

Appendix

Attached a CSV file listing all affected certificates.

Comment 5

[sooyoung.eo]

Weekly Update:

There are no changes to report at this time, and all action items are currently on track according to the plan.
We will continue to monitor the progress and provide an update next week.

Comment 6

[sooyoung.eo]

Weekly Update:

There are no changes to the incident status at this time. However, the development for Action Items related to Root Cause #1 and #3 has been completed and is currently in the testing phase. We anticipate reporting further progress on these items in next week's update.
We will continue to monitor the progress and provide the next update.

Comment 7

[sooyoung.eo]

Weekly Update:

There are no changes to the incident status at this time. Remediation for the Action Items related to Root Causes #1 and #3 has been implemented in the test environment. Following minor adjustments, we plan to deploy these changes to the production environment next week.
Accordingly, we expect to update the status of these Action Items in the next weekly report.

Comment 8

[sooyoung.eo]

Action Items Update

All remaining action items were successfully completed via the deployment concluded at approximately 06:40 UTC on January 29, 2026.
Below is the updated status of our Action Items:

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Update certificate profiles to remove Static CT API usage	Correct	N/A (Immediate Mitigation)	Profiles configured to use Non-static CT API only	2025-12-18	Completed
Revoke all affected certificates	Correct	N/A (Incident Response)	100% revocation of 82 certificates	2025-12-22	Completed
Implement correct SCT extensions encoding logic for Static CT API	Prevent	Root Cause #1 (Software Implementation Error)	SCT extensions contain properly decoded binary data when received from Static CT API logs	2026-01-31	Completed
Update regression test suite to include SCT signature validation	Prevent	Root Cause #2 (Insufficient Testing Scenarios)	Tests pass with valid SCT signatures	2026-01-31	Completed
Integrate pkimetal into certificate issuance linting	Detect	Root Cause #3 (Limitations in Certificate Linting)	Linter successfully flags invalid SCTs	2026-02-13	Completed

Comment 9

[sooyoung.eo]

Report Closure Summary

• Incident description:
  TLS server certificates issued via the Static CT API contained Signed Certificate Timestamps (SCTs) with malformed extensions. The issue arose from incorrect handling of binary data within SCT extensions during the certificate issuance process, resulting in SCTs that failed verification. Upon discovery, the usage of the Static CT API was immediately disabled in the certificate profiles.

• Incident Root Cause(s):
  #1 Software Implementation Error: The encoding logic for handling SCT extensions retrieved from Static CT API logs failed to correctly decode binary data, leading to malformed SCT structures in the final certificates.
  #2 Insufficient Testing Scenarios: The regression test suite lacked specific validation for SCT signatures and extension formats originating specifically from Static CT API sources.
  #3 Limitations in Certificate Linting: The linting tools integrated into the issuance pipeline at the time did not have rules capable of detecting these specific semantic errors within SCT extensions.

• Remediation description:

  • Usage of the Static CT API was disabled in certificate profiles on 2025-12-18 (UTC) to prevent further misissuance.
  • All 82 affected certificates were revoked by 2025-12-22 (UTC).
  • All Action Items are now complete as of the deployment concluded on 2026-01-29 (UTC):
    #1 Implement correct SCT extensions encoding logic for Static CT API — Completed.
    #2 Update regression test suite to include SCT signature validation — Completed.
    #3 Integrate pkimetal into certificate issuance linting — Completed.

• Commitment summary:
  We have incorporated the corrected encoding logic and established strict validation protocols for any future Static CT API integrations. To ensure ongoing compliance, we have mandated SCT signature validation in our regression test suites and integrated 'pkimetal' as a blocking pre-issuance linter. Additionally, we will continuously monitor evolving CT standards and linting rules to proactively address potential issues.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 10

[chrome-root-program]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-02-11.