1969296 - GoDaddy: Certificates with invalid embedded SCT signatures

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Steven Deitte]

Steps to reproduce:

Preliminary Incident Report - Certificate with an invalid embedded SCT

Summary

• Incident description: 2 subscriber certificates contained an invalid embedded SCT signatures. Both certificates have been revoked. Our investigation is ongoing, and a full incident report will follow.
  https://crt.sh/?id=18536829926
  https://crt.sh/?id=18535956439

• Relevant policies: 7.1.2.11.3 Signed Certificate Timestamp List

• Source of incident disclosure Notified via certificate problem reporting

Comment 1

[Steven Deitte]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000028

• Incident description: On 2025-05-22 15:33:00 UTC we received a report of a certificate signed by our GoDaddy Intermediate G2 where one of the embedded SCTs from DigiCert Yeti2025 had an invalid signature. During the investigation, an additional certificate with the same issue was surfaced. Both certificates were revoked at 2025-05-24 02:36:15.

• Timeline summary:

  • Non-compliance start date: 2025-05-21 10:31:01

  • Non-compliance identified date: 2025-05-22 15:33:00

  • Non-compliance end date: 2025-05-24 02:36:15

• Relevant policies:

BR 7.1.2.11.3 Signed Certificate Timestamp List

If present, the Signed Certificate Timestamp List extension contents MUST be an OCTET >STRING containing the encoded SignedCertificateTimestampList, as specified in RFC >6962, Section 3.3.

Each SignedCertificateTimestamp included within the SignedCertificateTimestampList >MUST be for a PreCert LogEntryType that corresponds to the current certificate.

• Source of incident disclosure: Notified via certificate problem reporting

Impact

• Total number of certificates: 2

• Total number of "remaining valid" certificates: 0

• Affected certificate types: DV Subscriber Certificates

• Incident heuristic: 3 See Appendix

• Was issuance stopped in response to this incident, and why or why not?: No, our investigation identified 2 impacted certificates and there was no evidence that this was an ongoing issue.

Timeline

2025-05-21 10:31:02 - Precert FF9295785F3AC53B: SCTs Obtained from Argon 2025H2 (Valid); Yeti 2025 (Invalid); Nimbus 2025 (Valid) - Compliance Incident Begins

2025-05-21 10:31:03 - SCTs are embedded in FF9295785F3AC53B and final cert issues

2025-05-21 10:31:24 - Precert B6AB57BCC08F43A: SCTs Obtained from Argon 2025H2 (Valid); Yeti 2025 (Invalid); Nimbus 2025 (Valid)

2025-05-21 10:31:25 - SCTs are embedded in B6AB57BCC08F43A and final cert issues

2025-05-22 15:33:00 – Report of certificate FF9295785F3AC53B with invalid SCT reported to Certificate Problem reporting – Compliance Incident Identified

2025-05-22 20:45:00 – GoDaddy begins investigation

2025-05-23 03:06:00 – Investigation uncovers certificate B6AB57BCC08F43A which has the same issue

2025-05-23 06:18:00 – Investigation concludes, no other affected certificates are found

2025-05-23 06:33:00 – GoDaddy CA responds to certificate problem report with findings.

2025-05-24 02:36:15 – Certificates FF9295785F3AC53B and B6AB57BCC08F43A are revoked within 5 day timeframe under BR 4.9.1.1.12 - Compliance Incident Ends

Related Incidents

Bug	Date	Description
1815874	2023-02-09 13:33:20	Google Trust Services: incorrect SCT in certificate
1952591	2025-03-08 12:33:48	CommScope: Validation issue in SCT extensions in certificates

These incidents relate to the problem of invalid embedded SCT signatures in Subscriber certificates. This incident is unique due to the invalid response coming from an otherwise valid CT log provider for the 2 certificates.

Root Cause Analysis

Contributing Factor #1: Insufficient Validation of SCT responses

• Description: Our validation of SCT signatures coming back from CT Logs included response code and payload structure. The validation did not check for a malformed signature in an otherwise healthy response. A strong check comparing the Signed Certificate Timestamp Signature against the expected value would have allowed us to detect and prevent embedding a malformed SCT in the certificate.

• Timeline: 2025-05-21 10:31:03 , 2025-05-21 10:31:25

• Detection: 2025-05-22 15:33:00

• Interaction with other factors: Invalid Signature from CT Log

Contributing Factor #2: Invalid Signature from CT Log

• Description: Our investigation leveraging application logs shows we submitted the 2 precertificates to the correct temporal CT log. The precertificates’ chain sent and received from the CT log matched, and the SCTs we embedded in the impacted certificates are identical to the response received from the Yeti 2025 CT Log. However, the signatures on the SCTs do not match the expected value pertaining to the to be signed certificates. We look forward to working with the CT Log operator to investigate this further.

• Timeline: 2025-05-21 10:31:02 , 2025-05-21 10:31:24

• Detection: 2025-05-23 06:18:00 **

• Interaction with other factors: Insufficient Validation of SCT responses

Lessons Learned

• What went well: Our application logging provided a detailed view of the requests and responses with the CT log for the impacted certificates. Impacted certificates were revoked within compliant timeframes.

• What didn’t go well: Further verification of SCTs from CT log providers is required

• Where we got lucky: Security researcher discovered and reported the issue that had limited impact on a small number of certificates

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke 2 impacted subscriber certificates	Mitigate	Primary Incident	Impacted certificates status shows revoked	2025-05-24	Complete
Patch to verify SCTs before inclusion in Final Certificate	Prevent	Insufficient Validation of SCT responses	Testing for SCT validations correctly block SCT embedding	2025-06-05	Complete
Add monitoring to alert detected invalid SCT signatures	Monitoring	Where we got lucky	Monitor alerts engineering on invalid SCT signatures being detected	2025-06-27	Ongoing
Compare notes with log operator	Prevent	Invalid Signature from CT Log	Communicate with log operator on https://groups.google.com/a/chromium.org/g/ct-policy to better understand what caused the invalid SCT to be returned	2025-07-01	Ongoing

Appendix

Precertificate SHA-256: BFDA6803A993617F359B2320603172E597FAE2C3B0CFDF90D63CE62A314B98BE
Certificate SHA-256: ACC973AEEE54FCF7B20CF02C64DC3C680B162693B90D3C174CBE08D0ADCE1E79
Subject: CN=karenkalkstein.com
Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certs.godaddy.com/repository/, CN=Go Daddy Secure Certificate Authority - G2
Not Before: 2025-05-21 10:31:01
Not After: 2025-08-19 10:31:01
Serial: FF9295785F3AC53B
Is revoked: Yes
Revocation Date: 2025-05-24 02:36:15
Revocation Reason: 4 (SUPERSEDED)
https://crt.sh/?id=18535952074 (pre) https://crt.sh/?id=18536829926 (final)

Precertificate SHA-256: FA4A2B3EEA40CED8B421BE5AE634F6C6DEB7A0A84567B5FB7ECFE865F84F5695
Certificate SHA-256: 73632F6A4C92DFA13896AE8499E494AE2EC1507BC92108B6ED5C53CD13183B2A
Subject: CN=ogradycenter.com
Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certs.godaddy.com/repository/, CN=Go Daddy Secure Certificate Authority - G2
Not Before: 2025-05-21 10:31:23
Not After: 2025-08-19 10:31:23
Serial: B6AB57BCC08F43AF
Is revoked: Yes
Revocation Date: 2025-05-24 02:36:15
Revocation Reason: 4 (SUPERSEDED)
https://crt.sh/?id=18535956439 (pre) https://crt.sh/?id=18813460780 (final)

Comment 2

[Steven Deitte]

We continue to monitor this incident for questions and are working on the remaining action items.

Comment 3

[Jeremy]

I think this should be closed as invalid.

1. There is no requirement to log to a CT log. Certificates do not work without logging, but a logging requirement is not expressly part of any root policy that I'm aware of.
2. The certificates were logged to a log, just one that violated the stated policy of the log. There isn't a requirement that says that certificates must be logged to the correct log. The certificate shouldn't work in browsers that use CT, but it's not a compliance violation if the log allows it for some reason.
3. The real issue is the log should have rejected the request as the cert validities were outside of the log's permitted policy. This isn't a compliance violation for the log, but it's something that should happen when a incorrect cert request is made.

Am I missing something?

Comment 4

[Andrew Ayer]

Jeremy, I think you meant to post Comment 3 to Bug 1970259? The incident here concerns a violation of BR 7.1.2.11.3.

Comment 5

[Wayne]

What Andrew said, additionally:
Starfield Technologies, LLC
Certificate Policy and Certification Practice Statement (CP/CPS) Version 5.02
February 28, 2025

4.4.2 Publication of the Certificate by the CA
CA certificates are published in the Starfield repository.
All SSL certificates are published in publicly accessible Certificate Transparency (CT) logs.

The full incident report also notes:

BR 7.1.2.11.3 Signed Certificate Timestamp List

If present, the Signed Certificate Timestamp List extension contents MUST be an OCTET >STRING containing the encoded SignedCertificateTimestampList, as specified in RFC >6962, Section 3.3.

So it's still a violation.

Comment 6

[Jeremy]

Thank you! Too many windows open and I meant to post on the other bug. Sorry about that.

Comment 7

[Steven Deitte]

We will respond to questions from Comment 3 on the bug it was meant for. Here's an update to our action items progress.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke 2 impacted subscriber certificates	Mitigate	Primary Incident	Impacted certificates status shows revoked	2025-05-24	Complete
Patch to verify SCTs before inclusion in Final Certificate	Prevent	Insufficient Validation of SCT responses	Testing for SCT validations correctly block SCT embedding	2025-06-05	Complete
Add monitoring to alert detected invalid SCT signatures	Monitoring	Where we got lucky	Monitor alerts engineering on invalid SCT signatures being detected	2025-06-27	Complete
Compare notes with log operator	Prevent	Invalid Signature from CT Log	Communicate with log operator on https://groups.google.com/a/chromium.org/g/ct-policy to better understand what caused the invalid SCT to be returned	2025-07-01	Ongoing

Comment 8

[Steven Deitte]

We continue to monitor for questions. Here's an update on our action items -

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke 2 impacted subscriber certificates	Mitigate	Primary Incident	Impacted certificates status shows revoked	2025-05-24	Complete
Patch to verify SCTs before inclusion in Final Certificate	Prevent	Insufficient Validation of SCT responses	Testing for SCT validations correctly block SCT embedding	2025-06-05	Complete
Add monitoring to alert detected invalid SCT signatures	Monitoring	Where we got lucky	Monitor alerts engineering on invalid SCT signatures being detected	2025-06-27	Complete
Compare notes with log operator	Prevent	Invalid Signature from CT Log	Communicate with log operator on https://groups.google.com/a/chromium.org/g/ct-policy to better understand what caused the invalid SCT to be returned	2025-07-01	Complete - https://groups.google.com/g/certificate-transparency/c/_AaK_UHdxp0/m/KCDCW9DSCwAJ

Comment 9

[incident-reporting]

Please provide a Closure Report if you feel this is ready for closure.

Comment 10

[Steven Deitte]

Report Closure Summary

• Incident description: Certificates issued with an invalid SCT signature
• Incident Root Cause(s): An invalid SCT returned by the CT log provider combined with a lack of validation by the CA.
• Remediation description: Revoked impacted certificates, added validation logic for SCT signatures coming back from CT Log providers before embedding on the certificate
• Commitment summary: GoDaddy continues to investigate opportunities to improve our testing and validations for CT log and other areas of certificate issuance.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 11

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-25.