1970259 - GoDaddy: Precertificates incorrectly logged to DigiCert SCT Logs

Status: RESOLVED INVALID

(CA Program :: CA Certificate Compliance, task)

Description

[Steven Deitte]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36

Steps to reproduce:

Preliminary Incident Report

Summary

• Incident description: 5079 Precertificates incorrectly logged to DigiCert 2026h1 logs
• Relevant policies: Apple’s Certificate Transparency Policy - https://support.apple.com/en-us/103703 minimum number of distinct SCT logs
• Source of incident disclosure: Certificate Problem Reporting and CT-Policy Google Group thread - https://groups.google.com/a/chromium.org/g/ct-policy/c/5gf-6ODjF70

Our investigation is ongoing, and a full incident report will follow.

Comment 1

[Steven Deitte]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000028
• Incident description: GoDaddy issued 5079 certificates which contained an SCT from a CT Log that was outside the CT Log’s stated temporal interval. Bugs in our CT logic and the CT Logs led to valid SCT signatures being embedded on the certificates from incorrect temporal CT logs. Affected certificates failed to meet the minimum SCT log entry requirements of the Apple Certificate Transparency policy.
• Timeline summary:
• Non-compliance start date: 2025-05-30 00:00:11
• Non-compliance identified date: 2025-05-30 23:35:00 
• Non-compliance end date: 2025-06-03 19:41:00  
• Relevant policies: Apple's Certificate Transparency policy - https://support.apple.com/en-us/103214
• Source of incident disclosure: Certificate Problem Reporting from Security Researcher and CT-Policy Google Group thread - https://groups.google.com/a/chromium.org/g/ct-policy/c/5gf-6ODjF70

Impact

• Total number of certificates: 5079
• Total number of "remaining valid" certificates: 4208
• Affected certificate types: DV: 5036, OV: 34, EV: 9
• Incident heuristic: 3 – Full list, see appendix
• Was issuance stopped in response to this incident, and why or why not?: No, all impacted certificates were BR-compliant. The impact was limited to certificates failing to reach Apple’s Certificate Transparency policy’s minimum for SCT logs.

Timeline

Time (UTC)	Event
2020-05-11 16:39:11	Bug where priority could override temporal interval during CT log selection introduced
2024-07-19 00:19:06	Sphinx 2026h1 CT Inclusion request submitted with stated temporal interval ending on 2025-07-01
2024-07-19 00:22:22	Wyvern2026h1 CT Inclusion request submitted with stated temporal interval ending on 2025-07-01
2025-05-30 00:00:11	First GoDaddy precert outside of Wyvern2026h1’s stated temporal interval obtains SCT (serial D057DFBEE8494848)
2025-05-30 09:45:39	Security Researcher starts inquiry thread with Digicert on the Certificate Transparency Policy Group
2025-05-30 18:56:54	Digicert confirms that Wyvern 2026h1, Wyvern 2026h2, Sphinx 2026h1, and Sphinx 2026h2 were configured to cut off at 2026-07-07 rather than 2026-07-01 but due to a typo the 2026-07-01 date is what was put on log_list.json
2025-05-30 21:46:12	Last GoDaddy precert outside of Wyvern2026h1’s stated temporal interval obtains SCT (serial 8F5E5F1869B524B8)
2025-05-30 21:46:50	First GoDaddy precert outside of Sphinx2026h1’s stated temporal interval obtains SCT (serial FB42D8E1079531DB)
2025-05-30 21:51:25	Last GoDaddy precert outside of Sphinx2026h1’s stated temporal interval obtains SCT (serial 27B58B28539127E6)
2025-05-30 22:04:53	Digicert reports that they reconfigured the Wyvern and Sphinx logs to match the stated temporal intervals in log_list.json
2025-05-30 23:27:30	Apple confirms that Safari uses the CT log’s stated temporal interval as part of SCT validity criteria
2025-05-30 23:35:00	Security researcher contacts GoDaddy at [e-mail] to report the discovery of certificates with valid SCTs outside Wyvern2026h1 and Sphinx2026h1’s stated temporal interval
2025-05-31 02:29:00	GoDaddy begins investigation
2025-05-31 05:00:00	GoDaddy determines that affected customers will see errors when using the certificates with Safari browsers
2025-05-31 23:25:00	GoDaddy responds to certificate researcher explaining our intent to prevent the issue in the future and work with Subscribers to reissue certificates
2025-06-02 15:51:00	GoDaddy deploys a production patch to ensure that precertificates are only submitted to CT logs with the appropriate temporal interval
2025-06-03 19:41:00	GoDaddy deploys a patch to cover edge case relating to timezone offsets and temporal intervals
2025-06-03 20:40:00	GoDaddy begins multiple campaigns to alert customers that certificates need to be reissued

Related Incidents

Bug	Date	Description
1969296	2025-05-29 16:25:22	GoDaddy: Certificates with invalid embedded SCT signatures. This also relates to Embedded SCTs and how they affect validity. However, in this case, while the embedded SCT list had enough SCTs to meet CT Policy requirements for Chrome and Apple, the presence of one that lacked a signature corresponding to the pre-certificate meant that BR 7.1.2.11.3 was violated and required revocation

Root Cause Analysis

Contributing Factor #1: Bug in GoDaddy CT Logic

• Description: In our logic for logging precertificates to CT logs, we assign priorities to CT logs to control the order that precertificate submission is attempted. Upon investigation, it was found that this system was overriding temporal interval considerations. This led the system to try some CT logs outside of the correct temporal interval when CT logs were sharded with temporal intervals smaller than 1 year, as in the case with Wyvern and Sphinx.
• Timeline: 2020-05-11 16:39:11
• Detection: Manual investigation by engineering investigating how certificates could be attempting to use incorrect temporal CT logs.
• Interaction with other factors: Our regular tuning of those priorities and the behavior of most CT Logs to return a 400 response instead of an SCT for a precertificate falling outside its temporal interval made this harder to detect.

Contributing Factor #2: CT Log Accepting Precertificates Outside its Stated Temporal Interval

• Description: CT logs generally reject submissions of precertificates that do not match their temporal interval. Wyvern2026h1 and Sphinx2026h1 logs were configured to accept precerticates expiring up until 2026-07-07 even though the stated interval was 2026-07-01. The precertificates GoDaddy submitted were within the log’s configured interval which resulted in a successful (but erroneous) SCT response.
• Timeline: 2024-07-19 00:19:06, 2024-07-19 00:22:22
• Detection: 2025-05-30 09:45:39
• Interaction with other factors: The CT log accepting temporal out of bound precertificates combined with GoDaddy’s CT logic bug led to this incident.

Lessons Learned

• What went well: Once we were aware of the issue, it wasn’t difficult to surface the impacted certificates. Digicert resolved the configuration on the CT log list quickly.
• What didn’t go well: GoDaddy’s CT logic bug resulted in a reliance on CT log behavior to ensure the correct temporal CT log was used.
• Where we got lucky: Security Researcher worked with both us and Digicert to identify and resolve the issue.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Patch system ensure CT Log Temporal intervals are respected	Prevent	What didn’t go well	Unit tests pass, logs reflect that only relevant logs are being submitted to	2025-06-03 19:41:00	Complete
Add additional check for temporal interval to SCT validation logic	Prevent	Where we got lucky	Testing for SCT validations correctly block SCT embedding if signature or temporal interval don’t match expectations	2025-07-31	Ongoing
Reach out to affected customers to encourage reissuance and revocation	Mitigate	What didn’t go well	Number of "remaining valid" decreases over time	2025-06-09	Complete

Comment 2

[Steven Deitte]

Attachment: bugzilla_1970259_affected_certificates_sphinx2026h1.csv

Comment 3

[Steven Deitte]

Attachment: bugzilla_1970259_affected_certificates_wyvern2026h1.csv

Comment 4

[Jeremy]

Posted to the correct bug this time:

I think this should be closed as invalid.

1. There is no requirement to log to a CT log. Certificates do not work without logging, but a logging requirement is not expressly part of any root policy that I'm aware of.
2. The certificates were logged to a log, just one that violated the stated policy of the log. There isn't a requirement that says that certificates must be logged to the correct log. The certificate shouldn't work in browsers that use CT, but it's not a compliance violation if the log allows it for some reason.
3. The real issue is the log should have rejected the request as the cert validities were outside of the log's permitted policy. This isn't a compliance violation for the log, but it's something that should happen when a incorrect cert request is made.

Comment 5

[Andrew Ayer]

I agree with Comment 4 - unless something has changed with Apple's policy this should be closed as INVALID and need not be noted on GoDaddy's next audit statement.

Nevertheless, I appreciate GoDaddy opening this and posting the incident report in Comment 1 as it helps raise awareness of pitfalls to avoid when logging to CT. I think that when a CA provides an incident report when it's not required, it helps to bolster trustworthiness in the CA and demonstrate how they are providing value to the ecosystem.

Comment 6

[Steven Deitte]

Thank you for the comments and we agree this may be an invalid bug. We will continue to monitor for questions and update the community on our action items' progress.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Patch system ensure CT Log Temporal intervals are respected	Prevent	What didn’t go well	Unit tests pass, logs reflect that only relevant logs are being submitted to	2025-06-03 19:41:00	Complete
Add additional check for temporal interval to SCT validation logic	Prevent	Where we got lucky	Testing for SCT validations correctly block SCT embedding if signature or temporal interval don’t match expectations	2025-07-31	Ongoing
Reach out to affected customers to encourage reissuance and revocation	Mitigate	What didn’t go well	Number of "remaining valid" decreases over time	2025-06-09	Complete

Comment 7

[incident-reporting]

Marking as "INVALID."

GoDaddy is free (and encouraged) to continue updating the community with regard to its progress as described in Comment 6.

Comment 8

[Steven Deitte]

Update on the action items -

Add additional check for temporal interval to SCT validation logic

was completed on 2025-07-28.