2077611 - GoDaddy: Delayed CRL Updates

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[Steven Deitte]

Steps to reproduce

Preliminary Incident Report

Summary

• Incident description: CRLs were reported as not being updated and published within 24 hours after certificates were recorded as revoked.
• Relevant policies: CA/Browser Forum TLS Baseline Requirements §4.9.7
• Source of incident disclosure: Certificate Problem Reporting

GoDaddy received a Certificate Problem Report identifying instances where revoked certificates first appeared on a published CRL more than 24 hours after their recorded revocation time.

GoDaddy has identified that the affected CRLs are being generated and published to the origin within the required timeframe, but older versions are being served from our CDN cache. We are investigating the CDN caching rules to determine the changes necessary to ensure updated CRLs are made available at the public CRL distribution points within the required timeframe.

Actual results

N/A

Expected results

N/A

Comment 1

[Rob White]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000028
• Incident description: GoDaddy received a Certificate Problem Report identifying instances where revoked certificates were not made available on publicly served CRLs within 24 hours after the certificates were recorded as revoked due to CDN caching of older CRLs.
• Timeline summary:
• Non-compliance start date: 2025-11-19
• Non-compliance identified date: 2026-10-01
• Non-compliance end date: 2026-10-03
• Relevant policies: CA/Browser Forum TLS Baseline Requirements §4.9.7:

A CA issuing Subscriber Certificates MUST update and publish a new CRL within twenty-four (24) hours after recording a Certificate as revoked.

• Source of incident disclosure: Certificate Problem Reporting

Impact

• Total number of certificates: Unable to be precisely determined. As part of its investigation, GoDaddy evaluated whether the complete affected population could be reconstructed. Because different CDN Points-of-Presence could have cached and served different CRL versions, the available historical data is insufficient to determine which version was served by each location at a given time. As a result, there is no reliable method to retrospectively determine the complete affected certificate population.
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: Subscriber Certificates
• Incident heuristic: N/A
• Was issuance stopped in response to this incident, and why or why not?: No. There was no certificate misissuance as part of this incident.
• Analysis: GoDaddy's investigation confirmed that updated CRLs containing the affected revocations were generated and available at the origin within the required timeframe. However, the CDN continued to serve older cached versions of certain CRLs after newer CRLs had been generated and published to the origin. As a result, some revocation information was not made available through the public CRL distribution points within 24 hours after the certificates were recorded as revoked. GoDaddy reduced the CDN cache TTL from 24 hours to three hours, cleared the cache, and updated the cache-control configuration to require revalidation of cached CRLs with the origin.
• Additional considerations: N/A

Timeline

All dates are UTC.

Date (UTC)	Description
2025-11-19 11:44:37	A CDN change caused the CDN cache policy to disregard the Cache-Control: max-age directive returned by the CRL origin. Consequently, cached CRL responses may have remained eligible for delivery beyond the origin-defined freshness period, causing some relying parties to retrieve stale CRL content. Incident begins.
2026-09-30 08:21:20	Certificate with serial number 0EF93A9CBBED7E52 was recorded as revoked.
2026-09-30 13:54:19	Certificate with serial number 3F927549963A1AD8 was recorded as revoked.
2026-09-30 20:50:30	CRL 91 for gdr1v1dv_s1-13.crl was generated containing serial number 0EF93A9CBBED7E52.
2026-10-01 02:13:27	CRL 112 for gdr1v1dv_s1-12.crl was generated containing serial number 3F927549963A1AD8.
2026-10-01 21:51:20	GoDaddy received a Certificate Problem Report identifying instances where revoked certificates first appeared on publicly served CRLs more than 24 hours after their recorded revocation time. Incident identified.
2026-10-02 15:46:00	GoDaddy began investigating the reported behavior by comparing CRLs generated at the origin with CRLs served through the public CRL distribution points.
2026-10-02 17:41:00	GoDaddy confirmed that the tested CDN endpoints were serving older cached CRLs while newer CRLs were available at the origin.
2026-10-02 18:12:00	GoDaddy determined that the CDN caching configuration could allow an older CRL to continue being served after an updated CRL had been generated and published to the origin.
2026-10-02 18:59:00	GoDaddy responded back to Certificate Problem Report.
2026-10-02 23:59:00	GoDaddy reduced the CDN cache TTL for CRLs from 24 hours to three hours and cleared the CDN cache.
2026-10-03 00:00:00	GoDaddy confirmed that the updated cache-control settings were taking effect. Incident ends.

Related Incidents

None identified.

Root Cause Analysis

The fundamental failure in this incident was that the caching configuration for GoDaddy's public CRL distribution infrastructure was not appropriately aligned with the frequency at which updated CRLs were generated.
GoDaddy generates updated CRLs approximately every 12 hours. However, the CDN used to serve CRLs from the public CRL distribution points was configured with a cache TTL of 24 hours. This created a condition where an updated CRL containing newly recorded revocations could be generated and made available at the origin while the CDN continued to serve an older CRL.
GoDaddy's existing monitoring did not detect this condition because it validated CRL generation and availability at the origin rather than validating the CRL ultimately served through the public CRL distribution points.

Contributing Factor 1: CDN caching configuration was not aligned with CRL generation frequency

• Description: GoDaddy generates updated CRLs approximately every 12 hours. The CDN caching configuration allowed an older CRL to remain cached for up to 24 hours. As a result, a newer CRL containing updated revocation information could be available at the origin while the CDN continued serving an older cached CRL. This could cause revocation information to remain unavailable through the public CRL distribution point for more than 24 hours after a certificate was recorded as revoked.
• Timeline: This condition existed prior to identification of the incident and continued until GoDaddy reduced the CDN cache TTL and cleared the cache on October 2, 2026.
• Detection: GoDaddy identified this contributing factor during its investigation following receipt of the Certificate Problem Report. GoDaddy compared CRLs available at the origin with CRLs returned through the CDN and confirmed that the CDN was serving older cached versions.
• Interaction with other factors: The caching configuration was not identified before the incident because existing monitoring validated CRLs at the origin rather than through the public CRL distribution points.
• Root Cause Analysis methodology used: 5 Whys.

Contributing Factor 2: Monitoring did not validate CRLs through the public CRL distribution points

• Description: GoDaddy had existing monitoring to verify CRL generation and publication at the origin. However, this monitoring did not retrieve and validate CRLs through the public CRL distribution points. As a result, the monitoring could confirm that an updated CRL had been successfully generated and made available internally without detecting that the CDN continued to serve an older cached CRL.
• Timeline: This monitoring configuration was in place prior to the incident and remained in place when the Certificate Problem Report was received.
• Detection: GoDaddy identified this monitoring gap while investigating why the CRLs observed by the reporter differed from the CRLs available at the origin.
• Interaction with other factors: Because monitoring did not validate the publicly served CRLs, it did not identify the effect of the CDN caching configuration described in Contributing Factor 1.
• Root Cause Analysis methodology used: 5 Whys.

Lessons Learned

• What went well: Following receipt of the Certificate Problem Report, GoDaddy promptly compared the publicly served CRLs with the CRLs available at the origin and identified that the CDN was serving older cached versions. GoDaddy reduced the CDN cache TTL from 24 hours to three hours, cleared the cache, and confirmed that the updated caching configuration was taking effect.
• What didn't go well: The CDN caching configuration was not aligned with GoDaddy's CRL generation frequency and could allow older CRLs to continue being served after updated CRLs were available at the origin. Additionally, existing monitoring validated CRLs at the origin rather than through the public CRL distribution points and therefore did not detect the stale CRLs being served by the CDN.
• Where we got lucky: GoDaddy's CRL generation process continued to generate updated CRLs containing the affected revocations within the required timeframe, which allowed the investigation to distinguish CRL generation from the public-serving issue and identify the CDN caching layer as the source of the delay.
• Additional: GoDaddy identified the need to validate revocation information from the same public distribution points used by relying parties rather than relying solely on monitoring of the underlying CRL generation and origin infrastructure.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Reduce CDN cache TTL for CRLs	MITIGATE	Root Cause 1	CDN cache TTL for CRLs is reduced from 24 hours to three hours, the existing cache is cleared, and the updated configuration is verified as taking effect	2026-10-02	Complete
Implement monitoring of CRLs through the CDN	DETECT	Root Cause 2	Monitoring retrieves CRLs through the public CDN path and detects when the publicly served CRL is not current with the CRL available at the origin	2026-12-01	In Progress
Implement monitoring of CDN configuration	DETECT	Root Cause 1	Monitoring detects changes to the CDN configuration that could affect CRL caching and delivery	2026-12-01	In Progress