2004845 - GoDaddy: CA Certificates Published in PEM format

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Steven Deitte]

Steps to reproduce:

Preliminary Incident Report

Summary

• Incident description: Two GoDaddy CA certificate files referenced in the CA Issuers URI (AIA) of issued certificates were being served in PEM format.
• Relevant policies: RFC 5280 4.2.2.1

Where the information is available via HTTP or FTP, accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in [RFC2585] or a collection of certificates in a BER or DER encoded "certs-only" CMS message as specified in [RFC2797].

• Source of incident disclosure: Certificate Problem Reporting

The hosted CA certificate files have been updated to the DER format, and a full investigation is in progress. A detailed incident report will follow.

Comment 1

[Steven Deitte]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000028
• Incident description: GoDaddy CA Certificates hosted in incorrect PEM format on CA Issuers URI of issued certificate
• Timeline summary: (UTC)
  • Non-compliance start date: 2025-09-24 12:00
  • Non-compliance identified date: 2025-12-06 00:13
  • Non-compliance end date: 2025-12-08 23:27
• Relevant policies: RFC 5280 Section 4.2.2.1 AIA CA Issuers URI must point to a DER encoded certificate.
• Source of incident disclosure: Certificate Problem Reporting

Impact

• Total number of certificates: 2
• Total number of "remaining valid" certificates: 0
• Affected certificate types: CA Subordinate Cross Sign Certificates
• Incident heuristic: 3
• Was issuance stopped in response to this incident, and why or why not?: No, there was no mis-issuance and the hosted certificate files were updated promptly to correct the issue.
• Analysis:
• Additional considerations:

Timeline (UTC)

• 2024-03-05 17:08 GoDaddy engineering updated our hosted CA Certificate naming convention and format updates on repository such that *.crt files were in DER format for generated files. This was not applied to existing CA certificate files.

• 2025-09-18 19:59 Cross sign CA certificates profile review ensured AIA CA Issuer URI was present in http format. Review skipped verification that the hosted CA certificate file was in DER format.

• 2025-09-24 12:00 R1 Cross signed CA certificates created with the G2 Root CA files listed in the AIA CA Issuers field. (Non-compliance start)

• 2025-12-06 00:13 Certificate problem reporting identified non-compliance with the file format of the ‘Starfield TLS Root CA – R1’ AIA CA Issuer URI.

• 2025-12-08 17:03 GoDaddy engineering confirmed the incorrect PEM encoded file being hosted for both Starfield and GoDaddy G2 root certificates.

• 2025-12-08 23:27 GoDaddy engineering deployed updated DER encoded CA files for the G2 root certificates. (Non-compliance end)

• 2025-12-19 16:39 GoDaddy engineering deployed monitoring for all CA Issuer URIs contained on issued certificates that ensures hosted files are DER format.

Related Incidents

Bug	Date	Description
[Related Bug ID](Related Bug URL)	Date Related Bug was opened	A description of how the subject Bug is related to the Bug referenced.
2004492	2025-12-05 23:02	CA Certificate not published in DER Encoded Format
1884461	2024-03-08 23:12	CA Certificates not published in DER Encoded Format

There are several past incidents from other CAs that prompted GoDaddy Engineering to update our naming and format conventions when uploading generated CA certificate files. At that time no CA Issuer field on issued certificates referred to the G2 Root CA files, and GoDaddy engineering did not make retroactive changes to the PEM formatted CA certificate files to match the updated standards.

A more recent incident that was reported was posted hours before the community member submitted a certificate problem report. We had not yet reviewed this incident when we received the certificate problem report.

Root Cause Analysis

Contributing Factor 1: Playbook for CA certificate profile reviews missing DER encoded CA Issuer field verification

• Description: GoDaddy reviews CA certificate profiles before the certificates are issued in a ceremony. This review did not include verification that the existing Root CA certificate file was being hosted in DER format.
• Timeline: 2025-09-18 19:59 Cross sign CA certificates profile review ensured AIA CA Issuer URI was present in http format.
• Detection: Incident investigation
• Interaction with other factors: During the profile review for these cross signed certificates it was believed that the older Root CA files were the same format as the standard. (Factor 2)
• Root Cause Analysis methodology used:

Contributing Factor 2: Assumption that older hosted CA certificate files matched DER format per the documented naming and format convention.

• Description: GoDaddy maintains a documented naming and format convention for the hosted CA certificate files on our repository. It was believed that the older CA Root certificate files matched that format.
• Timeline:

-- 2024-03-05 17:08 GoDaddy engineering updated our hosted CA Certificate naming convention and format updates on repository such that *.crt files were in DER format for generated files.

-- 2025-09-18 19:59 Cross sign CA certificates profile review ensured AIA CA Issuer URI was present in http format. Review skipped verification that the hosted CA certificate file was in DER format.

• Detection: Incident investigation
• Interaction with other factors: Factor 1
• Root Cause Analysis methodology used:

Contributing Factor #: title Lack of monitoring on the files hosted behind AIA CA Issuers’ values

• Description: GoDaddy lacked monitoring to ensure that all hosted CA certificate files at locations listed in the CA Issuers fields of certificates are in DER format.
• Timeline: 2025-09-24 12:00 R1 Cross signed CA certificates created with the G2 Root CA files listed in the AIA CA Issuers field. (Non-compliance start)
• Detection: Incident investigation.
• Interaction with other factors: N/A
• Root Cause Analysis methodology used:

Lessons Learned

• What went well: GoDaddy engineering was able to remediate the problem quickly, rolling out DER files at the hosted locations.
• What didn’t go well: The lack of monitoring left the GoDaddy CA vulnerable to any change of the hosted CA files causing an interruption to compliance.
• Where we got lucky: N/A
• Additional:

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Deploy monitoring for DER encoding on hosted CA Certificate files	Detect	Root Cause # 2	Alert on any non-DER format file detected	2025-12-19	Complete
Review CA Certificate generation playbook to ensure we are proactively validating for DER encoded CA Issuer field values, and any other areas that need further validation	Prevent	Root Cause # 1	Completed Review	2025-01-30	Ongoing
Improve monitoring for hosted CA Certificate Profiles	Detect	Root Cause # 2	Alert on any missing or mismatch of certificate	2025-01-30	Ongoing

Comment 2

[Steven Deitte]

Attachment: Bugzilla_2004845_appendix.csv

Comment 3

[Steven Deitte]

We are continuing to monitor this incident for questions and are still in progress on the remaining action items.

Comment 4

[Steven Deitte]

We are continuing to monitor this incident for questions and are making progress towards our open action items but no further update.

Comment 5

[Steven Deitte]

We continue to monitor this incident for questions and are making progress towards our open action items.

Comment 6

[Steven Deitte]

GoDaddy continues to monitor this thread for questions and expects to have the remaining action items completed this week.

Comment 7

[Steven Deitte]

Action Items Update

We have updated our CA certificate generation procedures to require explicit validation that Authority Information Access (AIA) issuer URLs resolve to the correct DER-encoded CA certificate files and new CA certificate files are added to the monitoring scripts.

In addition, we have enhanced monitoring for hosted CA certificate files to verify that each URL consistently serves the expected certificate, using SHA-256 certificate thumbprint validation to detect any unexpected changes.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Deploy monitoring for DER encoding on hosted CA Certificate files	Detect	Root Cause # 2	Alert on any non-DER format file detected	2025-12-19	Complete
Review CA Certificate generation playbook to ensure we are proactively validating for DER encoded CA Issuer field values, and any other areas that need further validation	Prevent	Root Cause # 1	Completed Review	2025-01-30	Complete
Improve monitoring for hosted CA Certificate Profiles	Detect	Root Cause # 2	Alert on any missing or mismatch of certificate	2025-01-29	Complete

Comment 8

[Steven Deitte]

Report Closure Summary

• Incident description: GoDaddy CA Certificates hosted in incorrect PEM format on CA Issuers URI of issued certificate
• Incident Root Cause(s): Validation steps missing from CA certificate generation process and insufficient monitoring led to the incident
• Remediation description: GoDaddy CA updated the files hosted behind the CA Issuers URI to be the DER format version of the certificates
• Commitment summary: GoDaddy CA will continue to proactively enhance its monitoring and operational processes to ensure ongoing compliance with disclosure requirements, including for hosted CA certificate files.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 9

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-02-11.