Closed Bug 1963456 Opened 9 months ago Closed 6 months ago

GoDaddy: CA Certificates with HTTPS URL in AIA Field

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: sdeitte, Assigned: sdeitte)

Details

(Whiteboard: [ca-compliance] [ca-misissuance])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36

Steps to reproduce:

Preliminary Incident Report

Summary

  • Incident description: Subordinate CA certificates created with HTTPS URL in the CA Certificate Authority Information Access field
  • Relevant policies: Servercert Baseline Requirement 7.1.2.10.3
  • Source of incident disclosure: Certificate Problem Reporting notification
Assignee: nobody → sdeitte
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Summary: GoDaddy - CA Certificates with HTTPS URL in AIA Field → GoDaddy: CA Certificates with HTTPS URL in AIA Field
Whiteboard: [ca-compliance] [ca-misissuance]
Type: defect → task

Where is the final incident report?

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000028
  • Incident description: 2 Cross sign subordinate CA certificates issued with HTTPS URL in the AIA field
  • Timeline summary:
    • Non-compliance start date: 6/6/2024
    • Non-compliance identified date: 4/28/2025
    • Non-compliance end date: 5/2/2025
  • Relevant policies: Section: 7.1.2.10.3 CA Certificate Authority Information Access that states for id-ad-caIssuers the value shall be "A HTTP URL of the Issuing CA's certificate."
  • Source of incident disclosure: Email sent to certificate problem reporting address.

Impact

  • Total number of certificates: 2
  • Total number of "remaining valid" certificates: 0
  • Affected certificate types: Cross-Certified Subordinate CA Certificate
  • Incident heuristic: 3
  • Was issuance stopped in response to this incident, and why or why not?: No, subscriber certificates were unaffected
  • Analysis: N/A

Timeline

5/2024 - cross signed subordinate test certificate profiles are created and vetted against certificate linters without errors

5/2024 - human error confirming profile met baseline requirements with the HTTPS url in the CA Certificate Authority Information Access field

4/28/2025 - investigated and confirmed cross signed subordinate misissuance
5/1/2025 - revoked Certificates and published updated CRLs
5/5/2025 - Linting tools patched to catch AIA non-HTTP url. Verfied fix in non-production environments.

Related Incidents

Bug Date Description

Root Cause Analysis

Contributing Factor # Linting processes failed to find HTTPS URL in AIA field

  • Description: Linters used in CA certificate generation process to test for compliance failed to detect the HTTPS URL in the AIA field.

  • Detection: Testing specifically for this problem after the CA was made aware of the issue.

  • Interaction with other factors: This combined with the human error when populating the HTTPS url in the AIA field led to the incident.

Lessons Learned

  • What went well: Certificates were revoked promptly.
  • What didn't go well: Missed requirement for AIA field to include HTTP only URLs
  • Additional: Proposing patches to certlint project to include HTTP check on AIA field.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Revoke CA Cross Subordinate Certificates Resolve Primary Incident Certificates properly revoked 2025-05-02 Complete
Patch linting tool used to catch HTTP url Mitigate Contributing factor Test passes with problematic certificates 2025-05-05 Complete

Apologies, I didn't include the appendix with my previous post, including here.

Appendix

Certificate: sf_int_cross-g2g5.cert.pem
SHA-256: Fingerprint 29E502995310403685DD6258F9D04856878963042D4749C6D5D9773A310C31C0
Subject: C=US, O=Starfield Technologies, Inc., CN=Starfield Secure Certificate Authority - G5
Issuer: C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Root Certificate Authority - G2
Not before: Jun 6 000000 2024 GMT
Not after: Jun 21 000000 2042 GMT
Serial: 32717d587b580b50b82b38a0f2084a80
Is revoked: Yes
Revocation: date 5/1/25
Revocation: reason 4

Certificate: gd_int_cross-g2g5.cert.pem
SHA-256 Fingerprint: 880FBC44FE71AA1BE40DB130917B1BAEC1BD889B2B3D7478EDA04728667E8B44
Subject: C=US, O=GoDaddy Inc., CN=GoDaddy Secure Certificate Authority - G5
Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., CN=Go Daddy Root Certificate Authority - G2
Not before: Jun 6 000000 2024 GMT
Not after: Jun 21 000000 2042 GMT
Serial: 9f6c551847cdc100e72bb4183d854931
Is revoked: Yes
Revocation: date 5/1/25
Revocation: reason 4

Q1: What linter are you using? Sounds like your linter was out of date?
Q2: How often are you updating your linter and how are you getting notifications about updated made to linters based on requirements?

I will note that blaming the linter is never a good idea. DId you have two people review the documentation?

Seems like you'd have an email or JIRA ticket where a human verified the contents:

5/2024 - cross signed subordinate test certificate profiles are created and vetted against certificate linters without errors
5/2024 - human error confirming profile met baseline requirements with the HTTPS url in the CA Certificate Authority Information Access field

Do you not have logs of the review process?

Thank you for the questions, answers are below.

Q1 A) For CA certificates, we use the Boulder CA tool for certificate generation. Internally that has linting processes. Our initial reported findings did not make note of profile configurations, which were reviewed and found to have configured the tool to skip AIA linting for these certificates.

We also run the certificates through certlint as a secondary lint after generation before production use which was done, however certlint's code had not yet been updated to cover the HTTP URL enforcement in AIA at that point.

Q2 A) Subscriber certificates are linted on updated versions of ZLint and Certlint and we are in the process of bringing PKI Metal into the workflow as well. We monitor when updates to linters on GitHub repositories are merged to ensure we keep the linting processes in our CA up to date.

Q3 A) We do not have logs for the ceremony specifically around the linting of the certificates. Moving forward, we are updating our ceremony procedures to ensure that we record results in our ceremony documentation.

Thank you for providing the full incident report in Comment 2. Unfortunately, this report does not adhere to the CCADB IRGs and should be updated.

(1) The “Timeline” Section does not include the expected timeline elements. For example, several relevant items are missing including (a) the time at which the incident began, (b) the time at which the CA Owner became aware of the incident, and (c) the time at which the CA Owner received a Certificate Problem Report.

(2) It seems that there have been similar incidents in the past 2 years that could populate the “Related Incidents” Section. For example, consider 1884714, 1908128, and 1965459.

(3) We would not consider the “Root Cause Analysis” Section to contain a detailed analysis of the conditions which combined to give rise to the issue. Rather, it just offers a single sentence that faults the linter. Further questioning by a community member in Comment 4 led to the extraction of additional Action Items not included in the full report. In general, we would encourage GoDaddy to document a much more robust RCA with a belief that it should lead to additional Action Items.

(4) Given the large amount of observed certificate misissuance in the Spring of 2024 before the subject CA certificates were signed, which included incidents with similar failure modes (e.g., 1884714) and highlighted deficiencies with several linting tools, we would have expected CAs to evaluate the efficacy of their linting solutions and make updates, where needed, at that time. Can you share why, at that time, GoDaddy did not consider the Action Items described in this report?

(5) The “Lessons Learned” Section shares very little that could be helpful to all CA Owners in building better systems, policies and/or processes. This is another area where we would encourage GoDaddy to offer more detail in a revised full report.

We continue to investigate this issue and are working on a more complete incident report. We will have a new report posted by Wednesday, May 28th.

Responses to questions from root store program below with an updated incident report.

Q1:See updated Timeline section below.Q2 Yes, there have been related incidents, see the updated Related Incidents section below.
Q3 We have included a more detailed Root Cause Analysis in the section below.
Q4 We reviewed these Bugzilla reports from other CAs.Their focus was on the authorityInfoAccess field in Subscriber Certificates. We did not extend that perspective to evaluate our CA certificate ceremony processes, a step that would have helped us identify this issue earlier.
Q5 More detailed learnings have been included in the Learnings section below.

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000028
  • Incident description: 2 Cross sign subordinate CA certificates issued with HTTPS URL in the AIA field
  • Timeline summary:
    • Non-compliance start date: 6/6/2024
    • Non-compliance identified date: 4/28/2025
    • **Non-compliance end date: ** 5/14/2025
  • Relevant policies: Section: 7.1.2.10.3 CA Certificate Authority Information Access that states for id-ad-caIssuers the value shall be “A HTTP URL of the Issuing CA's certificate.”
  • Source of incident disclosure: Email sent to certificate problem reporting address.

Impact

  • Total number of certificates: 4
  • Total number of "remaining valid" certificates: 0
  • Affected certificate types: Cross-Certified Subordinate CA Certificate, TLS Subordinate CA Certificate
  • Incident heuristic: 3
  • Was issuance stopped in response to this incident, and why or why not?: No, subscriber certificates were unaffected. The certificates involved were never used to issue any subscriber certificates.
  • Analysis: N/A

Timeline

2022-05-18 17:13:43 - Created Root, Intermediate, Cross profiles for new G5 generation

2022-09-17 21:00:00 - G5 Root ceremony performed; G5 Root certificates created

2024-01-24 17:46:00 - Initial drafts of G5 Intermediate Profiles are reviewed in an email chain with the Compliance team. At this time, the proposed profile showed the AIA id-ad-caIssuers record URL containing http.

2024-04-26 20:00:17 - G5 Intermediate and Cross Signed Profiles updated and generated for Dev/Test Environments with AIA id-ad-caIssuers record URL containing https

2024-06-04 19:36:29 - Added "mock" production profiles for G5 Intermediate + crosses. Created mock certificates which simulate the final production G5 intermediates and crosses. Linting on the mock certificates produced no errors.

2024-06-05 20:30:00 - Reviewed mock certificates with stakeholders and Compliance team and the AIA ‘id-ad-caIssuers' containing a url with HTTPS was overlooked.

2024-06-06 16:00:00 - Performed Ceremony to Create Production G5 Intermediates + Cross signs – Incident Begins

2025-04-28 17:47:00 – GoDaddy CA was notified via certificate problem reporting of 2 possible instances of mis issuance around HTTPS URL in AIA field.

2025-04-28 18:48:00 – Initial review of profiles found G5 cross certificates contained AIA id-ad-caIssuers record URL in https

2025-04-28 21:12:00 - Responded to practices@ email agreeing with findings, and committed to revoking surfaced intermediates within 7 days per BR 4.9.1.2.5

2025-04-29 16:00:00 - Prepared Revocation Ceremony Documentation

2025-05-02 16:30:00 - Performed Revoke Ceremony on G5 Cross sign certs with G2

2025-05-02 20:35:49 - Published Updated CRLs

2025-05-03 01:00:00 - Verified certificates as Revoked.

2025-05-14 22:30:00 – Conducted destruction ceremony of G5 trust material (root, intermediates etc) in preparation for creating new roots more in line with recent root store policies.

2025-05-23 18:25:00 – Continued review of incident uncovered 2 additional intermediates with https in the AIA field. The additional certificates’ key material was previously destroyed (2025-05-14) (Serials f5e3876991fd455fde96803d5e1efd25 28fe7197afd35291207b23dccdeac3ec)

Related Incidents

Bug Date Description
1884714 2024-03-11 09:29 PDT Problem with the Authority Information Access containing an LDAP-URL on Subscriber Certificates in the id-ad-caIssuers
1908128 2024-07-16 06:28 PDT Problem with the content of the id-ad-ocsp url (pointing to CA OCSP URL rather than subscriber URL) actual format of URL was OK
1965459 2025-05-09 04:36 PDT Problem where the id-ad-caIssuers URL for S/MIME subscriber certificates contained an OCSP URL rather than one for an issued certificate

We reviewed these Bugzilla reports from other CAs. Their focus was on the authorityInfoAccess field in Subscriber Certificates. We did not extend that perspective to evaluate our CA certificate ceremony processes, a step that would have helped us identify this issue earlier.

Root Cause Analysis

Contributing Factor #: Human error changing the reviewed CA Intermediate and Cross Signed Profiles

  • Description: Reviewed intermediate and cross signed CA certificate profiles were erroneously changed to use https URLs in the AIA field. This change led to the misconfiguration of the Boulder CA tool used, allowing the mis-issuance.

  • Timeline: 2024-01-24 17:46:00, 2024-06-05 20:30:00

  • Detection: Manual review of the ceremony prep work

  • Interaction with other factors: Boulder misconfiguration

Contributing Factor #: Boulder misconfiguration

  • Description: Linting rules relating to Authority Information Access were erroneously added to the lint skip list of the Boulder ceremony tool used to generate the production CA certificates.

  • Timeline: 2024-04-26 20:00:17

  • Detection: Manual review of CA ceremony profiles and tool configuration

  • Interaction with other factors: Linting false positive

Contributing Factor #: Linting false positive

  • Description: Additional linting completed with certlint failed to detect the https URL in the AIA field, giving the engineering team false support that the CA certificates were compliant.

  • Timeline: 2024-06-05 @ 13:30:00

  • Detection: Manual review and testing of certificates against certlint

  • Interaction with other factors: Boulder misconfiguration

Lessons Learned

  • What went well: Certificates were revoked promptly upon discovery. Affected CA certificates were never used for signing subscriber certificates..

  • What didn’t go well: Missed requirement for AIA field to include HTTP only URLs and human error in changing reviewed CA certificate profiles.

  • Additional: Review of bugzilla issues need to be completed from a lens of both CA certificates and subscriber certificates regardless of the impacted certificate type from the report.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Revoke CA Cross Subordinate Certificates Resolve Primary Incident Certificates properly revoked 2025-05-02 Complete
G5 Intermediates Key Destruction Resolve Primary Incident Key material destroyed 2025-05-14 Complete
Update the Bugzilla incident review process Mitigate Contributing factor Reviews that include lens of both Subscriber certificates and CA Certificates 2025-05-28 Complete
Propose a patch certlint to catch HTTP url for CA certificates Prevent Contributing factor Generation of PR on https://github.com/certlint/certlint 2025-07-06 Ongoing
Introduce additional linters into ceremony procedures Mitigate Contributing factor PKIMetal as part of ceremony lint suite 2025-07-01 Ongoing
Update the CA certificate profile review and approval process Mitigate Contributing factor Documented CA certificate profile review/approval process 2025-07-01 Ongoing

Appendix

Certificate: sf_int_cross-g2g5.cert.pem
https://crt.sh/?id=13420499398
SHA-256: Fingerprint 29E502995310403685DD6258F9D04856878963042D4749C6D5D9773A310C31C0
Subject: C=US, O=Starfield Technologies, Inc., CN=Starfield Secure Certificate Authority – G5
Issuer: C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Root Certificate Authority – G2
Not before: Jun 6 000000 2024 GMT
Not after: Jun 21 000000 2042 GMT
Serial: 32717d587b580b50b82b38a0f2084a80
Is revoked: Yes
Revocation: date 2025-05-01
Revocation: reason 4 (SUPERSEDED)

Certificate: gd_int_cross-g2g5.cert.pem
https://crt.sh/?id=13420482403
SHA-256 Fingerprint: 880FBC44FE71AA1BE40DB130917B1BAEC1BD889B2B3D7478EDA04728667E8B44
Subject: C=US, O=GoDaddy Inc., CN=GoDaddy Secure Certificate Authority - G5
Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., CN=Go Daddy Root Certificate Authority – G2
Not before: Jun 6 000000 2024 GMT
Not after: Jun 21 000000 2042 GMT
Serial: 9f6c551847cdc100e72bb4183d854931
Is revoked: Yes
Revocation: date 2025-05-01
Revocation: reason 4 (SUPERSEDED)

Certificate: gd_issuing_ca-g5.cert.pem
https://crt.sh/?id=13420507374
SHA-256 Fingerprint: E7F61F147D8C9A38F20D27C5C0EBD5CD0845F905C15C52C8634160AA9F802951
Subject: C=US, O=GoDaddy Inc., CN=GoDaddy Secure Certificate Authority - G5
Issuer: C=US, O=GoDaddy Inc., CN=GoDaddy Root Certificate Authority - G5
Not before: Jun 6 000000 2024 GMT Not after: Jun 21 000000 2042 GMT
Serial: F5E3876991FD455FDE96803D5E1EFD25
Is revoked: Key material destroyed

Certificate: sf_issuing_ca-g5.cert.pem
https://crt.sh/?id=13420490983
SHA-256 Fingerprint: 0090762B916EBD8CD3A771A4C199391823F75C41F91F8D5E2C3C9D27D6EA441A
Subject: C=US, O=Starfield Technologies, Inc., CN=Starfield Secure Certificate Authority - G5
Issuer: C=US, O=Starfield Technologies, Inc., CN=Starfield Root Certificate Authority - G5
Not before: Jun 6 000000 2024 GMT
Not after: Jun 21 000000 2042 GMT
Serial: 28FE7197AFD35291207B23DCCDEAC3EC
Is revoked: Key material destroyed

We continue to monitor this thread for any questions and work on our outstanding action items for this incident.

We continue to monitor this incident for questions and are working on the remaining action items.

Whiteboard: [ca-compliance] [ca-misissuance] → [ca-compliance] [ca-misissuance] Next update 2025-07-01

We continue to monitor this incident for questions. Here's an update on our action items -

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Revoke CA Cross Subordinate Certificates Resolve Primary Incident Certificates properly revoked 2025-05-02 Complete
G5 Intermediates Key Destruction Resolve Primary Incident Key material destroyed 2025-05-14 Complete
Update the Bugzilla incident review process Mitigate Contributing factor Reviews that include lens of both Subscriber certificates and CA Certificates 2025-05-28 Complete
Propose a patch certlint to catch HTTP url for CA certificates Prevent Contributing factor Generation of PR on https://github.com/certlint/certlint 2025-07-06 Ongoing
Introduce additional linters into ceremony procedures Mitigate Contributing factor PKIMetal as part of ceremony lint suite 2025-07-01 Complete
Update the CA certificate profile review and approval process Mitigate Contributing factor Documented CA certificate profile review/approval process 2025-07-01 Complete
Whiteboard: [ca-compliance] [ca-misissuance] Next update 2025-07-01 → [ca-compliance] [ca-misissuance] Next update 2025-07-08

Update on our action items -

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Revoke CA Cross Subordinate Certificates Resolve Primary Incident Certificates properly revoked 2025-05-02 Complete
G5 Intermediates Key Destruction Resolve Primary Incident Key material destroyed 2025-05-14 Complete
Update the Bugzilla incident review process Mitigate Contributing factor Reviews that include lens of both Subscriber certificates and CA Certificates 2025-05-28 Complete
Propose a patch certlint to catch HTTP url for CA certificates Prevent Contributing factor Generation of PR on https://github.com/certlint/certlint 2025-07-06 Complete - https://github.com/certlint/certlint/pull/24
Introduce additional linters into ceremony procedures Mitigate Contributing factor PKIMetal as part of ceremony lint suite 2025-07-01 Complete
Update the CA certificate profile review and approval process Mitigate Contributing factor Documented CA certificate profile review/approval process 2025-07-01 Complete

If everything is complete, then we suggest that GoDaddy file an Incident Closure Summary.

Whiteboard: [ca-compliance] [ca-misissuance] Next update 2025-07-08 → [ca-compliance] [ca-misissuance]

Report Closure Summary

  • Incident description: Subordinate CA certificates created with HTTPS URL in the CA Certificate Authority Information Access field
  • Incident Root Cause(s): Human error, boulder misconfiguration and a missing linter rule from certlint all contributed to HTTPS URLs being used in the AIA field
  • Remediation description: Impacted certificates were revoked
  • Commitment summary: GoDaddy continues to work on improving our validation and procedures around ceremonies for CA certificates.

All Action Items disclosed in this report have been completed as described, and we request its closure.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-24.

Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [ca-misissuance] → [close on 2025-07-24] [ca-compliance] [ca-misissuance]
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Flags: needinfo?(incident-reporting)
Resolution: --- → FIXED
Whiteboard: [close on 2025-07-24] [ca-compliance] [ca-misissuance] → [ca-compliance] [ca-misissuance]

(In reply to Steven Deitte from comment #14)

  • Incident Root Cause(s): ... a missing linter rule from certlint ... contributed to HTTPS URLs being used in the AIA field

https://www.sectigo.com/resource-library/root-causes-437-dont-blame-the-linter

You need to log in before you can comment on or make changes to this bug.