1904749 - GoDaddy : CAA checks passed when records contained incorrect variants of godaddy.com or starfieldtech.com

Assignee

Description

•

5 months ago

Preliminary Incident Report

Summary

On 6/23/2024, GoDaddy received a certificate problem report alerting us to potential concerns with CAA checking. As a result of the certificate problem report investigation, GoDaddy identified a software bug in the CAA process which allowed CAA validation to pass when CAA record values contained variants of the terms 'godaddy.com' or 'starfieldtech.com' rather than the matching the exact term which does not conform to the syntax of ‘issuer-domain-name’ as defined in RFC 8659, section 4.2. Non-conformance with RFC 8659 is a violation of the Baseline Requirements for the Issuance and Management of Publicly -Trusted TLS Server Certificates, section 3.2.2.8, which states "CA MUST retrieve and process CAA records in accordance with RFC 8659 for each dNSName in the subjectAltName extension."
A full incident report will be published by Friday, 7/5/2024.

Ben Wilson

Updated

•

5 months ago

Assignee: nobody → star

Status: UNCONFIRMED → ASSIGNED

Ever confirmed: true

Whiteboard: [ca-compliance]

Pouyan Fotouhi Tehrani

Comment 1

•

5 months ago

We observed and reported this bug alongside bug 1904748 as part of a regular scanning of CT logs and checking CAA records for all SANs in published certificates.

Observation

GoDaddy accepts values that contain godaddy (case insensitive) as issuer-domain-name even if it doesn't conform to the syntax described in RFC 8659 or it differs from the valid domain names specific in GoDaddy/Starfield CPS (i.e., godaddy.com, starfieldtech.com).

Examples

Certificate Digest	SAN	CAA
`9bf16ef6e33cfaf1d80ebf526bc3907fb98ee120`	`[sbc.capstarsolutions.com, www.sbc.capstarsolutions.com`]	`{"issue":["comodoca.com","digicert.com; cansignhttpexchanges=yes",``"issuedomainname:godaddy.com",``"letsencrypt.org", "pki.goog; cansignhttpexchanges=yes"], "issuewild":["comodoca.com", "digicert.com; cansignhttpexchanges=yes", "letsencrypt.org", "pki.goog; cansignhttpexchanges=yes"]}`
`3567f7db4b5e71de6bba9beb86f9d6f9807e02e5`	`[*.canadianfeedthechildren.ca, canadianfeedthechildren.ca]`	`{"issue":["comodoca.com", "digicert.com; cansignhttpexchanges=yes", "letsencrypt.org","pki.goog; cansignhttpexchanges=yes"], "issuewild":["comodoca.com", "digicert.com; cansignhttpexchanges=yes",``"https://www.godaddy.com/"``, "letsencrypt.org", "pki.goog; cansignhttpexchanges=yes"]}`
`f6e4ca5d613f2875e9567ffc31b4fc77c68edb36`	`[*.cito.borngroup.com, cito.borngroup.com]`	`{"issue":[``"GoDaddy"``, "comodoca.com", "digicert.com; cansignhttpexchanges=yes", "letsencrypt.org", "pki.goog; cansignhttpexchanges=yes"], "issuewild":[``"GoDaddy"``, "comodoca.com", "digicert.com; cansignhttpexchanges=yes", "letsencrypt.org", "pki.goog; cansignhttpexchanges=yes"]}`

In the first two examples the value contains valid godaddy.com identifier, yet it does not conform to the issuer-domain-name syntax.
Second example is also related to bug 1904748 as it includes both FQDN and wildcard, yet GoDaddy is mentioned only in issuewild.
Last example only lists GoDaddy as value and does not conform to identifiers defined in CPS

[:nickname] Star

Assignee

Comment 2

•

5 months ago

Incident Report

Summary

GoDaddy and Starfield certificates were issued when CAA records contained, in some variation, the term 'godaddy.com' or ‘starfieldtech.com’ but they did not conform to the syntax of issuer-domain-name as defined in RFC 8659:

https://www.rfc-editor.org/rfc/rfc8659#name-caa-issue-property

For example, if a certificate was requested for site.com and where the relevant CAA record is as follows:

site.com. 3600 IN CAA 0 issue "issuedomainname:godaddy.com"

a godaddy.com certificate would still be issued for the FQDN (site.com). The value above (issuedomainname:godaddy.com) is neither a valid domain name, nor is it mentioned as a valid identifier in the GoDaddy CP/CPS documents.

Impact

During the incident, we identified 168 active certificates (DV 164, OV 3, EV 1) that had this problem. We determined that the first mis-issuance happened on 2017-09-23 and the last one was on 2024-06-22. During this period there were an additional 793 certificates which would have required revocation if they had not already been revoked or expired. During the same time period we issued approximately 124.3M certificates overall. The code fix was deployed on 2024-06-26.

On 2024-06-28, we revoked 168 active certificates for this issue (see Appendix for detailed listing)

Timeline

All times are UTC.

2013-01-01 00:00 - RFC 6844 defining CAA is published

2015-01-15 00:00 - GoDaddy publishes CP/CPS 3.6 explaining in section 4.1.1 that CAA records are currently not checked under RFC 6844, but may be checked in the future

2017-08-09 00:42 - GoDaddy deploys CAA logic into main application

2017-08-15 00:00 - GoDaddy publishes CP/CPS 3.12 and details how CAA records are checked in Section 4.1.1

2017-09-08 00:00 - CAB Forum Releases TLS BR 1.5.8 which specifies CAs must check for CAA Records in section 3.2.2.8

2017-09-26 11:31 - GoDaddy issues first certificate with improper check of CAA record syntax (Serial: fa632daaf9892853)

2019-08-27 07:53 – GoDaddy refactors elements of CAA checking functionality into a microservice

2019-11-01 00:00 - RFC 8659 published, which included some clarifications to RFC 6844

2024-06-22 03:05 – GoDaddy issues last certificate with syntax problem (Serial: 217a29e4a8f3f638)

2024-06-23 13:54 - GoDaddy receives CPR email

2024-06-23 16:23 – GoDaddy Registration Authority (RA) Administrator notates the email/ticket as an CPR concerning CAA records and requests supervisor support.

2024-06-23 16:44 – GoDaddy RA Supervisor contacts the PKI Development Team alerting to possible TLS BR violations

2024-06-23 17:24 – GoDaddy responds to CPR reporter, providing update of investigation starting based on the provided information.

2024-06-24 15:08 – GoDaddy PKI Development Team starts reviewing issue and impact

2024-06-24 18:46 - GoDaddy PKI Development Team completes review and confirms the issue raised is a valid compliance issue

2024-06-24 19:59 – GoDaddy PKI Development Team presents preliminary findings confirming the presence of the bug identified in the CPR

2024-06-24 22:00 – GoDaddy PKI Leadership meets and assigns tasks for response and remediation

2024-06-25 00:23 – GoDaddy contacts customers whose certificates appeared in the examples provided in the CPR about pending revocation and steps to update CAA records to permit issuance

2024-06-25 16:16 – GoDaddy runs initial queries to identify potentially affected certificates

2024-06-26 02:14 – GoDaddy creates bug PRB1904749 and posts a Preliminary Incident Report to BugZilla

2024-06-26 18:31 – GoDaddy deploys fix to production

2024-06-27 20:28 – GoDaddy confirms final list of problematic active (see appendix below)

2024-06-28 03:51 - GoDaddy begins contacting customers whose certificates were identified during investigation via email and/or phone to inform them of pending revocations

2024-06-28 12:00 – GoDaddy revokes problematic certificates reported in the CPR (within 5 days of receipt of the CPR from 2024-06-23)

2024-06-28 22:08 – GoDaddy revokes problematic certificates surfaced during investigation (within 5 days of confirming the bug in the system on 2024-06-24 19:59)

HH:MM Example

Root Cause Analysis

Background

GoDaddy’s original implementation of the CAA record check was written to follow the requirements outlined in RFC 6844 in August 2017. Incremental changes were gradually included for following alias chains, wildcard record searches, enhance DNSSEC checks, etc. over the following months. The logic for handling the CAA record check was later moved to a microservice dedicated solely for this purpose around August 2019 for better distribution of our internal workload. The general algorithm for determining if a CAA record could be used for issuance involved two separate use cases. The first use case stated that if there were no CAA records, the CA was permitted to issue certs of any type (DV, OV, EV) for the domain. This use case was handled properly, and no issues were found with this workflow. The second use case was applied whenever one or more CAA records existed. Each CAA record issuer-domain-name was searched for the CA identifiers “godaddy.com” or “starfieldtech.com” as specified in the CP/CPS. The syntax of the issuer-domain-name search did not apply the formatting rules outlined in RFC 8659 which were later added to RFC 6844. Instead, a more general search for just the CA identifier in the issuer-domain-name was performed. This misinterpretation of 6844 requirements allowed text patterns such as "issuewild:godaddy.com" to be incorrectly evaluated as valid.

Missed Requirement

Misinterpretation of the grammar of the issuer-domain-name field in the original RFC 6844 Section 5.2 and subsequent missed opportunity to reevaluate when clarity was added to RC 8659 Section 4.2 in 2019.

Failure to programmatically enforce the explicit strings as stated in our CP/CPS

Requirement Fix

The bug fix included changes which brought the microservice into compliance with RFC 6844 and 8659. A more detailed CAA issue-value parser based on the ABNF grammar in RFC 8659 in Section 4.2 was added. It will reject values that do not conform to that grammar, such as ‘issuewild:godaddy.com’ . Additional unit tests were also written per RFC requirement which use a broader set of test data and verification checks to ensure compliance.

Deployment

The rollout of the fix was deployed two days before revocation of the affected certs to allow time for verifying the change. CAA record audit data before and after the incident was then analyzed for additional violations using database surfacing queries. No additional violations were found.

Lessons Learned

What went well

Certificate problem reporting process worked. A report was received which included a potential issue and examples and upon investigation specific findings were validated and bugs sourced within 24 hours of receipt.

Our logging of data used in CAA validation processing allowed us to narrow the scope of impacted certificates and having the data available allowed us to identify and respond swiftly.

Issue remediation allowed us the opportunity to enhance the parsing and validation logic to include tokenized evaluation of the issue property values. Ordering of each character is now tested for RFC 8569 compliance.

What didn't go well

RFC 8569 included examples of proper validation and parsing of the issuer-domain-name found in DNS CAA resource records. At the time we did not have a sufficient process to catch the updates in RFC 8569

Where we got lucky

Less than 1% of our customers (~ 338K domains) appear to have non-empty CAA records on their domains. This helped limit the negative effects of the mis-issuance.

GoDaddy DNS has validation checks which prohibit the usage of invalid data in a CAA Resource record. Issuer domain names are checked to ensure they are FQDNs and issuer-domain-name values are checked for RFC 8659 formatting. If these checks were not in place, customers may have been more prone to create invalid entries.

Action Items

| Action Item | Kind | Due Date |

| ----------- | ---- | -------- |

| Add additional unit tests to check CAA records scenarios | Prevent | 2024-06-26 |

| Add synthetic monitor tests to validate our system is correctly detecting CAA records which prevent issuance | Prevent | 2024-10-10 |

Appendix

Details of affected certificates

Wayne

Comment 3

•

5 months ago

Did GoDaddy at any point stop issuance? The Impact and Timeline sections implies issuance continued despite full knowledge of the issue.

Flags: needinfo?(star)

Pouyan Fotouhi Tehrani

Comment 4

•

5 months ago

We also observed cases, where issue or issuewild value was not even a domain name (see f6e4ca5d613f2875e9567ffc31b4fc77c68edb36 in table above).

Since you didn't mention that case, I was wondering if your algorithm also accepted patterns including only variations of godaddy (w/o .com label)?

[:nickname] Star

Assignee

Comment 5

•

5 months ago

(In reply to Wayne from comment #3)

Did GoDaddy at any point stop issuance? The Impact and Timeline sections implies issuance continued despite full knowledge of the issue.

Thank you for your question. No – we did not stop issuance. Once we isolated the bug, we focused efforts on the bug fix and subsequently helping affected customers through the rekey process in a compressed time window. We were confident in our ability to expeditiously identify and revoke any that may have been issued between the time we confirmed the bug and the time the bug was fixed. In addition, our investigation determined that this specific issue was extremely rare and that it was likely a very limited number or zero certificates would be affected between the time we confirmed the issue and applied the fix. Our assessment was, in fact, correct as no certificates were affected after the issue was confirmed. We elected to continue issuing certificates during this period, because stopping issuance would have been extremely impactful and disruptive to our customers whose certificates were not affected. Ultimately, we selected the path that would result in the least disruption, while still being compliant with the requirements of the BRs and our CP/CPS.

Flags: needinfo?(star)

[:nickname] Star

Assignee

Comment 6

•

5 months ago

(In reply to Pouyan Fotouhi Tehrani from comment #4)

We also observed cases, where issue or issuewild value was not even a domain name (see f6e4ca5d613f2875e9567ffc31b4fc77c68edb36 in table above).

Since you didn't mention that case, I was wondering if your algorithm also accepted patterns including only variations of godaddy (w/o .com label)?

Thank you for the question. It did not accept variations without the .com label. The bug in our code was looking for a full string of godaddy.com or starfield.com, but was looking for a “contains” rather than an exact match.

Ben Wilson

Comment 7

•

3 months ago

Please provide an update on "Add synthetic monitor tests to validate our system is correctly detecting CAA records which prevent issuance"

Flags: needinfo?(star)

[:nickname] Star

Assignee

Comment 8

•

3 months ago

Thank you for the question, Ben. We are actively working on adding the synthetic monitor tests to validate detection of CAA records which prevent issuance. We expect the rollout to be ahead of our defined timeline in early October and will update the incident once complete.

Flags: needinfo?(star)

Ben Wilson

Updated

•

2 months ago

Whiteboard: [ca-compliance] → [ca-compliance] [dv-misissuance] [ov-misissuance] [ev-misissuance]

[:nickname] Star

Assignee

Comment 9

•

2 months ago

As of 9/3/2024, synthetic monitoring has been deployed and is operating as expected. All action items related to this incident have been completed.

Ben Wilson

Comment 10

•

29 days ago

As there have been no questions or comments, and it appears that all action items related to this incident have been completed, I will close this matter on or about Wed. 30-Oct-2024.

Flags: needinfo?(bwilson)

Ben Wilson

Updated

•

23 days ago

Status: ASSIGNED → RESOLVED

Closed: 23 days ago

Flags: needinfo?(bwilson)

Resolution: --- → FIXED