1909948 - GoDaddy: Edge Case for Data Reuse Outside of Timeframes

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[[:nickname] Star]

Preliminary Incident Report

Summary

During a recent 3% Audit conducted on 2024-07-22 17:49:00 UTC, a UCC certificate was uncovered that had used 2 prior completed domain validations which were older than 398 days at the time of issuance. This was a violation of Section 4.2.1 (Performing identification and authentication functions) of the CAB Baseline Requirements which state:

For validation of Domain Names and IP Addresses according to Section 3.2.2.4 and 3.2.2.5, any data, document, or completed validation used MUST be obtained no more than 398 days prior to issuing the Certificate.

Once the offending certificate was uncovered (https://crt.sh/?id=13671722434), it was revoked on 2024-07-23 17:25:00 UTC within the 24 hour timeframe specified in CAB 4.9.1.1 Reasons for Revoking a Subscriber Certificate:

5. The CA obtains evidence that the validation of domain authorization or control for any Fully-Qualified Domain Name or IP address in the Certificate should not be relied upon (CRLReason #4, superseded).

As part of our investigation, we uncovered a bug in our code on 2024-07-24 22:30:00 UTC. We deployed a fix to production at 2024-07-25 06:43:00 UTC. Our current analysis shows the impact of this incident is limited to a specific set of edge-case conditions. We are in the process of analyzing existing certificates and will revoke any impacted certificates within the appropriate timeframe. This serves as our Preliminary Incident Report. We will publish a full incident report before Monday, 8/5/2024.

Comment 1

[[:nickname] Star]

Incident Report

Summary

During a recent 3% Audit conducted on 2024-07-22 17:49:00 UTC, a DV UCC certificate was uncovered where 2 of the 58 domains on the certificate (careers.sfd.gov.sa & careers.jso.gov.sa) used prior completed domain validations which were older than 398 days at the time of issuance. This was a violation of Section 4.2.1 (Performing identification and authentication functions) of the CAB Baseline Requirements which state:

For validation of Domain Names and IP Addresses according to Section 3.2.2.4 and 3.2.2.5, any data, document, or completed validation used MUST be obtained no more than 398 days prior to issuing the Certificate.

We revoked the mis-issued certificate within the required 24-hour window and simultaneously investigated. Our investigation identified a bug in a section of our code that checks whether domains that were designated as being able to use prior domain validation when the certificate was requested can still use that validation at the time of issuance. The case where the bug would occur was complex and was only present on 10 additional active certificates. We revoked these additional 10 certificates within the required 24-hour window following discovery of these additional impacted certificates.

We developed and deployed a fix on 2024-07-25 06:43:00. There have been no additional issues since this fix was deployed.

Impact

During the weekly 3% Audit in which the mis-issued certificate was discovered 113269 certificates were checked, and this certificate (https://crt.sh/?id=13671722434) was the only one that was mis-issued (0.00088 %). Additionally of our total active body of certificates, only 0.7% use prior domain validation. Once we identified the bug, we surfaced 10 additional active certificates that were issued with prior domain validation outside of 398 days. Each of these additional 10 active certificates was issued before the reported certificate, which was this edge case's last occurrence. All 11 affected certificates were DV UCC certificates.

Timeline

All times are in UTC

2019-12-18 04:09:00 - Additional prior domain validation check added to pre-issuance checks to guard against mis-issuance (which was included as an action taken in Bug 1605804).

2019-12-23 23:39:00 GoDaddy posts Bug 1605804, GoDaddy: Domain Validation Reuse Issue which impacted data reuse for domain validated certificates.

2020-07-08 03:11:00 – Expansion of pre-issuance check effort starts to touch prior domain validation age check code.

2020-12-07 04:18:00 – Pre-issuance domain validation age check refactored to operate on a list of domain entities rather than a single domain entity. New bug introduced.

2021-06-02 00:00:00 – CAB SC42 limiting prior domain validation age to 398 days takes effect

2024-07-22 17:49:00 – Internal 3% Audit reports potential mis-issued certificate

2024-07-22 17:53:00 – PKI team begins review of mis-issued certificate report

2024-07-22 18:31:00 – PKI team confirms mis-issuance of certificate

2024-07-22 20:44:00 – PKI team launches investigation into cause of mis-issuance

2024-07-23 17:25:00 – Reported mis-issued certificate revoked

2024-07-24 22:30:00 – Bug found in pre-issuance domain validation age check logic

2024-07-24 23:19:00 – Stakeholders meet to align on action items

2024-07-25 00:21:00 – Work on bug fix and surfacing commences

2024-07-25 06:43:00 – Bug fix and additional unit tests rolled out to production

2024-07-25 21:02:21 – Additional 10 active mis-issued certificates surfaced.

2024-07-25 22:23:58 – The 10 surfaced certificates were revoked

Root Cause Analysis

Background

GoDaddy’s PKI application allows for prior domain validation to be used in a subset of the Domain Authorization methods the BRs permit in Section 3.2.2.4 (Validation of Domain Authorization or Control), for example 3.2.2.4.7 DNS Change & 3.2.2.4.4 Constructed Email to Domain Contact. To adhere to the prior domain validation requirement in Section 4.2.1, there are two places in our code where that requirement is checked.

The first is when a request is submitted. At that point, we check to see if prior domain validation is usable for any of the domains on the certificate, and we attach a reference to the prior domain validation for any domains where that applies. For any domains that cannot use prior domain validation, we proceed with the normal Domain Authorization process. When all the validation on a certificate has been completed the certificate moves to pre-issuance.

Pre-issuance is the second place in the code where we check that the prior domain authorization (which was valid at the time of submission) is still valid. This was added in conjunction with the incident remediation efforts related to Bug 1605804. If the certificate passes all the checks, including the one for prior domain validation, the certificate moves towards issuance.

The logic for the checks in those two places is separate and both functioned correctly after the changes in December 2019. However, our 3% audit on 2024-07-22 17:49:00 uncovered a mis-issued certificate, our investigation discovered a new bug had been introduced to the pre-issuance check for prior domain validation in December 2020 as we were adding additional checks to our pre-issuance linter. During the refactoring we converted the check method from taking a single domain argument within a larger loop to iterating over all domains on the certificate using prior domain validation. In doing so, we introduced a condition where this check could unintentionally return early from the loop.

The new bug required multiple conditions to be true before it would lead to a mis-issuance, including the following:

1. The certificate is a UCC certificate

2. Multiple domains are validated using prior domain validation which is within 395 days at the time the request was made (we use 395 rather than the BR’s 398 to help reduce the risk of prior domain validation expiring mid-flight). The prior domain validation comes from more than one source (ex. Some domains are validated by a prior domain validation of example.com; others are validated by a prior domain validation of anotherdomain.net)

3. At least 3 full days pass from when the request was made to when all validation is completed, and the certificate moves to pre-issuance.

4. One or more of the domains on which the prior domain validation was based becomes older than 398 days during that time.

If any of the other pre-issuance checks returned as invalid, issuance would be halted. However, if the prior domain validation was the only problem with the certificate, issuance would continue. This caused 11 certificates (see Impact section) to use prior domain validation that was obtained over 398 days from the time of issuance.

The cause of this bug being introduced was due to refactoring the method that does this check from operating on single domains within a larger loop to operating on a list of domains in a separate loop. When that refactoring was done, the existing unit tests were insufficient to catch the change in behavior. Because of the number of conditions necessary to cause a mis-issuance due to this bug and the extremely limited number of impacted certificates, the bug remained undetected until our most recent 3% audit.

Missed Requirement

Section 4.2.1 requirements around prior domain validation were clear:
For validation of Domain Names and IP Addresses according to Section 3.2.2.4 and 3.2.2.5, any data, document, or completed validation used MUST be obtained no more than 398 days prior to issuing the Certificate.

The implementation of our code in the pre-issuance logic allowed this requirement to be missed. The requirement was not missed due to lack of understanding or confusion, but nonetheless, this edge case was not caught by testing or peer review due, at least in part, to the number of conditions necessary to cause a mis-issuance.

Requirement Fix

Additional unit tests have been added to the code to test this complex scenario and cause it to fail if the prior domain validation pre-issuance check does not behave as expected. We also have refactored the logic iterating over the list of domains to prevent premature short-circuiting leading to a false-positive valid return. The logic can only short-circuit to false, meaning at worst, we will not issue a certificate we should, rather than mis-issuing a certificate we shouldn’t

Deployment

Work on the fix began shortly after the bug was discovered and the fix was deployed to production on 2024-07-25 06:43:00. This included:

• refactoring the method where the bug was present to operate correctly
• including additional unit tests to verify that behavior under more complex conditions

The last mis-issuance due to the bug was on 2024-07-09 and since the deployment, there have been no further occurrences.

Lessons Learned

What went well

Doing weekly 3% Audits uncovered an issue that could have gone unnoticed

When building the pre-issuance check, a conscious choice was made to not reuse the same code as used for domain age checks in other places. This helped limit the blast radius since that other code did not have any bugs.

What didn't go well

Unit tests on the relevant code in the in pre-issuance checker were insufficient and were based on simpler single domain scenarios.

Where we got lucky

Most of our certificates do not use prior domain validation (only 0.7% of total active body of certificates) and of them, most were not affected by the bug, so the blast radius was extremely small.

Action Items

| Action Item | Kind | Due Date |

| ----------- | ---- | -------- |

| Add additional unit tests to check for this prior domain validation scenario | Prevent | 2024-07-25 |
| Add synthetic monitor tests for more complex prior domain validations scenarios (ex. UCC with prior domain validation from various domains) | Prevent | 2024-10-10 |

Appendix

Details of affected certificates

Revoked due to initial report (2024-07-23 17:25:00)
https://crt.sh?sha256=a73ab4f6abf4bef3bfde5319a9db11f926f09d1fffeb58446f71313850fd4cbd

Revoked after surfacing completed (2024-07-25 22:23:58)
https://crt.sh?sha256=1abcf13f7ce767902e3e28ceb6ea420e6d6a111ec6eea66f53eef3dee21c1ebf
https://crt.sh?sha256=af990fa3b60c5c989dfb494f712ba9885627b2c48bc79e79ea2209a67081eb9b
https://crt.sh?sha256=cd74927ecb74ef6e515f572a8e364983d9b3c64833b74f5ba06dfd3f362ed3b5
https://crt.sh?sha256=ac027b9ee98a816dbcb056d617f37ed04ec7f2a2dd3829ef0dc4a329c9f19131
https://crt.sh?sha256=219cd86f9d4c0475ceb8f346a1262fd0873674a02b3ebb1271a07233a20e841e
https://crt.sh?sha256=7fd1b4177ccebb5b68e13b72e8fb957fbd388cf4b53d3f142fbec9d66cb76cb4
https://crt.sh?sha256=e489ea653ef5f06d9613610b1da5cfb3e41179081d875e25b8da12df42ff26d9
https://crt.sh?sha256=d46ffc57888fb0fa282b3dd90451a6aaf0f1cd0a9c8892eb2926bd83475899f3
https://crt.sh?sha256=9423b0ce8674ff4878842603ba86384d720a320a88395283646a9ca7759e4ea6
https://crt.sh?sha256=10630e50d84728d8f61976a8044729dc2812de1ada3a07f32eaea5a0c054975f

Comment 2

[[:nickname] Star]

As of 10/04/2024 synthetic monitoring has been deployed and is operating as expected. All action items related to this incident have been completed.

Comment 3

[Ben Wilson]

As there have been no questions or comments, and it appears that all action items related to this incident have been completed, I will close this matter on or about Wed. 30-Oct-2024.