1605804 - GoDaddy: Domain Validation Reuse Issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Joanna]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On Tue 12/17/2019 4:51 PM, we identified a potential issue with a specific DV certificate discovered during the course of an internal 3% audit. On Wed 12/18/19 11 AM we confirmed that this specific DV certificate had been validated more than 825 days prior to issuing the certificate.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

12/17/19 4:51 PM Issue reported to compliance and investigation/research began.
12/18/19 11:00 AM Issue confirmed by compliance and engineering, work on patch began.
12/18/19 12:00 PM Contacted the customer via phone and email in an attempt to have customer rekey.
12/18/19 12:40 PM Worked with customer to issue new certificate with new validation.
12/18/19 4:30 PM Certificate successfully revoked.
12/18/19 8:00 PM Patch deployed.
12/18 - 12/20 Worked on querying database for all affected certificates then reviewing and confirming results with multiple sources.
12/19/19 10:00 AM List of affected NON-EV SSL’s confirmed, began customer outreach and revocation process.
12/20/19 9:00 AM List of affected NON-EV SSL’s revoked.
12/20/19 9:00 AM List of affected EV SSL’s confirmed, began customer outreach and revocation process.
12/21/19 8:30 AM List of affected EV SSL’s revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped issuing certificates with this problem on 12/18/19.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Total 35 certificates - 19 total EV / 16 NON-EV. Issued within the range of 4/5/18 3:55 PM to 12/16/19 1:06 PM.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Serial Number: 6406809394594968347
https://crt.sh/?id=442870131
https://crt.sh/?id=489484376
https://crt.sh/?id=410224165
https://crt.sh/?id=414883764
https://crt.sh/?id=378724880
https://crt.sh/?id=461974100
https://crt.sh/?id=451555545
https://crt.sh/?id=454131534
https://crt.sh/?id=484036234
https://crt.sh/?id=484033695
https://crt.sh/?id=484036234
https://crt.sh/?id=553396991
https://crt.sh/?id=543540358
https://crt.sh/?id=629232585
https://crt.sh/?id=738771897
https://crt.sh/?id=1119942885
https://crt.sh/?id=1352189208
https://crt.sh/?id=1443335784
https://crt.sh/?id=1657741978
https://crt.sh/?id=1666858603
https://crt.sh/?id=1722448919
https://crt.sh/?id=1722448919
https://crt.sh/?id=1955371905
https://crt.sh/?id=1931475287
https://crt.sh/?id=1940913045
https://crt.sh/?id=1865399221
https://crt.sh/?id=2027843082
https://crt.sh/?id=1980889610
https://crt.sh/?id=2057821515
https://crt.sh/?id=2051562836
https://crt.sh/?id=2066408281
https://crt.sh/?id=2122993498
Serial number: 15887854598594840708
Serial number: 3718715569795099077

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It was discovered that the code specific to domain-validation-reuse determined timeframes based on logic that was time-of-validation to time-of-certificate-request instead of time-of-validation to time-of-certificate-issuance.

The time from validation to issuance is typically a few seconds up to a few hours, which includes the automated check to verify the age of the domain validation documentation. Requests that require additional manual vetting are typically processed within a few days but are allowed up to 45 days to process. In roughly 0.0006% of cases, the domain validation documentation expired while waiting an extended period for additional vetting to complete.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have already implemented a patch that adds verifying timeframes to a linter so that prior to issuance we will reexamine the time between validation and issuance and stop the certificate from issuing if it is in excess of guidelines. We are also open to other options and exploring different ways of solving this issue going forward.

Comment 1

[Wayne Thayer]

Joanna: thank you for this incident report. Can you explain more about the root cause of the problem? How was it introduced back in 2018 and what steps can be taken to prevent similar issues from occurring in the future.

Comment 2

[Joanna]

GoDaddy acknowledges the inquiry and we will work to have a response to the community by January 7th. Please note that we may need some additional time due to the holidays.

Comment 3

[Joanna]

We researched and determined the program logic to determine when to perform checks within the vetting and issuance process predated the change in reuse periods. While we have change tickets, we would be unable to state the exact nature of what was discussed amongst the team at the time to explain what occurred. However, we have significantly increased our diligence in the PKI space over the last few years via audits and other similar efforts so that we carefully scrutinize our current environment to find and remove such issues, and, likewise to implement policies, requirements and procedures to ensure such things from being introduced at all in the future. For instance, in this issue we added an appropriate check to a linter. In the past year, we implemented a compliance function within the organization. This function is closely aligned with development efforts to provide additional oversight of changes, which has proven effective at identifying, correcting, and implementing appropriate checks for these types of issues. We are confident that our continued efforts will help to prevent such incidents from occurring again in the future.

Comment 4

[Ben Wilson]

I don't have any further questions. Does anyone else have any?