1647030 - GoDaddy: Agreed-Upon Website Domain Validation Issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Daniela Hood]

Attachment: Domain Validation - Problematic Certificates.xlsx

1- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On Wednesday, June 10 around 4:00 PM a developer performing system updates identified a potential bug whereby website control validation information one sub-domain was automatically used to validate a second sub-domain with the same primary level domain.

2- A timeline of the actions your CA took in response. A timeline timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2020-05-27 A newer developer on the team worked a change request related to BR section 3.2.2.4.18 and introduced a bug. See section 6 for details.
2020-06-10 4:00 PM Developers became aware of the bug
2020-06-10 6:53 PM Population of certificates affected defined as 454
2020-06-10 8:40 PM Patch was deployed and verified that there was no increase in population
2020-06-11 12:50 PM Completed 454 revocations

3- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
On June 10 at 8:40 PM we implemented the system fix. On June 11 after processing the revocations we confirmed that no more certificates with this problem were issued.

4- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
​454 Certificates affected that were issued between May 27, 2020 and June 10, 2020.

5- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
A full list of the affected certificates is attached.

6- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.​
On May 27, 2020 a newer developer processed a ticket to update a note on our system to show when domain was being validated via Agreed-Upon Change to Website 2​ (referred to as Website Control v2 or 'WSCv2') in order to be consistent with BR section 3.2.2.4.18. The developer also introduced a new variable for domain validation method WSCv2. Within our system, there is a section of code that only allows Agreed-Upon Change to Website (WSC) from a prior certificate to be used for a current certificate if the prior FQDN matches the new request. The developer did not add the new variable to this method, which caused the system to bypass this check when using the new variable. Even though the change was tested, reviewed and approved per our procedures, the reviewer did not take into account code that should have been changed. While performing an update for an unrelated change on June 10, 2020, a more experienced developer on the team identified the missed update and resulting bug.

7- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.​
We corrected the issue within hours of identifying the bug by making the appropriate change to the pre-verification method. The newer developer and change approver were coached and restrictions put in place to ensure peer reviews are always performed by senior engineers. Additionally, we improved our change request documentation by including an engineering review to ensure requirements are clear prior to assigning the task to be worked.

We added a test to our system that runs every 15 minutes to check for scenarios where Agreed-Upon Change to Website v2 validation was used from a prior certificate, but the FQDN for the new certificate does not match the old. If this test fails, it will generate alerts and notify our on-call personnel to correct the process.​

Comment 1

[Ben Wilson]

1- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On Wednesday, June 10 around 4:00 PM a developer performing system updates identified a potential bug whereby website control validation information one sub-domain was automatically used to validate a second sub-domain with the same primary level domain.

What do you mean by this? Particularly, information from "one sub-domain was automatically used to validate a second sub-domain with the same primary level domain."? Can you provide an example of what you mean there? I looked at the domains of the first several certificates in the spreadsheet list you provided - shop.hippiealien.com, act.remondis.com.au, ha29node4.yb876.com, fluidfile-endo.com, autodiscover.lab70.nicelab.de, onlinestore.damro.lk, store.marounchedid.com, ... - and I'm trying to figure out more about what happened.

6- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.​
The developer also introduced a new variable for domain validation method WSCv2. Within our system, there is a section of code that only allows Agreed-Upon Change to Website (WSC) from a prior certificate to be used for a current certificate if the prior FQDN matches the new request. The developer did not add the new variable to this method, which caused the system to bypass this check when using the new variable.

Do you mean that the check was not added when the new variable was added to the system?

Even though the change was tested, reviewed and approved per our procedures, the reviewer did not take into account code that should have been changed. While performing an update for an unrelated change on June 10, 2020, a more experienced developer on the team identified the missed update and resulting bug.

This means that the reviewer was either careless, inexperienced, or lacked proper training.

Thanks,
Ben

Comment 2

[Daniela Hood]

Hello Ben,

Thank you for your response.

The purpose of the change requested was just to document on our system the change in the BRs. In accordance with section 3.2.2.4.18 of the baseline requirements, we validate the FQDNs by requesting the customer to locate the random value in the "/.well-known/pki-validation" directory​, i.e. a customer would place the random value at shop.hippiealien.com/.well-known/pki-validation​. Also, our system is set to pre-validate that FQDN shop.hippiealien.com if that same exact FQDN has a new certificate request​ within the permitted reuse period of 825 days, so if the customer tried to request a certificate for home.hippiealien.com, they would have to go through the validation process again by adding a new random value to that FQDN, but if they requested a certificate for shop.hippiealien.com, within the proper data reuse period, the system would reused the already confirmed validation of the domain. However in this case, let's say the customer requested the certificate for shop.hippiealien.com and validated by method 3.2.2.4.18, and then a day later went on to rekey that certificate to a different FQDN home.hippiealien.com, the system should request a new validation, in this case specifically with the system bypassing the check, it pre-validated the domain home.hippiealien.com​ understanding to be the same as shop.hippiealien.com.

Yes, the check to the new variable was not added.

The reviewer was inexperienced, therefore moving forward the reviews will be done by only senior engineers, as one of our measures, as previously stated.

Comment 3

[Ben Wilson]

I intend to close this bug on or about 5-Aug-2020 unless additional issues or questions are raised.