1577913 - GoDaddy: Issues with State and Country fields

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Joanna]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Certificate problem report received Monday, August 19, 2019 7:54:39 PM.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Monday, August 19, 2019 7:54:39 PM Certificate problem report received.
Tuesday, August 20, 2019 7 AM Began investigation into details of each certificate listed, contacted affected customers and contacted the problem reporter informing that an investigation has began
Friday, August 23, 2019 10:30 AM Finalized path forward for resolution as described below.
Friday, August 23, 2019 7PM All affected certificates revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Even though the error rate is low when compared with tens of thousands of certificates issued over the range of the time for the certificates surfaced in this report, we agree that it should be zero. We are currently working on identifying an acceptable API source that will meet the requirement for the state/country correlation. Our goal is to find a source that contains both correct spellings of states/provinces and the applicable country ISO code and implement this logic throughout our systems. Our long-term solution consists of implementing an automation solution to ensure the fields are correlated and avoid any possibility of errors.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

7 certificates total from 10/19/2017 3:01 PM to 6/25/19 10:33 AM.
3 of the certificates had issues with the Country not being correct ISO code.
4 of the certificates had issues with the spelling of US State names.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

crt.sh URL(s)", notBefore, notAfter,
"https://crt.sh/?id=235504587 (precert); https://crt.sh/?id=237601342 (final)", 2017-10-19, 2019-11-07,
"https://crt.sh/?id=523818634 (precert); https://crt.sh/?id=529642569 (final)", 2018-06-13, 2020-06-16,
"https://crt.sh/?id=1282665500 (precert); https://crt.sh/?id=1288254507 (final)", 2019-03-13, 2021-03-13,
"https://crt.sh/?id=1612336808 (precert)", 2019-06-25, 2021-06-25,
"https://crt.sh/?id=299740858 (precert); https://crt.sh/?id=302404224 (final)", 2018-01-11, 2020-01-11,
"https://crt.sh/?id=344098570 (precert); https://crt.sh/?id=461180860 (final)", 2018-02-28, 2020-02-28,
"https://crt.sh/?id=510402948 (precert); https://crt.sh/?id=560307052 (final)", 2018-06-06, 2020-06-06,

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We have a couple of system checks in place that verify multiple aspects of the certificates that are issued, such as linters to ensure there are 2 characters in the country field. We have automation in place for a correlation between state and country in our client facing UI, but not in our Validation Specialist systems. In these few cases, our Validation Specialists had to override the automation to manually correct certificate requests. Since we do not have any automation in place for misspelling and correlation between state/country fields on the Validation Specialist system, these were missed. As mentioned above, we are already working on creating more thorough system checks that will identify such mistakes.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

As stated above, we are currently working on identifying an acceptable API source that will meet the requirement for the state/country correlation which we plan to implement on all systems. Once we are able to find a source we can respond with a timeline for integration.

Comment 1

[Ryan Sleevi]

Joanna: Thanks for filing this. A few questions:

1. Was GoDaddy aware of the past discussions of similar issues?
2. Has GoDaddy reviewed those CA’s approaches to data validation and consistency?
3. Will you have a timeline next week? Right now, there’s no clear commitment from GoDaddy about when they will be providing an update or how to measure and quantify GoDaddy’s progress.
4. Why the delay between August 23 and now, and the potentially indefinite delay for further updates? What was GoDaddy doing during this time that prevented filing this issue?

A clearer communication about the discrete steps GoDaddy is taking to evaluate, along with concrete and clear timelines as to when those evaluations will be performed, is necessary here. Given that multiple robust and viable solutions are identified here, I want to make sure we see progress in a timely fashion.

Comment 2

[Joanna]

(In reply to Ryan Sleevi from comment #1)

Joanna: Thanks for filing this. A few questions:

1. Was GoDaddy aware of the past discussions of similar issues?
   If you are referring to items such as https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/8o6-BIuBqDE and https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/GoIXEgl5tB4 then yes, we are following those discussions.

2. Has GoDaddy reviewed those CA’s approaches to data validation and consistency?
   We have reviewed the approaches from those CA’s who have chosen to publically disclose. We are currently reviewing the same API Digicert mentioned in https://groups.google.com/d/msg/mozilla.dev.security.policy/8o6-BIuBqDE/j_GrpdC5AAAJ thread as well as other options.

3. Will you have a timeline next week? Right now, there’s no clear commitment from GoDaddy about when they will be providing an update or how to measure and quantify GoDaddy’s progress.
   There is a bigger conversation happening within the server certificate working group that we believe will affect this. It's in the interest of all CA's to work through this process as a community to identify an acceptable source to avoid throw-away work or potential compliance issues. Our timeline will be dependent on a number of different factors, one of which will include the outcome of the conversations we're having. No, we won't have one next week. Once we as a community can establish some rules around a true source, and we work through our own product roadmap we will and when that happens we will share and update.

4. Why the delay between August 23 and now, and the potentially indefinite delay for further updates? What was GoDaddy doing during this time that prevented filing this issue?
   As for the time between August 23 and the publishing of the report August 30, GoDaddy was researching potential solutions that the whole community could benefit from, preparing this incident report and engaging the Starfield Governance and Policy Committee for review. We believe we responded within the acceptable time frame as defined by the incident response timeline of 2 weeks; if the timeline has changed, and we have missed it, please let us know. The potentially indefinite delay for further updates will be driven by the communities’ ability, or inability, to come to consensus on what an acceptable source will be to resolve this problem. We want to ensure that the problem is eradicated once and for all for both us and the community.

A clearer communication about the discrete steps GoDaddy is taking to evaluate, along with concrete and clear timelines as to when those evaluations will be performed, is necessary here. Given that multiple robust and viable solutions are identified here, I want to make sure we see progress in a timely fashion.

GoDaddy will continue to provide clear communication and will continue to actively work with the community to ensure we are being good stewards to our customers and the web and in complete compliance with both the root stores and the CA/Browser forum.

Comment 3

[Wayne Thayer]

Joanna:

(In reply to Joanna from comment #2)

(In reply to Ryan Sleevi from comment #1)

3. Will you have a timeline next week? Right now, there’s no clear commitment from GoDaddy about when they will be providing an update or how to measure and quantify GoDaddy’s progress.
   There is a bigger conversation happening within the server certificate working group that we believe will affect this. It's in the interest of all CA's to work through this process as a community to identify an acceptable source to avoid throw-away work or potential compliance issues. Our timeline will be dependent on a number of different factors, one of which will include the outcome of the conversations we're having. No, we won't have one next week. Once we as a community can establish some rules around a true source, and we work through our own product roadmap we will and when that happens we will share and update.

As I understand the discussion that's happening, it's about determining the proper value for Locality and stateOrProvince when there is some ambiguity. What does that have to do with misspelling a state name or using a mismatched country code? Other CAs, such as DigiCert and Sectigo, have implemented some level of automated checking for location information that is clearly in error.

The issue currently being discussed in the CAB Forum has been known for years. There is no simple solution, and it may be years before it is solved. Given this, please provide a timeline for remediating the class of errors described in this bug. In lieu of that, I will assume that GoDaddy does not intend to remediate the problem.

4. Why the delay between August 23 and now, and the potentially indefinite delay for further updates? What was GoDaddy doing during this time that prevented filing this issue?
   As for the time between August 23 and the publishing of the report August 30, GoDaddy was researching potential solutions that the whole community could benefit from, preparing this incident report and engaging the Starfield Governance and Policy Committee for review. We believe we responded within the acceptable time frame as defined by the incident response timeline of 2 weeks; if the timeline has changed, and we have missed it, please let us know. The potentially indefinite delay for further updates will be driven by the communities’ ability, or inability, to come to consensus on what an acceptable source will be to resolve this problem. We want to ensure that the problem is eradicated once and for all for both us and the community.

Mozilla's incident response guidelines state "We expect to see incident reports as soon as possible, and certainly within two weeks of the initial issue report." This response does not give me confidence that GoDaddy responded "as soon as possible" as opposed to as late as deemed minimally acceptable.

Comment 4

[Joanna]

(In reply to Wayne Thayer [:wayne] from comment #3)
Wayne:
Thank you for your note. GoDaddy has been actively researching ways to address this. For instance, we reviewed and considered Google’s GeoCode API. Unfortunately, we found some issues in the vetting process that reduced our confidence in its accuracy and raised the risk of introducing new issues and errors. For example:

• API inconsistently returns administrative_area_level_1 or administrative_area_level_2 results for GB. Example, when searching for Bedfordshire, ISO shows "Central Bedfordshire" API returns "England" as the correct entry.
• API adds superfluous information. Example, "State of Sergipe" instead of "Sergipe" in Brazil.
• API appears to translate all results to English, how do we ensure localization will translate correctly across the globe?

We’d love to understand what Mozilla’s and Google’s position would be in these types of scenarios. Depending on the type of scan, and the organization scanning, these could introduce new validation errors.

These examples highlight the complexity associated with choosing a source of truth for this type of information, and why what might appear to be a relatively simple request takes a bit more time of research and analysis.

If Mozilla or Google could provide us the source of truth they would accept moving forward, it would expedite the process and we are probably looking at 3 (2 week) sprints of work to get this into production. This time frame is dependent on the quality of the API’s offered by the chosen service. If not, and we need to do the research, then it’s going to take a bit more time to ensure we do the proper amount of due diligence.

Unfortunately, GoDaddy can’t provide an adequate time frame because we’re unclear of what is available. Our inability to give a concrete answer is not indicative of our lack of desire to fix the problem, but rather the challenges that come the ambiguous environment our community has created.

Comment 5

[Ryan Sleevi]

Joanna: For questions for Google, you can direct them to [e-mail] or on m.d.s.p. Using Bugzilla for that sort of communication has, unfortunately, created too much confusion for CAs in the past.

Personally, think it's reasonable to be concerned that, even in spite of the bugs referenced, GoDaddy's approach to being a CA appears to be to specifically ask Root Programs how to be a CA, rather than to proactively look for ways to improve. We've seen multiple approaches used by other CAs, not simply limited to Google's GeoCode API (for which I have zero knowledge about its accuracy or suitability for purpose), including ISO data sources or that of GeoNames. For the CAs that have responsibly looked as to how to improve their CA operations, they've approached it as a risk management: Their compliance team evaluates possible data sources and suitability, and looks for practices to reduce the risk of human error. For example, if a name comports with something that the compliance team has vetted, then it's allowed. If it doesn't, it might be rejected, or it might be escalated for multi-party review. CAs further have sought to provide assurance about their criteria and approach by being transparent about the datasources they use and how they determine them, including possible revisions to their CP/CPS to be clear on that.

Those approaches seem to better acknowledge the complexities and look for holistic mitigations, with multiple checks and controls, and regular review, to ensure things are correct.

GoDaddy's approach appears to be "We're not sure how to fix this. Will Google and Mozilla tell us what the minimum required is?". Comment #3 focused on specific issues that entirely avoid the ambiguity issues being highlighted above. For example, it does not provide assurance that GoDaddy doesn't know how to spell "Nevada" or "California", nor even design a system to detect that issue. If effective controls can't be found by GoDaddy, perhaps it's better to not issue OV and EV certificates? That would certainly mitigate the risk with zero ambiguity.

Comment 6

[Joanna]

(In reply to Ryan Sleevi from comment #5)
Ryan:
As previously stated, we do not require the Root programs guidance. It was an ask in the spirit of cooperation/partnership.

GoDaddy has an internal source it will use to introduce the appropriate automation. The new control will be deployed in production no later than end of September, sooner if possible. We will update this thread when the new controls have been deployed and the issue mitigated.

GoDaddy also agrees with using a risk management approach. Our current process contains a Compliance team that evaluates data sources and suitability that recommends processes to reduce human error. The Validation Specialist team reviews each certificate application against the sources used and data provided by the Compliance team to determine if something is allowed or not, and yes sometimes management is involved in the decision making. This information is then reviewed by a higher level Validation Specialist prior to issuance. Even with all these safeguards in place, approximately .03% of EV’s were found issued with some errors in State/Country fields. This is why we believe automation is the key to solving it, and it’s why we’ll have a solution implemented shortly.

Comment 7

[Ryan Sleevi]

(In reply to Joanna from comment #6)

GoDaddy has an internal source it will use to introduce the appropriate automation. The new control will be deployed in production no later than end of September, sooner if possible. We will update this thread when the new controls have been deployed and the issue mitigated.

GoDaddy also agrees with using a risk management approach. Our current process contains a Compliance team that evaluates data sources and suitability that recommends processes to reduce human error. The Validation Specialist team reviews each certificate application against the sources used and data provided by the Compliance team to determine if something is allowed or not, and yes sometimes management is involved in the decision making. This information is then reviewed by a higher level Validation Specialist prior to issuance. Even with all these safeguards in place, approximately .03% of EV’s were found issued with some errors in State/Country fields. This is why we believe automation is the key to solving it, and it’s why we’ll have a solution implemented shortly.

I don't think this level of detail really goes to provide public understanding about the processes involved, or how they're being fixed.

I encourage you to compare this response with that in Bug 1576013, specifically Comment 11, as well as the follow-up on https://groups.google.com/d/msg/mozilla.dev.security.policy/DxtWaeIQKfM/3bJVAQI0AQAJ

This is an example of a response that helps provide assurances that things are being fixed. But if you think that's too recent (despite predating GoDaddy's response here), then look at the incident response in Bug 1551374, which is held as an example in Responding to an Incident. If you're not sure how to capture it, https://groups.google.com/d/msg/mozilla.dev.security.policy/oP8XuNXrANw/oIYt70IiAAAJ provides an example of providing good incident responses.

If it sounds like there's frustration, it's because other CAs have substantially raised the bar for expectations and transparency, as well as dealing specifically with this class of errors and issues and not only taking a risk based approach, but explaining and communicating it, and the many mitigations in place.

Comment 8

[Daniela Hood]

GoDaddy is pleased to announce that on October 1st, at 1330 PST, we deployed a solution that fully mitigates this issue of misspellings and improper correlation between states / provinces and their corresponding jurisdictions.

Technical Details:

We created an array of acceptable states / provinces and their corresponding jurisdictions for the locations we service; we then made them available via a drop-down feature inside our customer and admin validation applications. We also removed any free form text fields from the corresponding applications.

Additionally, we deployed controls that do not allow the issuance of certificates that are not within our predefined list of states / provinces and their corresponding jurisdictions. This provides additional assurances that if a state / province doesn't conform to the existing business logic it can't be issued (e.g., a certificate with Ontario as the state/province will always be correlated to Canada, if that rule is not met the system will not issue a certificate).

We believe this fully mitigates the potential of this ever happening again.

This new validation automation is in affect globally for all certificates issued by GoDaddy.

Comment 9

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.