1667842 - SecureTrust: Inaccurate value in stateOrProvinceName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrea Holland]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0

Comment 1

[Andrea Holland]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

   A problem report was submitted to our problem reporting mechanism on September 26, 2020 at 5:28 AM CDT.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

   09.26.2020, 05:28 AM CDT Security researcher submitted problem report to our problem reporting mechanism
   09.26.2020, 09:40 AM CDT Email incorrectly categorized
   09.28.2020, 09.39 AM CDT Certificate Problem Report is circulated to compliance team
   09.28.2020, 10:11 AM CDT Compliance team confirms certificates have been mis-issued and begins revocation procedure
   09.28.2020, 10:50 AM CDT Revocation of problem certificates
   09.28.2020, 11:05 AM CDT Problem reporter notified of revocation

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

   These certificates were issued prior to the validation and protocol changes, where individual specialists have a more focused approach, made as a result of the “Some-State” bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1551374). There are no other instances of certificates with Netherlands in the State/Province field for the Netherlands. We are still investigating whether there are other instances of Country name in the State/Province field.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

   Two certificates were issued as a part of a chain of reissuances for the same product instance.
   The initial product issuance was on 02.15.2018, 03:51 CST.
   The last product issuance was on 12.04.2018, 01:58 CST.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

   Please see:
   https://crt.sh/?id=350604899
   https://crt.sh/?id=1007092879

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

   An examination of our issued certificates was done for the Some-State bug in 2019 (https://bugzilla.mozilla.org/show_bug.cgi?id=1551374). While other certificate instances were discovered to have location issues such as the State/Province and Locality were swapped or misspelled, the manual evaluation missed checking for instances of valid Country names in the State/Province field. See item 7 for the process improvements being made.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

   After the Some-State bug was remediated, we continue to improve upon our location validation implementation automation to include checking against ISO-3166-2 State/Provinces in conjunction with GeoNames to allow for the agreement of two sources of data. This continues to be an on-going effort to reduce the advisory controls and increase the technical controls based on the Country selected. Implementation of this process requires thorough evaluation of each country where these technical controls plan on being enforced. Netherlands is currently being evaluated to be added for State/Province technical controls with the goal of implantation by year-end 2020. We will also examine our existing certificates for additional discrepancies or divergence. Furthermore, we will implement a process to run existing certificates through new technical controls prior to deployment in order to more timely detect historic mis-issuances. This step will be added to our existing template for adding new State/Province technical controls.

Comment 2

[Ryan Sleevi]

Andrea: Thanks. I think SecureTrust's example in https://bugzilla.mozilla.org/show_bug.cgi?id=1551374#c8 from Bug 1551374 is quite admirable. However, it's not really clear to me the "why" in why this was missed as part of that investigation.

You note in the response to Question 6 that it was missed, but I don't really see a clear explanation of why (which could have caught this), which would also inform what's being done in the future.

The answer to Question 7 doesn't really have a clear timeline to deliver these changes, and that's one of the expectations for closing this out. You mention Netherlands being targeted for EOY 2020, but it's unclear if the remaining steps are also being proposed to be complete there. If so, it's unclear why it would take until EOY 2020 to "examine our existing certificates for additional discrepancies or divergence", and that also makes it hard to believe that the answer to Question 5 is complete.

Can you help provide clarity here?

Comment 3

[Andrea Holland]

Please expect a formal response early next week.
Expected Timeline:

• Netherlands State/Province implementation target date EOY 2020
• Supplemental review - target date mid-November 2020
• Template update for adding new State/Province technical controls - Completed

Comment 4

[Andrea Holland]

To clarify there are no other instances of certificates with Netherlands in the State/Province field for the Country Netherlands, but we are conducting a supplemental review to confirm there are no other certificates issued with incorrect State/Province or Locality information. There is a lack of documentation on the steps taken during the last supplemental review to determine why these certificates were missed in the Some-State remediation. In this supplemental review we will fully document our internal investigation. For our internal investigation we reran our existing certificates through our location validation process and collected the ones where the indicator flagged an invalid entry. Our team is examining that data in order to discuss and document any discrepancies.

SecureTrust’s location validation process utilizes both ISO 3166 and GeoNames to ensure the accuracy of location information. The list of possible country codes comes from ISO 3166-1; it is not possible to issue a certificate with a country code that is not in this list. For most countries, we source states/provinces from both ISO 3166-2 and GeoNames. If the validation specialist has entered a state/province that is not in one of these databases, they will be shown an indicator that the state/province is invalid. These indicators are advisory due to factors like the datasets not being a complete list of all locations in the world or that there are often multiple ways to express locations. For a subset of countries, states/provinces are taken only from ISO 3166-2. If the validation specialist has entered a state/province that is not in the list for a country with that technical control, they will be shown an invalid indicator, and they will be unable to issue the certificate with that state/province value. We implement this technical control preventing invalid state/province for only a subset of countries because many countries use addressing conventions that do not match the names in ISO 3166-2. We are continuing to improve upon this process and research more countries where these technical controls can be implemented. Localities are taken from GeoNames only, as ISO 3166-2 does not enumerate them. These indicators are advisory due to factors like the datasets not being a complete list of all locations in the world or that there are often multiple ways to express locations.

Comment 5

[Andrea Holland]

An update:

• Netherlands State/Province implementation - still on target for EOY 2020
• Supplemental review – the review continues, but the target date has shifted to mid-December 2020 due to personnel departures and assignment adjustments caused by COVID-19

Comment 6

[Andrea Holland]

We have completed our supplemental review with respect to the certificate location data and have revoked the certificates that were not in compliance with the baseline requirements. We found these certs to have errors:

Certificate Information:
https://crt.sh/?id=3825461272
Problem discovered: Repeat of Country for State/Province: “Oman” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1176280589
Problem discovered: Repeat of Country for State/Province: “Serbia” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=2536472097
Problem discovered: Incorrect Locality: Street address entered in the locality field
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=2258107265
Problem discovered: Misspelling: Locality is set to “Lousia” should be “Louisa”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1613151268
Problem discovered: Incorrect State for the Locality: “South Australia” should be “Western Australia”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=3349456552
Problem discovered: Misspelling: State/Province “Whales” should be “Wales”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1237692715
Problem discovered: Incorrect State/Province: “Alicante” should be “Balearic Islands”
Resulting action: Revoked
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1490058537
Problem discovered: Repeat of Country for State/Province: “France” should be “Ile-De-France”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=2273487377
Problem discovered: Misspelling in Locality: “Teginmouth” should be “Teignmouth”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1511633700
Problem discovered: Repeat of Country for State/Province: “Hong Kong” should be “New Territories”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=3825461323
Problem discovered: Repeat of Country for State/Province: “Hong Kong” should be “Kowloon City”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1422336500
Problem discovered: Incorrect State/Province: “Dublin 2” should be “Leinster”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1422334605
Problem discovered: Incorrect State/Province: “Dublin 2” should be “Leinster”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1425066962
Problem discovered: Incorrect State/Province: “Dublin 2” should be “Leinster”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1231840244
Problem discovered: Incorrect Locality: “Dublin Airport” should be “Dublin” with blank State
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1232086645
Problem discovered: Incorrect Locality: “Dublin Airport” should be “Dublin” with blank State
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1380734671
Problem discovered: Misspelling of State/Province and Locality: “Chsinau” should be “Chisinau”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=2735094942
Problem discovered: Incorrect State: “Mexico City” should be “Estado de Mexico”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=3825461219
Problem discovered: Incorrect State: “Kuala Lumpur” should be “Selangor”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=323359463
Problem discovered: Incorrect State/Province: “Central Province” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1663746987
Problem discovered: Misspelling of Locality: “Ladprao” should be “Lat Phrao”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=3818028220
Problem discovered: Incorrect State/Province and Locality: “Midridge” should be “Midrand, Gauteng”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=3825461309
Problem discovered: Misspelled State/Province: “Nood-Holland” should be “Noord-Holland”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=3825461258
Problem discovered: Misspelled State/Province: “Nood-Holland” should be “Noord-Holland”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1384278513
Problem discovered: Repeat of Locality for State/Province: “Nassau” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1135923978
Problem discovered: Repeat of Locality for State/Province: “Kongens. Lyngby” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

During our supplemental investigation, we ran our certificates against our source of valid locations and manually reviewed the certificates that did not have a match. Our examination revealed certain certificates that should have been revoked during the Some-State remediation due to misspelled or incorrect values. As stated before in Comment #4, there was no detailed documentation to explain why those certificates would have been missed previously. That remediation discussed that the State/Province field should be omitted if the QIIS/QGIS did not include that field, but we did not revisit existing certificates that had utilized that previously expected process. Those have now been revoked. We followed up on that action item in our system, changing the location validation implementation to provide a green icon for the State/Province field if it is empty instead of a red icon. Thus, removing the indication to the analyst that an empty State/Province field is incorrect and must be filled out.

The Netherlands technical controls for State/Province have been applied and we continue to add stricter limits on a Country’s allowed values for State/Province, moving from an advisory control to a technical control on a country by country basis. This technical control will prevent instances of misspelled or incorrect State/Provinces. We continue to investigate solutions to alert fatigue, especially with the locality field. We have visual advisory controls to notify the analyst if a locality is in Geonames, but Geonames does not encompass all valid localities. Analysts who continue to see red exclamation marks for valid localities may tend to become desensitized to those notifications. We continue to explore other methods to provide validation for locality to break the cycle of valid localities being visually marked as invalid.

Comment 7

[Andrea Holland]

During a support team review of a certificate renewal request, another older incorrect organization location error was discovered. This prompted an investigation into the location review process which uncovered an error that the organization locations reviewed did not pull from EV certificates. As a result, we have reviewed all of the active EV certificates and found only one organization validation which had a total of four certificates issued for it with a repeat of the Country for the State/Province information. This also triggered a review of the active certificate’s jurisdiction location information. The jurisdiction review found older instances of abbreviated names in the jurisdiction state field and an incorrect entry. These certificates have been revoked in compliance with the baseline requirements.

Certificate Information:
https://crt.sh/?id=1351987773
https://crt.sh/?id=1573789097
https://crt.sh/?id=1952987985
https://crt.sh/?id=2009674336
Problem discovered: Repeat of Country for State/Province: “Slovenia” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1160208530
https://crt.sh/?id=1821998959
https://crt.sh/?id=1822432891
https://crt.sh/?id=1560824908
https://crt.sh/?id=4065929495
Problem discovered: Abbreviated State for Jurisdiction State/Province: “NY” should be New York
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1706736734
Problem discovered: Abbreviated State for Jurisdiction State/Province: “VA” should be Virginia
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1757371985
Problem discovered: Abbreviated State for Jurisdiction State/Province: “ACT” should be Australian Capital Territory
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=1865427444
Problem discovered: Incorrect Jurisdiction State/Province: “England and Wales” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Our current implementations would have prevented issuance of these older certificates. Our system has executed the changes that allow analysts to use blank values for State/Province without having a negative indicator. Also, with the changes applied for ballot SC30, strict enforcement has also been added for the jurisdiction information based on the published list of the Incorporating Agency and Registry Agency.

Comment 8

[Andrea Holland]

If there are no further comments, we request this bug be closed.

Comment 9

[Ben Wilson]

I'll schedule this to be closed next Wednesday, 7-Apr-2021.