Open Bug 1720723 Opened 5 months ago Updated 21 days ago

SecureTrust: Invalid localityName

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: aholland, Assigned: aholland)

References

Details

(Whiteboard: [ca-compliance] Next update 2022-01-05)

On July 14, SecureTrust Digital Certificate Support personnel were reviewing new Organization entries and flagged a certificate that was issued with an invalid locality name:

https://crt.sh/?id=4866682152

A preliminary investigation confirmed the certificate mis-issuance and the certificate has been revoked within 24 hrs.
A full report will be posted in the coming days.

Assignee: bwilson → aholland
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]
  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

    On July 14, SecureTrust Digital Certificate Support personnel were reviewing new Organization entries and flagged a certificate that was issued with an invalid locality name.

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

    07.14.2021, 11:52 AM CDT Certificate was issued.
    07.14.2021, 12:10 PM CDT Certificate was flagged during new Organizations review and locality information was corrected.
    07.14.2021, 12:47 PM CDT Certificate was revoked.

  3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

    The organization with the incorrect locality value was updated within an hour of confirming the problem. No additional certificates with this incorrect locality field where issued.

  4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

    A single certificate was issued with the street name in the locality instead of the correct locality.

  5. The complete certificate data for the problematic certificates.

    https://crt.sh/?id=4866682152

  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

    SecureTrust’s location verification system uses a text input field for the locality entry. The text field has an advisory control that flags entries which are not in GeoNames according to the other location fields, (State/Province and Country). These indicators are advisory due to factors like the datasets not being a complete list of all localities in the world, only listing larger population localities, or that there are often multiple ways to express locations. In this instance the locality was flagged as a potential error, but the analyst believed the entry in the organization validation document for ‘Peoples Place’ was equivalent to locality. The second analyst also missed the warning indicator for the locality. We believe this misstep by two analysts was due to alert notification fatigue. Our implementation can flag localities as invalid which are in fact valid. This causes false positives alerts which the analysts continue to run into throughout the validation process and leads to alert notification fatigue from the locality advisory indicators.

  7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

    The volume of false positives indicating a potential error in locality information is the key factor here. We are addressing the larger issue of alert fatigue in the long term, but are adding a more formal evaluation in the short term.

    • Immediate:
      • Analysts and Reviewers will be reminded of the importance of the warning notification and that each one requires a second look to determine that it is actually a false positive alert for a valid locality.
    • Short term:
      • We will be adding an additional formal check by a supervisor when examining an organization with an invalid indicator for the locality. This should temporarily combat the alert notification fatigue as the analyst will be prompted that a supervisor review is required for a flagged locality. Implementation in mid-August.
      • We will also be investigating certificates that were flagged as invalid localities by GeoNames and issued after our investigation on Bug #1667842, to see if other instances of similar nature have occurred. Completion late September.
    • Long term:
      • We are planning on implementing an allowlist procedure to address the false positives that repetitively appear to analysts. We will be preloading that list with valid locality, State/Province, and Country combinations that our current GeoNames implementation would have flagged the locality as incorrect. Once preloaded, the procedure going forward will require a supervisor to proactively add a valid locality to the allowlist, so that the next time the analyst comes across the same location information the locality will not be incorrectly flagged as invalid. By focusing on the more frequently seen localities, we will begin to decrease the alert fatigue by fixing the false positives alerts. Implementation mid-November.

Andrea: Could you explain more about the lack of expected updates here?

Flags: needinfo?(aholland)
See Also: → 1667842
    * We are planning on implementing an allowlist procedure to address the false positives that repetitively appear to analysts. We will be preloading that list with valid locality, State/Province, and Country combinations that our current GeoNames implementation would have flagged the locality as incorrect.  Once preloaded, the procedure going forward will require a supervisor to proactively add a valid locality to the allowlist, so that the next time the analyst comes across the same location information the locality will not be incorrectly flagged as invalid.  By focusing on the more frequently seen localities, we will begin to decrease the alert fatigue by fixing the false positives alerts. Implementation mid-November.

Have you reviewed past CA incidents to find discussions about the risks of such an approach?

As it stands, it seems like there's a few risks here that you don't seem to discuss, and so it's unclear if you're aware of them, and if so, how you propose to mitigate them. For example:

  • The risk of a supervisor inappropriately saying something is allowed, and then that indefinitely being allowed (preventing further examination)
  • The risk of a value a supervisor determines is allowed ends up being disallowed at a later point (e.g. a city or municipality is formally absorbed into a neighbor, a city is renamed, etc)

There's been a lot of discussion on the validation of State/Province and Country, and it's important to understand what steps are being taken here.

Ryan: The lack of updates was my mistake. I had set the schedule for updates on the remediation and planned to address any comments that came across, but failed to provide the weekly update. I have set a weekly calendar reminder to provide updates for this bug, so that this does not occur again.

For the long-term implementation, we did review the previous bugs and remediations. We are still discussing the procedure for this solution; thank you for providing additional items to consider. This will be a solution for the Locality field data tied to State/Province and Country entries and only for the frequently reoccurring locality data, which causes the false positive notifications. We will make sure we include the process details we will be using to mitigate the potential risks of an allow list solution along with our scheduled review cycle for those localities on the allow list.

Flags: needinfo?(aholland)

I have set a weekly calendar reminder to provide updates for this bug, so that this does not occur again.

I appreciate this, although I flag this to make sure it's more systemically thought of as part of the incident management process, since these sorts of mistakes are easy to make, but can easily be quite problematic.

Ryan: I appreciate the comment, a step was added to our incident management process to ensure weekly updates are performed.

Remediation Status:

  • Immediate: Done
  • Short Term Item 1: Done
  • Short Term Item 2: In process
  • Long Term: On schedule - in procedure discussions

No new updates.

Remediation Status:

  • Immediate: Done
  • Short Term Item 1: Done
  • Short Term Item 2: In process
  • Long Term: On schedule - in procedure discussions

No new updates are planned until Short Term Item 2 is completed. I am requesting the next update date to be September 22nd.

Ben: Consideration of a Next-Update for you, based on Comment #8, which is reference to Comment #1, specifically:

  • We will also be investigating certificates that were flagged as invalid localities by GeoNames and issued after our investigation on Bug #1667842, to see if other instances of similar nature have occurred. Completion late September.
Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] → [ca-compliance] Next update 2021-09-22

We have completed the scan of the certificates and will provide a report in the next few days.

We have completed our supplemental review with respect to the certificate location data and have revoked the certificates that were not in compliance with the baseline requirements. We found these certificates to have errors:

Certificate Information:
https://crt.sh/?id=5303280454
Problem discovered: Misspelling in Locality: “Hasting” should be “Hastings”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280460
Problem discovered: Incorrect Locality: “Lippo Center” should be “Admiralty”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280464
Problem discovered: Incorrect Locality: “Lippo Center” should be “Admiralty”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280443
Problem discovered: Incorrect placement for State/Province: “Admiralty” should be in Locality field
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280462
Problem discovered: Misspelling in Locality: “Wilmingtion” should be “Wilmington”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280461
Problem discovered: Misspelling in Locality: “Wilmingtion” should be “Wilmington”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280445
Problem discovered: Misspelling in Locality: “Wilmingtion” should be “Wilmington”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280470
Problem discovered: Misspelling in Locality: “Wilmingtion” should be “Wilmington”
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Certificate Information:
https://crt.sh/?id=5303280450
Problem discovered: Repeat of State/Province for Locality “Guateng” should be blank
Resulting action: Revoke
Current validation protocols would have discovered the error (Yes/No): Yes

Remediation Status:

  • Immediate: Done
  • Short Term Item 1: Done
  • Short Term Item 2: Done
  • Long Term: On schedule - in procedure discussions

Remediation Status:

  • Long Term: On schedule
    • in procedure discussions
    • potential allowList population in review

Remediation Status:

  • Long Term: On schedule
    • in procedure discussions
    • potential allowList population in review

Remediation Status:

  • Long Term: Delayed
    • in procedure discussions
    • potential allowList population in review

Remediation Status:

  • Long Term: Delayed
    • in procedure discussions
    • potential allowList population in review

Remediation Status:

  • Long Term: New implementation date Q1 2022
    • in procedure discussions
    • potential allowList population in review

Ben: With the change in implementation date I am requesting the next update date to be January 5th.

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] Next update 2021-09-22 → [ca-compliance] Next update 2022-01-05
You need to log in before you can comment on or make changes to this bug.