Closed Bug 1708834 Opened 4 years ago Closed 3 years ago

GlobalSign: Invalid stateOrProvinceName and locality pair

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: fozzie, Assigned: arvid.vermote)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(4 files)

Incident report
7.13 KB, text/plain
Details
incident report
7.13 KB, text/plain
Details
20210505_listofcerts.xlsx
9.65 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
1708834_20210531.xlsx
11.71 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
Assignee: bwilson → arvid.vermote
Status: NEW → ASSIGNED
Whiteboard: [ca-compliance]

Hello George, thank you for this bugzilla ticket and for reaching out to us via report abuse. "Amsterdam" is an allowlisted value for GlobalSign. We have allowlisted it based on Qualified Independent Information Sources, which showed Amsterdam as the appropriate State value for a certificate request for a different customer. We have urgently looked into this approval, and it was approved because the locality included that request is a municipality of Amsterdam. For these municipalities, Amsterdam as a city is then the next step up in the administrative subdivision of the Netherlands. Just to give some additional explanation for people who may be less familiar with this, the system of municipalities is one found in many countries around the world, and historically, these smaller municipalities were themselves independent towns or cities, with their own mayor for example. Often these have been merged (or were incorporated) into larger administrative subdivisions for practical or financial reasons. These former cities or towns still have a defined territory, and in some cases, still have some administrative authority themselves, independent from the subdivision they are now part of, and are often still listed in the Incorporating sources as the physical location of companies. As they do fall under a better-known administrative subdivision (in this case, a city), and as this subdivision is "one step up" from their location, and generally better known than even the Province (which is another "one step up" in terms of administrative subdivisions, compared to the city) I imagine that's why sources accept places like Amsterdam as a State or Province value.

Hi Eva, thanks for the quick and detailed response.

It looks to me that you have simply switched the ST and L values of these certificates (with Noord-Holland being the province and Amsterdam being the locality), indeed GlobalSign has themselves issued hundreds of certificates with these fields being the other way around - https://censys.io/certificates?q=tags.raw%3A+%22trusted%22+AND+validation.nss.had_trusted_path%3A+true+AND+tags.raw%3A+%22leaf%22+AND+parsed.subject.province%3A+%22Noord-Holland%22+AND+parsed.subject.locality%3A+%22Amsterdam%22.

So I'd like you to clarify, are you saying Noord-Holland is the locality and Amsterdam is the province? If so can you provide what source you used to verify this? To me it looks like Noord-Holland is the province and a quick Google suggests that Amsterdam is a municipality within this province so should be placed in the locality field. Thanks.

Flags: needinfo?(eva.vansteenberge)

Hello George, initially we only looked at the State value due to the nature of the ticket. We acknowledge the inaccurate value in the L field and have suspended issuance from that approved profile immediately pending update of the values, which was completed today at 10:04 UTC. We will be working with the customer to replace these certificates at the earliest convenience, and these certificates will be revoked within the timeframe set by the Baseline Requirements. We will provide a detailed report on this issue on Wednesday, 5th of May at the latest.

Flags: needinfo?(eva.vansteenberge)

Thanks Eva, perhaps the summary should be "Invalid stateOrProvinceName and locality pair" or something similar so I'll update it now, if anyone else has anything better in mind then feel free to update.

Summary: GlobalSign: Invalid stateOrProvinceName field → GlobalSign: Invalid stateOrProvinceName and locality pair
Attached file Incident report —
Attached file incident report —

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware following the creation of the bug ticket, which was filed at 21:55 UTC on 30/04/2021. We also received a certificate problem report via our report-abuse@globalsign.com email address at 21:50 UTC on 30/04/2021. Both these reports created the corresponding tickets in our internal SOC, which escalated to the Compliance team, who started the investigation for those tickets at 01/05/2021 - 06:53 UTC . The initial focus of the report was the State field. The fact that the issue actually lay within the locality field became apparent upon closer inspection following Comment 2 at 09:44 UTC.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY - Time in UTC Description
11/04/2019 Profile containing L=Noord Holland and S=Amsterdam was activated.
22/10/2019 Due diligence by a person who is not responsible for the collection of information becomes mandatory for all profile based vetting. reviously it was only required for EV profile based vetting.
29/02/2020 Allowlist State values Netherlands completed. S=Amsterdam was added, following an earlier customer request (non-SSL) containing S=Amsterdam in QIIS (Locality for that request was Amsterdam Zuidoost)
30/04/2021 21:50 and 21:55 Bugzilla ticket and certificate problem reports created. Corresponding GlobalSign internal SOC tickets created. SOC operators escalate to the compliance team.
01/05/2021 - 06:53 Start of investigation by the compliance team. Focus on the State value.
01/05/2021 - 09:44 Comment 2 at 09:44
01/05/2021 - 09:46 Deactivated profile, pending update to values. Combined vetting management and compliance review for all requested certificates.
01/05/2021 - 10:04 Reactivated profile containing updated information.
01/05/2021 - 10:14 Started the investigation for other non-reported certificates with exact same problem.
01/05/2021 - 10:31 All certificates identified (see #4), replacement process starts.
01/05/2021 - 14:03 Started the review of currently active profiles and full historic issuance review.
05/05/2021 - 20:00 All affected certificates listed in this incident report revoked.
07/05/2021 - 20:00 Targeted completion date of the review of currently active profiles
31/05/2021 - 08:00 Targeted completion date of full historic issuance review.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We halted issuance on the profile within minutes of Comment 2, at 09:46 UTC on 01/05/2021, pending updates to the profile. Our on-call vetting staff were contacted, and the updated profile was activated at 10:04 on the same day.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

See attached

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See attached

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The Locality and State information were entered in the wrong fields, and this was not picked-up by a vetting operator. As we mentioned in Comment 1, the certificate requests were not flagged because the State value had been allowlisted. This allowlisting happened following the inclusion of the value in a different customer request. For that specific request, the Incorporating Agency listed the Locality as Amsterdam Zuidoost ("Amsterdam-Southeast"), which is a borough of Amsterdam. Even though these boroughs are part of the City of Amsterdam, they have a clearly defined territory and a directly elected district committee. The City of Amsterdam is one step up in terms of administrative levels.

Since the original activation of the profile, GlobalSign has introduced additional mandatory due diligence checks for all profile based vetting, to reduce the the risk of human error on any validation aspect (not just content of the certificate).

We are in the process of reviewing all currently-active profiles to make sure the Locality information is accurate. In parallel, we are performing a full review of historically issued certificates for inaccuracies in the Locality information.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We are looking for automated ways in which to address this issue. We are analysing the implementation requirements for multiple solutions and will share the implementation date as soon as the analysis is complete. We are looking at the possibility of introducing an allowlist similar to the allowlist we currently have for the State field, or alternatively, seeing how we can leverage data from a third party source like Geonames to help safeguard the accuracy of the values in the Locality field. For this, we are also keeping in mind the comments made about non-obvious BR violations on https://bugzilla.mozilla.org/show_bug.cgi?id=1707073.

While this analysis is being performed, and pending a root cause fix, any new certificate requests are being reviewed by vetting management and compliance, to ensure that they only contain accurate values.

End date DD/MM/YYYY Status Action
01/05/2021 Completed Combined vetting management and compliance review for L values for all requested certificates.
05/05/2021 Ongoing All affected certificates listed in this incident report revoked.
07/05/2021 Ongoing Target end-date for the review of currently-active profiles.
31/05/2021 Ongoing Target end-date of full historic issuance review.
TBD Ongoing Analysis of implementation requirements for multiple solutions.

We have finished the review of currently-active profiles, and are continuing the full historic issuance review. We continue to have a combined vetting management and compliance review for newly requested certificates to detect any issues, pending implementation of a full solution. The further analysis of the implementation requirements for multiple solutions is still ongoing.

We have nothing new to report this week, all previous stated actions are still ongoing.

We have nothing new to report this week, all previous stated actions are still ongoing.

Thanks for the continued weekly updates Eva. I look forward to reading the update on 2021-05-31, as I'm assuming that's still on track based on Comment #9 / Comment #10

Attached file 1708834_20210531.xlsx —

Please find attached the additional certificates which were determined to be mis-issued following the finalization of our historic issuance review on the 30/05/2021. During this review, we compared the historically issued locality and state values to a number of publicly available databases which hold locality and/or locality and state information. This resulted in a number of flags, which were then subject to an additional review based on the evidence recorded as part of the validation process. This additional step was required because of a large number of false positives.

We are working with the customers to replace the affected certificates, latest by Friday 04/06/2021 - 17:00 UTC. Initial root cause analysis indicates there are different and specific root causes for the cases we've identified. We will complete the root cause analysis for all cases by Friday 04/06/2021 and will define measures to further prevent the issue.

While this is ongoing, we continue to have a pre-issuance review by compliance and vetting management on the information contained in all newly requested certificates.

In this comment, we have two updates: Firstly, we want to confirm all the above identified certificates have been revoked, with the latest revocation happening by 04/06/2021 - 17:00 UTC. Secondly, we have concluded our root-cause analysis on Friday, June 4, and we are looking at ways in which these root causes can be addressed. We will be updating this ticket again on Wednesday 9th of June to share the root cause findings.

When looking at the evidences gathered in support of the issuance of the certificates, the root causes can be split up in two broad categories:

Category 1: Mistakes by the validation specialist (i.e. the data source contained the correct information, but the certificate included different information). There were two types of these errors:

  • Spelling errors in the locality fields: the data is properly spelled in the datasource but misspelled in the certificate.
  • Errors where the state and locality fields were correctly identified in the data source but the information in the order wasn’t updated. The majority of this cases could be attributed to a single validation specialist who already left the organization.

Category 2: Wrong interpretations of the data source. In these cases, the data source contained the values included in the certificates, but not in the corresponding fields. This was partially due to ambiguity of the sources, which did not label the different elements of the address, or ambiguity about the meaning of "locality" for non-native language speakers.

We are now reviewing the ways in which these root causes can be addressed. We have already revised the definition of locality and communicated this revision to our validation specialists. Additionally, we are also highlighting the ambiguous situations for the sources identified during this review.

Please note, that while we are analyzing the most appropriate solution for this problem, we continue to have a pre-issuance review by compliance and vetting management to help ensure no new certificates are mis-issued. We aim to have a further update on our actions to address the issues and their root causes by the end of this month.

Eva: It's been 8 days since Comment #14, and Comment #14 didn't give a clear ETA for that analysis of solutions. Could you provide an update here - or an update on when you expect an update? These do sound like tricky issues, but we want to make sure that there's good communication in place here.

Flags: needinfo?(eva.vansteenberge)

Hello Ryan, we have no new updates this week. We will update again next week, and aim to have a more clear picture of an ETA then.

Flags: needinfo?(eva.vansteenberge)

We are further conducting an impact assessment of the different solutions that could address the mentioned root causes. As part of this impact analysis, we are soliciting feedback from different international stakeholders by the means of a survey. The deadline for the feedback is the 2nd of July. We continue to review new requests prior to issuance in order to help ensure no wrong values are included in new certificates.

After receiving initial responses from our survey we identified the need for a second round of feedback was necessary, we have initiated this with a deadline of the 9th of July. We continue to review new requests prior to issuance in order to help ensure no wrong values are included in new certificates.

Noting that there were 10 days between Comment #17 and Comment #18 , despite Comment #15, https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed, and https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/BOwcbWbZTg0/m/27F-UhedBgAJ , I want to make sure we're not further delaying updates from Comment #18, especially since follow-up was expected on 2021-07-09.

I also want to highlight that Comment #17 and Comment #18 seem devoid of meaningful and useful data. For example, disclosing what the survey was in Comment #17 would have been useful, and similarly, the followup in Comment #18. The goal here in these updates is to provide transparency as to what is going on, so that we can all collectively work to arrive at useful solutions.

Can you please share the survey (questions) from Comment #17 and Comment #18?

Flags: needinfo?(eva.vansteenberge)

The objective of the survey was to gather insights on the use cases and demand for the locality field, as observed by all relevant stakeholders within GlobalSign, considering the complexity of the field in terms of meaning and validation across jurisdictions. We analyzed the second round of feedback and the outcome is that we keep supporting the field and we will address the root cause with the following preventive measures:

Category 1: Mistakes by the validation specialist (e.g. spelling, vetting information not updated).

Category 1 issues affected 16% of the cases.

We have raised awareness of this issue since the start of this incident. The attention to detail is an important part of our training, monitoring, evaluation and feedback to our vetting agents. The agents of the affected cases have been approached and we will keep monitoring the accuracy of our validation specialists.

Category 2: Wrong interpretations of the data source (e.g. meaning of locality, labelling of the fields in the source)

Category 2 issues affected the majority of the cases (84%), with 52% due to wrong interpretation of data sources and 32% related to the meaning of "locality" for non-native language speakers.

For the sources identified as part of this incident, we have already highlighted the ambiguous situations to our validation specialists. We are now reviewing all sources to verify if other sources are affected and updating our internal guidelines to highlight these specific sources to our validation specialists. We are providing the specialists with additional resources to validate the information of the data source when the source is in some way not clear or ambiguous. We plan to finish this project by the end of Q3 of this year.

The meaning of "locality" for non-native language speakers has already been addressed with further clarification and training, and we will be putting a validation rule in place to help prevent issuance to names that relate to locations which are not localities.

Similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1714968#c10, this rule checks if certain conditions are met before allowing validation specialists to close the case. We will be checking for certain indicators that denote a wrong interpretation.

For example, we will block known bad values, as well as mentions of "street" and equivalences or abbreviations (also in different languages), check for the inclusion of numbers (except for a limited number of situations where the number is justified, see https://bugzilla.mozilla.org/show_bug.cgi?id=1711208#c1 as an example). If the criteria are met, an error message will appear and closing the case will be impossible until the underlying error is fixed. These criteria will be continuously refined.

This validation rule will be in place by 30/07/2021.

Flags: needinfo?(eva.vansteenberge)

We are still on track to implement the validation rule by 30/07/2021. We continue to review new requests prior to issuance in order to help ensure no wrong values are included in new certificates.

We are still on track to implement the validation rule by 30/07/2021. We continue to review new requests prior to issuance in order to help ensure no wrong values are included in new certificates.

We have implemented the validation rule in production to help prevent issuance to names that relate to locations which are not localities. We are progressing with the review of all our validation sources and updating our internal guidelines on specific sources where information may be ambiguous, and to provide the validation specialists with additional resources to validate the information of the data source. We aim to complete this review by the end of Q3 of this year. We continue to review new requests prior to issuance in order to help ensure no wrong values are included in new certificates.

We continue to progress with the review of all our validation sources and updating our internal guidelines on specific sources where needed. We are on track to complete this review by the end of Q3 of this year. We continue to review new requests prior to issuance in order to help ensure no wrong values are included in new certificates.

Ben: Not sure if you wanted to NextUpdate to 2021-09-30, based on Comment #23.

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] → [ca-compliance] Next update 2021-10-01

Changes to the sources database have been implemented as described in Comment 20. This concludes the identified remedial activities, unless there are any further questions we believe this issue can be closed.

I'll close this on next Wed. October 6, 2021.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next update 2021-10-01 → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: