1714968 - GlobalSign: Incorrect RegNumber-Org Type combination

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Eva Van Steenberge]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36

Comment 1

[Eva Van Steenberge]

GlobalSign was notified through a problem report on June 3, 2021 20:51 UTC that 4 certificates were discovered with an incorrect registration number and organization type combination (Subject:businessCategory = Private Organization and Subject:serialNumber denoting the subject is a Government Entity).

https://crt.sh/?id=1324045689 (expired before notification)
https://crt.sh/?id=2452329039
https://crt.sh/?id=3583968463
https://crt.sh/?id=3011177364

Following this investigation, we reviewed our issued certificates and found five other certificates at on June 4, 2021 12:25 UTC:

https://crt.sh/?serial=0186b850cd67ac2bb1efa44b
https://crt.sh/?serial=11cb1cd7e350bb30bdb7655d
https://crt.sh/?serial=1b3418a7a839b0287cba0e21
https://crt.sh/?serial=55fea69c7e161968305ba632
https://crt.sh/?serial=78807a9a423143649951a950

We are in the process of working with the customers to replace these certificates, and they will be revoked by June 8, 2021 20:51 UTC at the latest for the initially reported certificates, and June 9, 2021 12:25 UTC for the certificates found during our investigation.

Investigation has started on the root cause for the issue and in the meantime, we've requested to block new certificates with this registration number and organization type combination to be issued.

We expect to provide a detailed incident report as soon as we have concluded our analysis, but no later than June 11, 2021.

Comment 2

[Eva Van Steenberge]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware following a certificate problem report, which was filed at 03/06/2021 20:51 UTC (All times in this report are in UTC). The report created the corresponding ticket in our internal SOC, which escalated to the Compliance team, who started the investigation on 04/06/2021 08:09. We confirmed the reported certificates (except one which had already expired) required revocation on 04/06/2021 08:44, and started the replacement process. Following the evidence review, we responded to the reporter on 04/06/2021 2:00.

In parallel, at 04/06/2021 09:17, we started reviewing our historically issued certificates for the same issue. On 04/06/2021 2:25 we confirmed that five additional certificates were affected and similarly initiated the process of revocation and replacement for those certificates.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY - Time in UTC	Description
03/06/2021 20:51	Report received
04/06/2021 08:09	Investigation started
04/06/2021 08:44	Evidences reviewed, confirmed certificates in report required revocation. Replacement initiated.
04/06/2021 08:57	Completed review of currently outstanding requests and found no pending requests affected by this issue. Vetting and compliance put on high alert to monitor incoming requests, monitoring reports set up.
04/06/2021 09:17	Started review historically issued certificates
04/06/2021 12:00	Responded to reporter.
04/06/2021 12:25	Completed review of historically issued certificates. 5 additional certificates identified. Started revocation and replacement process.
08/06/2021 18:00	All affected certificates revoked.
10/06/2021 16:03	System changes deployed in production.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Monitoring reports were set up on 04/06/2021 08:57, with no pending requests affected by this issue. At the same time, vetting management was put on high-alert with monitoring reports set-up and shared with Compliance for review. These measures will remain in place until all remedial actions have been completed (see #7). System changes on CRM level were deployed in production on 10/06/2021 16:03.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Reported:

https://crt.sh/?id=1324045689 (expired)
https://crt.sh/?id=2452329039
https://crt.sh/?id=3583968463
https://crt.sh/?id=3011177364

Identified following review:

https://crt.sh/?id=3119464131
https://crt.sh/?id=3220620874
https://crt.sh/?id=3015970925
https://crt.sh/?id=4292104199
https://crt.sh/?id=3258740569

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See #4

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In all cases, the information was provided by the customer during the order. The "Business Category" is a required field, with a drop down which defaults to "Private Organization". The "Registration Number" field is again a required field, and the format is free text (due to the complexity of the values that can be accepted). The guidance given to the customers explains that if the organization is a Government entity and that if there is no Registration Number or readily verifiable date of creation, they should enter language that denotes that the organization is a Government Entity. Customers can proceed without giving much thought about the "Business Category", but consciously provide information in the "Registration Number" field. When both sets of information are presented to the vetting agent, the UI abbreviates the terms to PO, GE, NC and BE. In the UI, the next field presented is the Registration Number, which included the full term "Government Entity" for these certificate requests. At the point of validation of all the orders, the vetting agent correctly identified the organization as being a Government Entity and provided the relevant evidences, but failed to update the information in the Business Category field, which was missed because of the more prominent information from the Registration Number field. The business category and registration number fields were not checked by our system for consistency.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

End date DD/MM/YYYY	Status	Action
11/06/2021	Completed	Awareness training rolled out to all vetting agents.
11/06/2021	Completed	System change on CRM level to ensure Registration Number which denotes Government entity can only be used for the Government Entity Business Category. Additional system change to ensure that evidences provided for a certain Business Category will only be accepted if the corresponding Business Category is reflected in the order.
TBD	Ongoing	UI changes in ordering process and CRM level.
30/06/2021	Ongoing	We are analyzing the possible different combinations of the Business Category and Registration Number fields and are defining a comprehensive set of criteria for custom linting. We will share the implementation date as soon as the analysis is complete.

Comment 3

[Eva Van Steenberge]

We have no new updates this week.

Comment 4

[Ryan Sleevi]

Eva: It would sound like from Comment #3 that no progress has been made on determining when the UI changes can be made.

If progress has been made - e.g. discussions being had, new complications identified, new follow-up items to investigate to determine timing - can you share those? Just trying to understand how/when the timeline for changes will be known.

Comment 5

[Mathew Hodson]

This is similar to previous bug 1599775. In that bug, the customer incorrectly chose Non-Commercial Entity, and in this one the customer incorrectly chose Private Organization. In both cases, the value was provided by the customer and then not properly validated by GlobalSign.

This report says that customers can proceed without giving much thought about the "Business Category", so has GlobalSign considered that they shouldn't accept that field from customers? Instead GlobalSign could determine the correct value from their data sources.

Comment 6

[Eva Van Steenberge]

The UI changes will happen on two levels: ordering process and the CRM level. The CRM level changes will be implemented by the 2nd of July (Friday next week). The UI changes in the ordering process are a bit more complicated to implement, but will be implemented similar to the suggestion made in Comment 5, in our July release (scheduled for the 27th of July).

Comment 7

[Eva Van Steenberge]

Changes have been implemented at CRM level. We have defined additional combinations of the Business Category and Registration Number fields which we are looking to cover as criteria for custom linting. These combinations include different language versions which we have scanned for, but found no previous issuance for. This additional list of criteria will be implemented on a CRM level by the 16th of July at the latest. We will provide a timeline for the custom lint in our update this week.

Comment 8

[Ryan Sleevi]

Eva: I'm concerned that the delayed response to Comment #4, and the delays in Comment #7, suggest that GlobalSign is having issues meeting https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed , which is concerning, especially given the recent discussions

I appreciate the description of "changes have been implemented" in Comment #7, but as with https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report , I'm hoping you can make sure to concretely emphasize what the situation was before and after. I realize Comment #2 suggests some explanation, but I think Comment #6 / Comment #7 may be leaning too heavily on GlobalSign's own understanding of its systems, and so it's not entirely clear which (portion of) changes are made and how the two systems interact. For example, Comment #2 doesn't really provide sufficient context to understand the significance of the differences (ordering process and CRM) in Comment #6.

This might be useful to include screenshots, to help communicate the differences in before/after

Comment 9

[Eva Van Steenberge]

The additional list of criteria to be implemented on CRM level will further detect and prohibit additional combinations of registration number and business category that could lead to mis-issuance, including different language versions, and will prevent similar incorrect combinations as this incident. Together with the CRM changes that we've already implemented (as described and confirmed in Comment 2 and Comment 7, and which we will further clarify as requested beginning of next week), and the changes to the UI during the ordering process, this addresses the root causes of this incident.

To further solidify our preventive measures, we are planning a more holistic approach for custom linting in Q3-Q4 2021, including a custom lint that will address allowed/prohibited values and combinations of serialNumber and Business Category.

Comment 10

[Eva Van Steenberge]

Here’s more detailed information of the changes that we have implemented or that we have scheduled to implement.

• CRM Validation of Registration Number: We have introduced a validation rule which checks the registration number and compares it to the business category. This rule checks if certain conditions are met before allowing validation specialists to close the case. If a registration number denotes a specific type of entity, then it can only be used for that type of entity. For example, if the registration number contains “Government”, then the case can only be closed if the Business Category is “Government Entity”. If this is not the case, the following error message appears and closing the case is impossible until the underlying error is fixed: “Registration number does not match business category”. This change was introduced on 10/06/2021 for the values which were first reported in this incident and as mentioned will be further expanded upon on 16/07/2021 with other language variants.
• CRM Validation of Business Category: As we noted in our initial report, in all instances the vetting agent correctly identified the organization as being a Government Entity and provided the relevant evidences, but failed to update the Business Category on the actual order. Similar to the change described above, we have introduced a validation rule to ensure that evidences provided for a specific Business Category will only be accepted if the corresponding Business Category is reflected in the order itself. If this is not the case, the following error message appears and closing the case is impossible until the underlying error is fixed: “Validated business category does not match business category in request”. This change was introduced on 10/06/2021.
• CRM UI: All information that requires validation is presented to the validation specialists on their vetting cases. The Registration Number was displayed directly underneath the Business Category field and said “Government Entity” in full. We verified with the validation specialist that the Business Category field is abbreviated and have updated the UI so the Business Category is now also displayed in full. This change was introduced on 02/07/2021.
• GCC UI: while ordering EV SSL certificates, customers are required to provide EV specific information, including the Business Category. When investigating the incident, we noted that the option for the Business Category defaulted to “Private Organization” instead of a selection option. "Private Organization" would appear as the value selected by the customer, even if they had not consciously selected it as the Business Category relevant to them. We are changing this to force customers to take an action to select their Business Category: there will not be a default value anymore, so this action cannot be overlooked. This change will be introduced on 27/07/2021.

Comment 11

[Ryan Sleevi]

Thanks Eva. Comment #10 is exactly the sort of detailed explanation that I was hoping for.

Ben: Sending to you. I realize there's still a suggestion of an outstanding UI change (for 2021-07-27), but it doesn't seem like that would need to block resolution, as it's not strictly related to proximate misissuance, and more of a belt-and-suspenders approach.

Comment 12

[Mathew Hodson]

(In reply to Eva Van Steenberge from comment #10)

• GCC UI: while ordering EV SSL certificates, customers are required to provide EV specific information, including the Business Category. When investigating the incident, we noted that the option for the Business Category defaulted to “Private Organization” instead of a selection option. "Private Organization" would appear as the value selected by the customer, even if they had not consciously selected it as the Business Category relevant to them. We are changing this to force customers to take an action to select their Business Category: there will not be a default value anymore, so this action cannot be overlooked. This change will be introduced on 27/07/2021.

Are customers knowledgeable enough to correctly determine the applicable business category themselves? It would seem to be less risky if GlobalSign determined the value themselves and didn't rely on potentially incorrect values from the customer.

Comment 13

[Eva Van Steenberge]

As explained in Comment 2, the information provided in the Registration Number field was provided by the customer. This indicates that the customer was able to correctly identify the correct Business Category, but due to UI issues, did not record the correct value in the Business Category field.

The risk described is being addressed by the changes on a CRM level. These changes do not only cover the mis-match between a Business Category and a Registration Number, but also a mis-match between the value the customer picked for the Business Category and the Business Category as independently validated by our vetting agents.

Comment 14

[Eva Van Steenberge]

Changes to the GCC UI have been deployed in production as described on the 27th of July. This concludes the identified remedial activities - unless there are any further questions we believe this issue can be closed - thank you.

Comment 15

[Ben Wilson]

I will close this on Friday, 30-July-2021, unless any unresolved issues are raised.