1658932 - GlobalSign: Incorrect Jurisdiction of Incorporation information for Japan

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Eva Van Steenberge]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36

Comment 1

[Eva Van Steenberge]

It was discovered that three EV SSL certificates included Jurisdiction of Incorporation information on State and Locality level for Japan, where it is deemed that the incorporation happens at country level. 

The three certificates are:

https://crt.sh/?serial=62d9245406c9228e44294c97 (discovered today , 13th of August 2020 at 6.35 BST)
https://crt.sh/?serial=04e17df642d83e30d474f183 (discovered after further investigation at 8.30 BST)
https://crt.sh/?serial=1e7ba8523fc6aeab55bb9c04 (discovered after further investigation at 8.30 BST)
We are in the process of working with the customer to replace these certificates, and they will be revoked by Tuesday 18th of August 2020 at 6.35 BST.

We are conducting both an investigation and further analysis on other jurisdictions. We expect to give a detailed report, following the usual format, as soon as we have concluded our analysis, but no later than Friday 21st of August 2020, EOD.

Comment 2

[Eva Van Steenberge]

This is an intermediate update to confirm that we have revoked the original three certificates, and are on track to provide the more detailed report no later than Friday 21st of August.

Comment 3

[Eva Van Steenberge]

Attachment: 1658932 - Update 20200821.xlsx

Comment 4

[Eva Van Steenberge]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

While we were reviewing the Jurisdiction of Incorporation information for the upcoming publication of the Incorporating Sources, we discovered an initial first certificate (https://crt.sh/?serial=62d9245406c9228e44294c97) on the 13th of August 2020 at 6.35 AM BST which included redundant Jurisdiction of Incorporation information. This was internally escalated at 7.12 AM BST, and further investigated specifically for the jurisdiction of Japan at 8.10AM BST. After the initial scan ran, we discovered a further 2 certificates at 8.30 AM BST. We immediately started the process to replace these certificates with the customer, and revoke them by the deadline mandated by the Baseline Requirements.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All actions were focused around these areas:

• Investigation into the root causes, in order to address these;
• Analysis of other jurisdictions, including finding other affected certificates
• Remedial and preventative actions
• Public reporting

Time (BST)	Description
13/08/2020 6.35 AM	discovery of initial certificate with redundant Jurisdiction of Incorporation information
13/08/2020 7.12 AM	internal escalation for further review.
13/08/2020 8.10 AM	confirmation of the issue, further scan for more certificates. Incident ticket created.
13/08/2020 8.30 AM	2 additional certificates found. Kick off to replace certificates in order to revoke
13/08/2020 8.32 AM	Escalation of incident to wider compliance group. Start root cause investigation. This will continue for the rest of the day.
13/08/2020 9.30 AM	Set up of compliance review mechanism for newly ordered EV certificates.
13/08/2020 11.30 AM	Draft first post to Bugzilla
13/08/2020 3.21 PM	First post Bugzilla
14/08/2020 1.00 AM	Training goes out to all vetting teams globally informing the validverification agents of the potential issue and what to look out for.
14/08/2020 8.00 AM	Kick-off gathering JOI level information from our localization guide for weekend scan. This information is also to be used in updated escalation filter
15/08/2020 9.00 AM	Start weekend scan - all Jurisdictions, all Business Categories.
17/08/2020 8.00 AM	Result weekend scan shared, review of potential certificates with redundant Jurisdiction of Incorporation information.
17/08/2020 9.00 AM	Confirmation of individual certificates. Kick off to replace certificates in order to revoke
17/08/2020 2.00 PM	Deployment of updated escalation filter in production, but access only to Compliance department for testing purposes.
18/08/2020 8.30 AM	Notification to vetting management about deployment of updated escalation filter, scheduled following day.
19/08/2020 8.30 AM	Deployment of updated escalation filter to whole of vetting.,
21/08/2020	All revocations completed

Revocation date are published in attached spreadsheet.

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

GlobalSign has not stopped issuing EV Certificates. We have put additional controls in place to mitigate the problem, and have addressed the customer profiles which contained redundant Jurisdiction of Incorporation information.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Refer to the attached spreadsheet.

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

Refer to the attached spreadsheet.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

There are a few different factors that have contributed in different gradations to the materialization of this incident:

Visibility of the fields in the UI.

Jurisdiction of Incorporation Locality and Jurisdiction of Incorporation State are generally terms that are confusing for customers. It is difficult to explain to a lay-person what the difference is between the normal Locality and State/Province fields (which relate to physical address), and the jurisdiction of incorporation. In order to avoid any confusion, the irrelevant Jurisidiction of Incorporation fields were removed from the customer's UI. This was only done for those jurisdictions where there is known level of incorporation, and where that level of incorporation is pretty static regardless of the type of organization. Crucially though, this wasn't enforced via the API, and some orders got through including this information.

The restriction in the UI was also mirrored in the back-end where GlobalSign's verification agents can view (and edit) the order details. Editing details can only be done by verification agents in GCC. After initial authentication in GCC by the first verification agent, the order is received in the RA for issuance by the second verification agent who performs the Due Diligence checks. In the RA, the information can not be changed - this is done to enforce these Due Diligence checks.

In these cases, the verification agent reviewed the orders in GCC, where not all the information was visible. The information was visible in the RA, but the field name shows as the OIDs. verification agents missed this information, as they didn't realize that these populated fields were showing up in the certificate.

Changes made:

• Additional training for all agents on what information goes in the certificate, based on the RA. This has been completed on Monday.
• We are working on fixing the visibility issues in GCC. In the meantime, a change has been made to clearly present the agent with all the information on the cases they work on instead, including fields with blank values.

System controls and escalation filters

Jurisdiction of Incorporation levels are dependent on a number of factors. Of course, in first instance, there is the Country of incorporation itself. However, that doesn't determine the level by itself. The level of incorporation will depend on the type of entity. JOI has, as a concept, a stronger link to Private Organizations. However, the JOI doesn't always correspond with Business Category as defined in the EV Guidelines. For example, in Vietnam, Security Companies are registered at a Country level. Other companies are registered at Country ans State level. Both Security Companies and regular Companies are Private Organizations as per the EV Guidelines. There is no set of rules for Government Entities. There is also a factor of the unknown - for Jurisdictions we haven't dealt with, we investigate this when we receive the first order.

Because there is an element of "it depends", GlobalSign has historically opted not to systematically enforce the Jurisdiction of Incorporation levels, except in some clear cases via the UI (as described above). Instead, we opted to include the Jurisdiction of Incorporation guidance in our Country database, which also includes our approved sources. We call this our Localisation Guide. Our Verification Procedures, our training and our internal audit framework reference the information about Jurisdiction of Incorporation in the Localization Guide as the primary source when it comes to Jurisdiction of Incorporation levels, especially for Private Organizations. Jurisdiction of Incorporation levels in our Localisation Guide are set by Compliance. We have more general procedures, which historically didn't include escalation to Compliance for non-Private Organizations.

The check for relevant Jurisdiction of Incorporation information is a mandatory check performed as part of our 3% internal audit. This was done on an individual certificate basis, rather than a structural per-Jurisdiction check. We addressed this with the escalation filter prior to issuance.

We have introduced escalation directly to Compliance for requests for non-Private Organizations specifically since the 20th of April this year, after a customer (State Police in Germany) didn't fit the mold of the regular Jurisdiction of Incorporation level. We have reviewed all of these certificates during our weekend scan. Next week, we will review the evidences of the non-Private Organizations that do have the expected Jurisdiction of Incorporation levels as defined in our Localisation Guide, to make sure the information included in the certificates is appropriate for the named entity.

We have updated that escalation filter, similarly to as suggested in the following incident https://bugzilla.mozilla.org/show_bug.cgi?id=1623356. The escalation filter now works as follows:

• Any non-Private Organization is being escalated to compliance for additional scrutiny with regards to the JOI.
• Any country where we have not defined the JOI levels for, is being escalated to compliance to determine the level of JOI.
• For Private Organizations, we have set flags for anything that doesn't fit at least one of the defined JOI levels. What this means in case of a jurisdiction where multiple JOI levels are possible, is that every single order is flagged for review to make sure the appropriate JOI level for the specific organization is selected. We are still looking if we can refine this, for example based on the format of the Registration Number, because this does cause a number of false positives.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Time (BST)	Description
14/08/2020 8.00 AM	Gathering of all information with regards to the already defined Jurisdiction of Incorporation levels in our localization guide. This information was going to be used to run a report over the weekend to highlight potential uncompliant certificates, as well as feed into the escalation filter.
17/08/2020 2.00 PM	We deployed the updated escalation filter in production. This was first made available only for Compliance, which monitored the flagging for the first 2 days to make sure everything was working as expected. Compliance acted on the limited number of escalations, by reaching out to the individual verification agents.
18/08/2020 8.30 AM	Vetting Management was notified that this escalation filter was going to be introduced the next day for the wider vetting department, if the second day of testing was successful.
19/08/2020 8.30 AM	This updated escalation filter has been introduced for the whole of vetting.
22/08/2020 8.00 AM	Completion of the revocation of certificates with redundant jurisdiction of incorporation information.
By 28/08/2020, EOD	Finishing review individual certificates for non-Private Organizations that matched the general Jurisdiction of Incorporation Level definition in our Localisation guide. Further update to this ticket.

Comment 5

[Eva Van Steenberge]

The process for review of non-private organizations is ongoing and we've completed approximately 80% of the set of certificates. The remaining 20% of individual certificates requires further extended analysis.

Government Entities are not always created by law - sometimes it's a case of recognition after the fact, rather than creation. Even when a Government Entity was created by law, we found that recognition levels change over time. Something that was created at Country level may have been devolved to be a State matter (for example, fusion of municipalities in Belgium).

Since this requires a lot of local knowledge, we are reaching out to relevant vetting teams within GlobalSign to help us understand local situations which we as a Compliance team, may be less familiar with. We have planned additional two weeks of capacity for the team to review the individual cases that require additional clarification.

Meanwhile, the escalation filter remains in place for non-Private Organisations and any new requests will be reviewed and taken into account during this effort.

Comment 6

[Eva Van Steenberge]

Attachment: 1658932 - Update 20200914.xlsx

The additional review has revealed another 37 certificates (for 11 customers) which included wrong Jurisdiction of Incorporation information. In these cases, the Jurisdiction of Incorporation level was set the same as for Private Organizations in the relevant Jurisdictions. After initial investigation and identification of potentially wrong certificates by the compliance incident team, Subject Matter Experts were involved for further review. On Monday 7th of September, we confirmed that these certificates didn't meet the requirements with regards to the Jurisdiction of Incorporation information and therefore the certificates have been revoked by the Saturday 12th of September 8.30AM BST and 11.30AM respectively. We have added the crt.sh links and the revocation deadlines in the attachment.

Comment 7

[Ben Wilson]

I have set the next update for 1-Oct-2020. Meanwhile, what remains to be done for remediation of the underlying issue(s) related to this bug?

Comment 8

[Eva Van Steenberge]

In line with the Baseline Requirements, GlobalSign has published a list of Incorporating and Registration Agencies and the relevant Jurisdiction of Incorporation information. It is important to note that Incorporating and Registration Agencies are not used for all business categories. We continue to have the escalation filter in place for Government Entities which are not subject to an Incorporating and Registration Agency check.

Please let us know more information is required or this ticket can be closed.

Comment 9

[Ben Wilson]

Thanks, Eva. I will schedule this to be closed on our about 16-October-2020 unless there are other questions or issues raised by anyone.