1668007 - GlobalSign: Invalid stateOrProvinceName value

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Arvid Vermote]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Comment 1

[Arvid Vermote]

We have issued several certificates that have an incorrect stateOrProvinceName value containing the country of the subject. Refer to https://misissued.com/batch/180/ and https://misissued.com/batch/181/.

We are currently working on revoking the affected certificates and addressing the root cause. We wil provide a full incident report by Wednesday October 7 including correlation between this incident and Bug 1575880: "GlobalSign: SSL Certificates with US country code and invalid State/Prov".

Comment 2

[Eva Van Steenberge]

Attachment: Bug 1668007 - 20201007.xlsx

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

GlobalSign received a number of Certificate Problem reports, of which the first one was received on 25/09/2020 19:55 UTC.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Time (UTC)	Activity
25/09/2020 19:55	Certificate problem report #1 regarding ST fields with "Sweden" received on email.
25/09/2020 20:37	Certificate problem report #2 regarding ST fields with "Denmark" received on email.
27/09/2020 19:21	Compliance team starts investigation into reports.
27/09/2020 19:54	Compliance team responds to reporter that they confirm they are aware of the issue, will investigate further and will address the issue described.
29/09/2020 14:00	Compliance team responds to the reporter that they will be revoking the certificates
29/09/2020 11:00	Compliance team discovers root cause for Cloud State Whitelist issue.
29/09/2020 18:00	Fix for Cloud State Whitelist issue is released.
30/09/2020 14:00	Cloud State Whitelist is released in production.
30/09/2020	All certificates have been revoked before the deadline for Certificate problem report #1 and 2
30/09/2020 19:53	Compliance team confirms to the reporter that the certificate revocation has been completed for Certificate problem report #1 and 2
01/10/2020 22:08	Certificate problem report #3 regarding ST fields with "Chile" received on email.
01/10/2020 22:44	Compliance team confirms to reporter that they confirm they are aware of the issue, will investigate further and will address the issue described.
02/10/2020 07:00	Compliance team responds to the reporter that they will be revoking the certificates
03/10/2020 09:25	Certificate problem report #4 regarding ST fields with "El Salvador" received on email.
03/10/2020 10:36	Compliance team starts investigation.
03/10/2020 20:01	Compliance team responds to the reporter that they will be revoking the certificates or Certificate problem report #4
06/10/2020	All certificates have been revoked before the deadline for Certificate problem report #3
06/10/2020 15:57	Compliance team confirms to the reporter that the certificate revocation has been completed for Certificate problem report #3
08/10/2020	Scheduled revocation of the certificates to happen before the relevant deadlines for Certificate problem report #4

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

GlobalSign has updated all affected validated profiles and Cloud SSL certificates. We are currently reviewing which other certificates may be affected, and putting measures in place to stop the re-issuance of these certificates.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Refer to attached file.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Refer to attached file.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Country as state problem

Originally our internal policies did allow the Country name, written in full, to be used in the State field due to a wrong reading of 7.1.4.2.2. Subject Distinguished Name Fields, section f. "If the subject:countryName field specifies the ISO 3166-1 user-assigned code of XX in accordance with Section 7.1.4.2.2(g), the subject:stateOrProvinceName field MAY contain the full name of the Subject’s country information as verified under Section 3.2.2.1.". While this has since been fixed, the certificates that were issued in accordance with the wrong interpretation of this section of the BRs hadn't been addressed. While we created a whitelist for values of the subject:stateOrProvinceName field, we hadn't addressed this group of particular mis-issuances. Certificates with these values are currently being reviewed, and will be revoked if needed. GlobalSign will update this ticket with the relevant information.

Cloud Certificates bypassed the whitelist system

As mentioned in ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1654544#c5, we have the concept of Certificate Orders. Customers can update that Order over time by adding and removing SANs and they use the RV for that purpose. This creates a new certificate under that Certificate Order. Crucially, it's not just the Certificate Order that stays the same - both the date/time of application, as well as the issue date also remain the same. The whitelist flagged certificates with non-whitelisted values based on the date/time of the application of the Certificate Order. This allowed the new certificates, requested under the same Certificate Order, to not be included in the whitelist checks. This has been addressed by updating the whitelist check to also check existing Certificate Orders based on the update date, rather than the application date (which for these type of certificates is only recorded once for the initial Certificate Order request).

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Time (UTC)	Activity
30/09/2020 14:00	Cloud State Whitelist is released in production.
08/10/2020	Scheduled revocation of the certificates to happen before the relevant deadlines for Certificate problem report #4
11.10.2020 - 23:59	We are currently reviewing all the certificates specifically for Country names in the subject:stateOrProvinceName field, reviewing the evidences provided to confirm the existence of the relevant. We expect to have that investigation concluded by the end of this week (11.10.2020).

Comment 3

[George [:fozzie]]

Thanks for the initial incident report.

When did GlobalSign identify this misinterpretation of section f? After the first incident report for "Sweden" was sent or before it?

Comment 4

[Eva Van Steenberge]

Attachment: Bug 1668007 - 20201029.xlsx

We concluded the scan for the certificates including specific country names in the subject:stateOrProvinceName field on the 11.10.2020  20:00UCT. We found the list of certificates attached to include the country name in the subject:stateOrProvinceName, and revoked them all prior to 16.10.2020 20:00UCT. 

Our internal policies did change prior to the initial certificate problem report, but certificates issued before in accordance with the wrong interpretation of this section of the BRs, hadn't been addressed. This also meant that issuance could still occur after the policy was changed because not all affected validated profiles and Cloud SSL certificates were updated. Since then, we have put additional checks in place to make sure that any changes to internal policies are reviewed specifically to see if they could or should be applied retro-actively.

Additionally, similarly to the suggestion in https://bugzilla.mozilla.org/show_bug.cgi?id=1664328#c3, we are looking to revisit any changes made to the verification procedures made in the last 3 years and to review those changes to check if there is any need to examine historical issuance. We expect that review to be finished by the end of this calendar year. As Paul mentioned in that ticket (https://bugzilla.mozilla.org/show_bug.cgi?id=1664328#c4) we are also working on expanding both technical and human resource compliance bandwidth to ensure for each compliance-driven change the compliance team can, in parallel and using independent resources, perform an analysis on historical data wherever required. Current resources limitations within the compliance area require reliance on the engineering teams for some of these analyses. We expect to arrive at a level were each issuance analysis can be performed twice for all changes (by engineering and independently by compliance) by the first quarter of 2021.

Comment 5

[Ryan Sleevi]

The calendar year is now complete. Do you have an update?

Comment 6

[Eva Van Steenberge]

We revisited all changes to our internal verification procedures in the last three years and verified if we should have taken action on certificates that were already issued because of changes related to external requirements (i.e. similar to this ticket).

We performed an extensive review on the changes and categorized them according to the cause of the change. Based on the categorization, we then reviewed the historical data contained in the certificates, and, where applicable, also reviewed the certificate specific evidences.

Our analysis indicated that historical changes to our verification procedures did not require revocation of active certificates which were issued according to previous versions of the procedures. I hope this clarifies how we approached this review.

Comment 7

[Ben Wilson]

Eva, I'm still unclear on whether all remediation tasks have been completed. In Comment#4 you say you expect to complete the review "by the first quarter of 2021". Is that what you are saying in Comment#6 that you've now completed? Is there anything else remaining or that you are working on to control the contents of this certificate field in the future? Thanks, Ben

Comment 8

[Ben Wilson]

Can you provide any updates or clarifications on where you are to complete the remediation of this incident?

Comment 9

[Eva Van Steenberge]

Apologies for the delay in the reply and for the ambiguity. In our desire to want to provide detail on how we've completed the task, we haven't been clear that we have finished the above mentioned review by the end of 2020. I hope this clarifies that. Please let us know if there are any questions.

Comment 10

[Ben Wilson]

I will close this bug on or about Friday 12-Feb-2021.