1728796 - Entrust: Incorrect value in Business Category field for Government Entities

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Paul van Brouwershaven]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.84

Steps to reproduce:

On 1 September 2021, our verification team discovered 3 incorrect EV profiles during our re-validation process. The team discovered that the Business Category values had been set to “Private Organization” and that it should have been set to “Government Entity” based on the verification that they had performed and the documentation that was collected.

The subscribers were contacted to notify them that these certificates must be revoked and replaced within 5 days. A revocation deadline of 6 Sept 2021 11:30 UTC was set, and the certificates were scheduled for revocation by Entrust.

https://crt.sh/?id=5138483838
https://crt.sh/?id=5138566511
https://crt.sh/?id=5110548224

A full report will be posted here in the coming days.

Comment 1

[Paul van Brouwershaven]

Please note that the initial incident report incorrectly listed 3 instead of 2 EV profiles and only included 3 of the 4 crt.sh links.

Comment 2

[Paul van Brouwershaven]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 1 September 2021, our verification team discovered 2 incorrect EV profiles during our re-validation process. The team discovered that the Business Category values for these 2 profiles had been set to “Private Organization” instead of the correct value “Government Entity” based on the verification that they had performed and the documentation that was collected.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 1 September 2020: The EV Profile for the first government organization was set to “Private organization” during the initial verification for that profile.
• 8 September 2020: The EV Profile for the second government organization was set to “Private organization” during the initial verification for that profile. Please note that both EV profiles are part of the same customer account.
• 29 December 2020: 1 EV certificate was issued with an incorrect Business Category based on EV Profile 1 that was previously verified on 1 Sept 2020.
• 26 August 2021: 1 EV certificate was issued with an incorrect Business Category based on EV Profile 1 that was previously verified on 1 Sept 2020.
• 31 Aug 2021: 2 EV certificates were issued with an incorrect Business Category based on the profile that was previously verified on 8 Sept 2020.
• 1 September 2021 10:00 UTC: The two EV profiles came up for re-validation in our system, as they were approaching the 398 day expiration date. The agent started to perform the re-validation process by collecting the necessary documentation from the registry and noticed that the Business Category from the previous verification was incorrect. This change in values detected as part of a new UI change that was implemented to show changes to the certificate profiles in bug 1685370, which set off a red flag that something was wrong with the previous validation.
• 1 September 2021 10:30 UTC: The issue was escalated to our management team in Europe for review.
• 1 September 2021 11:30 UTC: The problem was confirmed by our compliance team. It was confirmed that 3 certificates were issued with incorrect Business Category value.
• 1 September 2021 11:35 UTC: The subscribers were contacted to notify them that these certificates must be revoked and replaced within 5 days. A revocation deadline of 6 Sept 2021 11:30 UTC was set and the certificates were scheduled for revocation by Entrust in case the subscriber does not revoke the certificates themselves before the deadline.
• 2 September 2021 14:00 UTC: An internal meeting is held to interview members of the Verification team as part of the investigation and to discuss a remediation plan with our Product Management and Engineering teams.
• 3 September 2021 12:30 UTC: All four certificates revoked ahead of schedule.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The 2 EV profiles were corrected as 1 September 2021 and all future certificates for this subscriber will reflect the correct Business Category.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

4 certificates were issued with invalid Business category “Private Organization”
https://crt.sh/?id=5138483838
https://crt.sh/?id=5138566511
https://crt.sh/?id=5110548224
https://crt.sh/?id=3845699227

5. The complete certificate data for the problematic certificates.

https://crt.sh/?id=5138483838
https://crt.sh/?id=5138566511
https://crt.sh/?id=5110548224
https://crt.sh/?id=3845699227

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Similar to bug 1685370, this problem was introduced due to human error during the initial verification of the 2 EV profiles in this customer account.

As part of bug 1685370 and bug 1696227, we have introduced new UI into our verification system that makes it easier to detect these types of issues. As a reminder to the community, we introduced UI to clearly highlight any changes that occur during re-validation and to clearly show fields in the Verification profile that will be part of the subject DN. We have received feedback from our verification team that these UI changes have already been paying dividends and have helped our second level approvers identify potential mis-issuances before they were ever approved. The fact that we were able to detect these particular mistakes as part of the re-validation process demonstrates that these other enhancements are working. For example, in bug 1685370, we had performed multiple re-validation starting in 2013 without detecting the invalid Business Category. However, in this case, we were able to detect the problem as soon as the profile came up for re-validation.
As part of bug 1685370, we performed a full scan on our certificate population to check for any additional Government entity profiles that were incorrectly labeled as another type based on key words in the Organization name field such as County, State, Government, Agency, Department, City, Province. However, this scan did not take into consideration the various languages that our customers use in the organization name field. In this case, both of these EV profiles would have been flagged had we checked for the terms Department or Government in Finnish.

One of the other reasons why this mistake may have occurred is due to the formatting of the registration number for Government entities in Finland. In discussing the issue with our verification team, one of the things that stood out to them was the fact that this government entity had a registration number, which is not something that is common in many countries. This was likely one of the reasons why Private organizations was selected.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We will perform additional scans on our certificate population to check to see if there are any other certificates or EV profiles that contain the following key words that have Business Category != Government Entity:

County, Ministry, Government, Agency, Department, City, State, and Province

We will be checking for values in multiple languages based on the most common countries and local languages where we issue certificates.
In addition, we will also implement a system enhancement that will flag any verifications where these above words are used in the Organization name field and the business category is not set to “Government Entity”. We will implement a new UI to highlight this potential issue to make sure that the issue is resolved or that an appropriate explanation is provided as to why we would allow this business category value for the organization.

In order to address the potential issue related registration numbers for government entities in Finland, we will be implementing registration number formatting checks based on the way registration numbers are formatted in Finland and other countries. The goal will be to implement this check for a small number of countries in initially and continue to add more checks as we conduct additional research.

We will commit to timelines over the next couple of weeks as we discuss the plan internally with our Product Management and Engineering teams.

Comment 3

[Paul van Brouwershaven]

Our Product Management and Engineering teams are working out the details on the system enhancements and scanning our certificate population in other languages as stated in the incident report.

We will commit to timelines over the next couple of weeks.

Comment 4

[Paul van Brouwershaven]

Over the past few weeks, we checked our database in some key languages. Using this scan we identified and corrected two more profiles with the same issue, but these had not been used to issue any certificates. We continue to scan our database for other languages and extend our search to find matches without diacritics.

For the release scheduled for December 2021, we have been able to free up limited resources to implement a system enhancement that will flag any organizations with a matching keyword and where the business category is not set to “Government Entity” as described previously.

In addition, we plan to implement the registration number format checks for a limited number of sources in our March 2021 release, the coverage will be extended over time. Therefor this system change will get more value in the longer term.

The decision for this schedule is made based on the short-term effectiveness of the changes and the available resources.

Comment 5

[Paul van Brouwershaven]

Correction we plan to implement the registration number format checks for a limited number of sources in our March 2022 release.

Comment 6

[Paul van Brouwershaven]

The UI mockups were approved last week and are currently being implemented.

We are currently on track to deliver in the release scheduled for December 2021 as communicated previously.

Comment 7

[Ben Wilson]

Do you have a suggested "Next update" so that I can set that and relieve the weekly reporting requirement?

Comment 8

[Paul van Brouwershaven]

The UI changes have been implemented and are now in the QA process.
We expect that these changes to be released as planned in early December.

I suggest a next update for half November.

Comment 9

[Paul van Brouwershaven]

The QA process is completed, we expect these changes to be released in early December as planned.

I suggest we provide a next update when the release has been deployed.

Comment 10

[Paul van Brouwershaven]

The new feature has been released to production as planned.

If there are no further questions, I suggest closing this bug.

Comment 11

[Ben Wilson]

I will close this on Friday, 10-Dec-2021, unless anyone raises reasons why it should be kept open.