1685370 - Entrust: Incorrect Business Category Value Discovered in an EV SSL Certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Dathan Demone]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0

Steps to reproduce:

During a regular EV re-validation process, we discovered that a previously validated EV business profile had a business category that was set to an incorrect value, resulting in the mis-issuance of a single EV SSL Certificate on 18 March 2020. We are conducting an investigation and will post a full incident report in the coming days.

Comment 1

[Dathan Demone]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 5 January 2021 at approximately 17:00 UTC, the Entrust Verification team discovered a single EV SSL certificate with an invalid Business Category value. This was discovered during a regular EV business profile re-validation. It appears that during a contact update change on the business profile in 2019, the business category was inadvertently updated by an agent from its correct value (Government entity) to an incorrect value (Private Organization).

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

28 January 2019: EV validation was performed for the organization and the correct Business Category field (Government Entity) was set at this time. Note that many other EV verifications had been performed for this profile, dating back as far as 2013.

18 April 2019: The subscriber contacted Entrust to request a contact change for one of their verified contacts. During this contact update, the first level Verification Specialist accidentally changed the Business Category field from “Government Entity” to “Private Organization”. Not expecting there to be any changes to the Business Category field during this contact change, the second level Verification Audit Specialist approved the changes to the EV business profile

13 January 2020: EV re-verification was performed by Entrust on this business profile, as it was approaching the end of the 13-month data re-use period. At this time, the incorrect Business Category from the previous verification was not detected and the same Business profile data was re-approved for the next 13 months along with the rest of the EV business information.

18 March 2020: The subscriber issued an EV SSL certificate based on the EV business profile with the incorrect Business Category value.

5 January 2021 18:00 UTC: The Entrust Verification team discovered the incorrect Business Category value for the business profile during a re-verification check.

5 January 2021 18:05 UTC: An internal investigation is started

5 January 2021 19:00 UTC: The issue is confirmed by the Entrust compliance team

6 January 2021 15:43 UTC: The subscriber is notified that the certificate must be revoked within 5 days. A 5-day revocation deadline is set for 10 January 2020 at 19:00 UTC.

8 January 20201 17:20:35 UTC: The certificate is revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The business profile for this specific customer has been corrected as 5 January 2021 and all future certificates for this subscriber will reflect the correct Business Category.
In addition, a scan was performed on all other EV certificates that contain the keywords “County”, “Government”, “Agency”, “Department”, “City”, and “Province” in the Organization Name field, and none of these profiles were set to Private Organization or Business Entity.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

A single EV certificate was impacted by this issue

https://crt.sh/?id=2598420157

5. The complete certificate data for the problematic certificates.

https://crt.sh/?id=2598420157

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Entrust performs validation on existing EV business profiles every 13 months, as per the requirements in the EV guidelines. The original mistake that introduced the incorrect Business Category value was made in April 2019 and was not detected in the next re-verification in January 2020. To explain how this issue was introduced and not detected, we need to explore what happened on 18 April 2019 and also on 13 January 2020.
After conducting multiple interviews with senior agents along with agents who were involved with the original mistake on 18 April 2019, we found that the main reason this mistake was first introduced is due to a lack of highlighting changes with the previous verification data when updating a business profile in our Verification system. When this incident was being reviewed on 5 January 2021, it was immediately clear that the Business Category should have been set to Government entity based on the name of the organization (it was a County) and the supporting verification documents that were collected. This was not an issue with understanding the verification process for the Business Category. In addition, we received feedback that our current interface does not clearly list which fields in our EV verification tickets are being used as values in the certificate Subject DN.

With respect to the re-verification of this incorrect data on 13 January 2020 that eventually led to the certificate including the incorrect business category, it was determined that this was ultimately a case of human error that was driven by potentially confusing fields in our Verification system when dealing with Government entities. When reviewing the Jurisdiction information for this EV business profile, there are two fields labeled “Incorporating Agency” and “Registration Agent” that were both set to “Government Entity, in addition to a field labeled “Business Category” that was set to “Private organization”. The drop-down values for “Incorporating Agency” and “Registration Agent” fields should not use the term “Government entity”, as it can easily be confused with the Business Category value that will eventually appear in the certificate.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

In order to reduce the risk of future incidents like this one where information is inadvertently updated and not detected by a second-level verification agent, we are proposing major UI changes that will clearly show what information has changed from a previously verified business profile, especially information that will appear in the certificate Subject DN.

In addition, we will also introduce a UI that clearly displays what information is going to appear in the certificate Subject DN when a business profile is being verified so that these can be distinguished from other fields that we use to track evidence sources and contact information, for example.

We will also change the drop-down values for the two tracking fields “Incorporating Agency” and “Registration Agent” so that they do not include values such as “Government entity”, which is the exact same value that would appear in the Business Category drop-down and the certificate subject DN.

We have not yet identified a target release date for these changes. As a next step, we will work with our engineering and Product Management teams to schedule these changes in a future release.

Comment 2

[Ryan Sleevi]

Thanks Dathan.

I'm actually encouraged to see this report, because I believe it's a sign of Entrust improving on its incident reporting. From it, I believe I was able to get an understanding of what went wrong, and it also appears that your mitigations - looking at the systemic factors such as the human interface - are meaningful to address root causes, as opposed to suggesting it's an operator training issue (with the operators being trained to use the existing interface).

I'm setting N-I to get a concrete date on when you expect to have things scheduled. It's important, when considering incidents, to understand that the CA is able to prioritize ensuring the good and correct results, and so we'd like to get a clearer commitment as to deliverable dates. If there are planning meetings to be held and impact assessments to be considered, we'd like a clear commitment about when to have those done, as well as regular updates to ensure that "timeline-to-a-timeline" is met. That said, I'm encouraged, especially in light of past incidents, to see quality reports like this.

Comment 3

[Dathan Demone]

Thanks, Ryan. We have meetings scheduled next week to discuss a plan to implement these UI changes. I will provide an update by the end of the next week with a timeline.

Comment 4

[Dathan Demone]

Our engineering and Product Management team met today to discuss the UI changes described in this bug. Unfortunately, these requests are coming at the tail end of a release that the engineering team is currently working on. We will address these UI enhancements in the following release cycle, which is scheduled for delivery in July.

I will provide additional updates as needed as we work through the design process to provide updates on any additions or deviations to the plan we described in this bug report.

Comment 5

[Dathan Demone]

We will be starting the work for the UI changes I described above in our next development sprint. We are still on track to deliver these changes in our July 2021 release.

Comment 6

[Dathan Demone]

Mockups and Wireframes for the UI changes have been completed and we are still on track to complete the changes described earlier by end of July.

Comment 7

[Dathan Demone]

Development on the UI changes is scheduled to start in our next sprint on May 5th

Comment 8

[Dathan Demone]

The engineering work is completed for the UI changes. The changes are being tested by our QA and verification teams.

We were also able to accelerate the deployment of these changes and they are now scheduled for the second half of June 2021.

Comment 9

[Ryan Sleevi]

Dathan: Thanks for the update. Note that, unlike Bug 1696227, this doesn't have a NextUpdate presently set, and https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed includes the following language:

You should also provide updates at least every week giving your progress, and confirm when the remediation steps have been completed - unless Mozilla representatives agree to a different schedule by setting a “Next Update” date in the “Whiteboard” field of the bug

Comment 10

[Dathan Demone]

Thanks for pointing this out.

We are still on track to deliver the UI enhancements for the second half of June. I should have a more specific date in one of my next updates.

Comment 11

[Dathan Demone]

We are still on track to deliver the UI changes in the second half of June.

Comment 12

[Dathan Demone]

All of the system enhancements described in this report have been fully implemented as of today as part of our latest release. Please let me know if there are any other questions.

Comment 13

[Ryan Sleevi]

Dathan: Is there any chance you could share screenshots of what this UI looks like, so that other CAs can learn and consider?

Comment 14

[Ryan Sleevi]

Ben: This can probably be closed out. While I left a request in Comment #13, I don't think that would block closure here. Entrust's explanation in Comment #1 gives a rough idea of the changes.

While Entrust struggled with updates at first (and this took overall 6 months), the later comments definitely showed improvement and awareness of expectations, which is really positive (along with the improvements commented on in Comment #2)

Comment 15

[Dathan Demone]

Attachment: Changes to Profile - Orange Badge Notification.png

Comment 16

[Dathan Demone]

Attachment: Changes to Profile - Final Approval by Second Vetter.png

Comment 17

[Dathan Demone]

Attachment: Business Profile - DN Summary.png

Comment 18

[Dathan Demone]

I added 3 screenshots to show the major UI changes that were discussed.

The first screenshot " Changes to Profile - Orange Badge Notification" is what the first level verification agent will see when they perform re-validation of a business profile. The orange badge is an indication that some field data has changed from the previous validation. In all cases, it will raise questions as to why this change was made. The idea is to prevent changes to the information without additional scrutiny, which will help us to identify issues and avoid human error.

The second screenshot " Changes to Profile - Final Approval by Second Vetter" is what the second verification approver will see when they are completing the verification process. We added an additional checkbox where the second level approver must acknowledge that they reviewed the changes, supporting evidence, and the reasons for these changes.

The third screenshot "Business Profile - DN Summary" is where all Verification agents can see a preview of the certificate DN based on the business profile fields to ensure they are paying extra attention to these values.

Comment 19

[Ben Wilson]

I will plan to close this bug on or about this Friday, 16-July-2021.