1696227 - Entrust: Incorrect Jurisdiction Country Value in an EV Certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Dathan Demone]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0

Steps to reproduce:

During an internal review, the Entrust Verification team discovered a single EV certificate with an incorrect Jurisdiction Country value.

https://crt.sh/?id=4064079643

We are conducting a full investigation and will provide a full report by Monday, March 8.

The issue was confirmed today - Wednesday, March 3 at 14:15 UTC.

The certificate will be revoked before Monday, March 8th before 14:15 UTC.

Comment 1

[Dathan Demone]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On Wednesday, March 3 at approximately 14:15 UTC, the Entrust Verification team discovered an EV certificate that was populated with an incorrect Jurisdiction country value.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

28 January 2020 – The subscriber applies for Extended Validation

6 February 2020 – The subscriber’s EV profile is approved by Entrust after all of the required checks are completed. At this time, the Jurisdiction country is set to the value of “ZA” when it should have been set to “BW”

6 February 2020 – 2 EV certificates are issued with incorrect Jurisdiction country values based on the incorrect EV profile. Note, these 2 certificates expired on 27 February 2021

11 February 2021 – 1 EV certificates is issued with an incorrect Jurisdiction country value based on the incorrect EV profile

3 March 2021 14:00 UTC – During the re-validation process for this EV profile, a Verification agent discovers that there might be a problem with the previously verified EV profile and notices that the country value for the place of business does not match the Jurisdiction country. The issue is escalated to management for immediate review.

3 March 2021 14:15 UTC – The issue is confirmed and an internal investigation is kicked off. Interviews are conducted with the Verification agents involved with the original EV profile verification from January/February 2020.

3 March 2021 15:30 UTC – The EV profile Jurisdiction Country value is updated to “BW”

3 March 2021 15:35 UTC – The subscriber is notified that they should re-issue their certificate immediately and that the original certificate will be revoked on or before 8 March 2021 14:15 UTC

3 March 2021 16:00 UTC – Additional internal discussions are held to determine the root cause and to get feedback on how this mistake can be avoided in the future.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The EV profile for this specific customer was corrected on 3 March 2021 so that any new certificate that is issued for that profile will include the correct Jurisdiction Country value.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

A single EV certificate was impacted by this issue

https://crt.sh/?id=4064079643

During our investigation, we found that 2 other expired EV certificates were issued in February 2020

https://crt.sh/?id=2569669700
https://crt.sh/?id=2423336069

5. The complete certificate data for the problematic certificates.

https://crt.sh/?id=4064079643

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As detailed in the timeline of events, the issue was introduced for this EV profile back on 6 February 2020 when the profile was verified incorrectly. As the profile approached the 13-month data re-use limit, the Entrust verification team was in the midst of performing a re-validation of the data and noticed that the information that was previously verified was not correct.

During our interviews with the Verification team members who worked on this original verification, it was determined that this incorrect value was not approved due to any issues with the verification process or documentation that was collected. Similar to bug 1685370 ( https://bugzilla.mozilla.org/show_bug.cgi?id=1685370 ), the mistake was made due to UI deficiencies in our Vetting system. In our verification system, there are numerous fields in the EV profile vetting record that relate to other non-certificate fields that are an important part of the verification picture. In this particular case, the reseller was based in ZA, along with one of the subscriber contacts. The value “ZA” correctly appeared in numerous non-certificate fields, which may have contributed to the mistake.

Another point to note is that Entrust issues many certificates in ZA and often uses a QGIS known as “CIPC”. The QGIS that was used for this particular profile in BW is known “CIPA”. It was noted during our investigation that the agent mistook “CIPA” for “CIPC”, which also led to the agent entering the incorrect value of ZA. Also, note that the field for the Jurisdiction Country value in our vetting system is currently a country drop-down that is selected by the agent that is independent of the vetting source being used.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Although this bug has a different outcome and root cause from bug 1685370 ( https://bugzilla.mozilla.org/show_bug.cgi?id=1685370 ), there are some similarities in terms of our verification UI leading to human errors. In the case of this incident and incident 1685370, it was noted during our investigation that the error would likely have been caught had there been a UI that clearly distinguished the certificate fields from the non-certificate fields. As noted in 1685370, we are planning to enhance our UI to add a section that clearly shows which fields in our vetting records will populate the certificate Subject DN fields. We believe this will prevent future issues like these from occurring and strengthen the two-level process that we have in place for both OV and EV. This change will be implemented in our July 2021 release.

Another change that we will implement is to populate the Jurisdiction country based on the vetting source. This change is also being scheduled for our July 2021 release.

Comment 2

[Dathan Demone]

The certificate was revoked on 5 March 2021 at 13:13: UTC

Comment 3

[Dathan Demone]

We will be starting the work for the UI changes I described above in our next development sprint. We are still on track to deliver these changes in our July 2021 release.

Comment 4

[Dathan Demone]

Mockups and Wireframes for the UI changes have been completed and we are still on track to complete the changes described earlier by end of July.

Comment 5

[Dathan Demone]

Development on the UI changes is scheduled to start in our next sprint on May 5th

Comment 6

[Dathan Demone]

The engineering work is completed for the UI changes. The changes are being tested by our QA and verification teams.

We were also able to accelerate the deployment of these changes and they are now scheduled for the second half of June 2021.

Comment 7

[Dathan Demone]

We are still on track to deliver the UI changes in the second half of June.

Comment 8

[Dathan Demone]

All of the system enhancements described in this report have been fully implemented as of today as part of our latest release. Please let me know if there are any other questions.

Comment 9

[Ryan Sleevi]

Dathan: I want to make sure that Entrust is following dev-sec-policy and is aware of this discussion. As with Bug 1685370, it suggests Entrust may not be.

Ben: Beyond my question on Bug 1685370, I don't have any follow-ups, and suspect you can close this out.

Comment 10

[Dathan Demone]

Ryan: I was not personally following this recent discussion but I do think it helps clarify the expectations moving forward. I thought that my last comment was an indication that this bug could be closed and I was waiting to see if anyone had any questions. Moving forward, we will follow Ben's recommendations. Thank you for pointing this out.

Comment 11

[Ben Wilson]

I think we can close this bug and will plan to do so on or about this Friday, 16-July-2021.