1593357 - QuoVadis: Incorrect EV businessCategory

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Stephen Davidson]

This disclosure has been reported in detail in https://bugzilla.mozilla.org/show_bug.cgi?id=1590171 but is being repeated in a new bug at the request of Ryan Sleevi.

Comment 1

[Stephen Davidson]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1590171#c10

Hi Ryan: As noted before, I responded to the problem report within an hour of receiving it. There was a mixup across internal email systems, and neither I nor others copied on the email caught it. As noted before, the original certificate reported by Cynthia Revstrom led to a greater investigation described below.

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We received a problem report from Clint Wilson at DigiCert who had received an email from Cynthia Revstrom regarding a certificate issued to the National Library of Scotland with possible incorrect use of “Non-Commercial Entity” in businessCategory field.

https://crt.sh/?q=362705555bb4fa780a0050e96cbf6b682f2796c5335dc8edd6fd843618a4b003

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

14 Oct 2019 20:30 GMT Received problem report from Clint Wilson at DigiCert. Ideally problem reports should be directed to [e-mail].
15 Oct 2019 21:30 GMT Following investigation, instructions given to replace and revoke certificate. Response was sent to Cynthia Revstrom (discovered to have not been delivered following the opening of this bug).
16 Oct 2019 Additional investigation identifies more certificates at National Library of Scotland; instructions given to replace and revoke.
18 Oct 2019 Research started into larger issue with EV clients miscategorized as “Non-Commercial Entity.”
20 Oct 2019 14:30 GMT additional problem report received from Cynthia Revstrom to [e-mail].
20 Oct 2019 15:06 GMT Response sent by QuoVadis (discovered to have not been delivered following the opening of this bug). I have reached out to Cynthia Revstrom to apologise.
21 Oct 2019 12:53 GMT Initial certificate for National Library of Scotland confirmed revoked.
21 Oct 2019 23:30 GMT Following investigation, an additional 66 certificates were identified for revocation, customers contacted starting morning of 22 Oct 2019. The investigation continues.
22 Oct 2019 all 20 certificates at National Library for National Library of Scotland confirmed revoked.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

QuoVadis has reviewed all clients with the “Non-Commercial Entity” for EV businessCategory. We stopped issuing certificates resulting from the miscategorisation on 18 Oct 2019.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There were 20 problem certificates issued to the National Library of Scotland certificates. The first was issued on 18 Feb 2018 and the last on 11 Jan 2019.

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The attached batch 1 includes the initial certificate reported by Cynthia Revstrom and others at the same customer. In our batch revalidation of clients tagged as “Non-Commercial Entity” we have identified another 66 certificates, attached as batch 2. This includes the certificates identified by Cynthia Revstrom in her problem report of 20 Oct 2019. The investigation continues.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

RAs believed that “Non-Commercial Entity” applied to charities and other non-commercial entities. This was not the specification laid out in QuoVadis training manuals.

We notice that this same “Non-Commercial Entity” miscategorisation is widespread amongst EV issuing CAs internationally.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

As part of the integration of QuoVadis into DigiCert, the QuoVadis compliance team now reports to DigiCert. In addition, an additional experienced internal auditor has been hired by DigiCert whose role includes the 3% review of QuoVadis TLS issuance.

Previously TLS validation was performed by separate national subsidiaries of QuoVadis for their respective client bases. Since the acquisition by DigiCert, QuoVadis validation has been centralized into one team with a group leader, and reports into the larger DigiCert validation group. Three highly experienced validation staff from DigiCert have been seconded to the group.

Short term, updated training has been provided to QuoVadis RAs relating to EV JOI and businessCategory fields.

Longer term, QuoVadis TLS issuance will move to DigiCert’s platforms, benefiting from DigiCert’s significant investment in automation and validation tools. A roadmap will be provided at a later date.

Comment 2

[Stephen Davidson]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1590171#c14

For the sake of consistency, QuoVadis confirms here the revocation of the certificates identified in Batch 2 by end of day 10/27. A further update will be provided this week.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1590171#c15

For the sake of consistency, QuoVadis continues this disclosure here. As noted above we continued our investigation, and a third batch of certificates was identified. The issue was communicated to affected customers on Friday 25 October, and completion of the revocation of the certificates was completed by end of 30 October. We continue to investigate EV fields in QV-issued certificates using the enhanced tools and methodologies used by DigiCert.

Comment 3

[Stephen Davidson]

Attachment: Batch 1

Comment 4

[Stephen Davidson]

Attachment: Batch 2

Comment 5

[Stephen Davidson]

Attachment: Batch 3

Comment 6

[Stephen Davidson]

Historically QuoVadis has operated its own certificate management system (“Trustlink”) and PKI from datacenters located in Bermuda, Switzerland, and the Netherlands. In addition to managing the issuance of TLS certificates, Trustlink is used to manage other digital certificate types such as SMIME, Qualified, authentication, and private trust certificates.

The Trustlink system has checks that enforce technical standards such as the Baseline Requirements; however, QuoVadis has experienced some quality issues when trying to scale TLS validation using processes that are largely manual.

As with other acquisitions, such as the Symantec brands, DigiCert’s intent is to consolidate issuance platforms to its CertCentral platform. QuoVadis will benefit from DigiCert’s significant investment in validation tools, which include guided validation paths, automated checks, and pre-issuance linting. However, we are second in priority queue for consolidation, pending shut-down of all the legacy Symantec systems. DigiCert has told us this will happen in Apr 2020, and they will commence migration of Trustlink shortly after. Planning has already begun for this transfer with the migration paths for TLS being identified.

Like the Symantec migration, the full integration will be done in multiple phases focusing on different customer segments. In its earliest phases, the migration will focus on large enterprise customers or consortia which are not geographically sensitive. In its later stages, DigiCert’s roadmap is to provide a regional version of CertCentral, such that validation data and certain PKI operations can be operated in-region such as in the EU or Switzerland, which is respectful of the needs of customers who are geographically sensitive. This geographic sensitivity is the largest hurdle in migration as DigiCert data is currently stored only in the US. Figuring out the date when that can be accomplished is the primary obstacle to provide a shut-down date for the Trustlink system for TLS.

We have already started consolidating into a single validation team operating under DigiCert’s Dublin based validation group, and are adopting DigiCert’s validation training, standards, and methodologies. Although QuoVadis is still treated separately by DigiCert, we are trying to implement their practices and procedures, including their requirements around incident disclosure and revocation, well before the integration.

As TLS accounts transition from Trustlink to CertCentral, we will check existing organizational details and certificate profiles against DigiCert’s validation platform and revalidate when necessary.

Comment 7

[Stephen Davidson]

We are conducting a review of the Subject DN settings used in QuoVadis EV certificates including businessCategory. We will provide an update on our progress by Dec 15.

Comment 8

[Stephen Davidson]

We request that this bug be closed; the findings of our EV Subject DN review have primarily resulted in JOI corrections which are being updated at https://bugzilla.mozilla.org/show_bug.cgi?id=1589047.
Many thanks!

Comment 9

[Wayne Thayer]

Closing this bug per comment #8 stating that remediation of businessCategory is complete.