1581234 - QuoVadis: EV JOI Issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Stephen Davidson]

Attachment: EV JOI revocations batch 1, crt.sh by serial

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We are aware the “DigiCert JOI issue” described in https://bugzilla.mozilla.org/show_bug.cgi?id=1576013.

Although we await the results of DigiCert’s own investigations into QuoVadis’ issuance using their tools, QuoVadis proactively commenced our own reviews specifically related to the EV jurisdictionStateOrProvinceName. We have identified 413 certificates in an initial batch.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

29 Aug QV becomes involved in DigiCert JOI effort
30 Aug QV begins preemptive investigation
3 Sept QV clean up of templates commences
7 Sept Batch 1 customers informed
12 Sept Batch 1 revocations

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

QuoVadis has clarified our validation procedures to rely upon only ISO 3166-2 and other Government provided references for subdivision entries.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Certificates included abbreviations in the jurisdictionStateOrProvinceName. Although impacting a small subset of pre-approved templates in our certificate management system, some abbreviations were in place up to two years.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See attachment “EV JOI revocations batch 1". In addition to our own analysis, we now have received DigiCert’s scan of our issuance using their tools. We are sorting that into batches by issue type for review; as such additional certificates may be added to this disclosure at a later date. We are posting this interim report to demonstrate our positive steps towards remaining compliant. We’ll provide an update on or before 20 Sept.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our standards for geographic names included “local usage” such as Zuerich for Zürich, St. Gallen for Sankt Gallen, etc. Due to the preference to use ISO 3166-2 “code elements” rather than full names in some countries for subdivisions, such as SG for St. Gallen, their use crept into a small subset of our pre-approved templates.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

In some cases, Government departments may use different geographic names than those on the official list of subdivisions for their country, particularly where administrative boundaries may have changed.

We have made a multiple reviewer manual review of jurisdictionStateOrProvinceName and StateOrProvinceName input used in our pre-approved templates using ISO 3166-2 and other Government official subdivision references making corrections and consistency improvements. These references will be used by our RAs creating new templates going forward. This is still a manual process; however QuoVadis operates primarily as an enterprise rather than retail issuer. QuoVadis will eventually integrate platforms and tools from DigiCert; we cannot provide dates at this time.

Comment 1

[Stephen Davidson]

12 Sept Investigation completed on use of abbreviations in EV JOI; customer informed
17 Sept Certificates revoked, as follows:

https://crt.sh/?serial=331badcbdd8e4dab3e89ccad2657c8ebd409a0d5
https://crt.sh/?serial=626b5af3284e10c7097f5166d324e41c138f4695
https://crt.sh/?serial=63840228a638376dfc00dd8ee0e1aa6cc58f697a

This concludes the QV revocations for the specific JOI abbreviation issue.

Comment 2

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.

Comment 3

[Ryan Sleevi]

Wayne: That was pretty quick closure :)

I’m reopening, because I’m concerned at the lack of detail in the response, especially when compared to other CA’s responses as to similar issues. This response mostly seems like “we require an extra person to sign off”, which doesn’t give any confidence of any systemic remediation.

Stephen: It would be helpful if you could more fully describe the process and decision making logic here. Obvious questions include how you determine whether or not a geographic name is acceptable and how you ensure consistency. A meaningful answer would be to more fully describe your validation processes, as well as what controls, if any, you have to check that they’re being followed. Right now, the level of detail provided leaves it entirely up to imagination; for example, what’s to prevent the multi-party control being “I’ll buy you lunch if you approve this”, which meets the definition of a multi-party control, but clearly not the spirit or intent here.

I encourage a review of other incident reports in the past quarter regarding geographic information, the controls being put in place, and the relevant override process, as well as the overall process for ensuring consistency and public transparency.

Comment 4

[Stephen Davidson]

Hi Ryan: We use a master list of incorporation agencies (recently provided by DigiCert publicly) to verify JOI fields. There is no deviation from this list without approval by the Compliance team (ie not Validation).
The origin of this disclosure was inconsistent naming in the JOI fields. As explained, we have a master list of allowed names (at this stage for jurisdictionStateOrProvinceName, although we will expand that to include jurisdictionLocality). Deviations from this list must be approved by the Compliance team.
The caveat is that these lists are manual - there is no system enforcement of names at this time when an RA sets up JOI fields in "subject templates". As noted, we are focussing additional effort on internal audit of those "subject templates" in addition to the certificate audits.

Comment 5

[Ryan Sleevi]

Thanks. This is slightly more reassuring, but it'd be useful to understand what the process for approval is, how it's reviewed, whether or not QuoVadis publicizes those deviations, etc.

I understand that deviations must be approved, but it's not clear the process in place to minimize those deviations. Similarly, it seems that those deviations create consistency issues for Relying Parties that wish to use this field, since they have to handle any deviations QuoVadis allows. Are there any steps being taken to reduce or eliminate them? If not, what steps are being taken to ensure both consistency and transparency?

Comment 6

[Stephen Davidson]

Hi Ryan: There are two aspects here - consistency in use of name forms versus deviations from the list.
As described above, we have improved the reference for our allowed geographic names. Deviations from that list are generally noncontroversial as they'd be based on Government sources defining their subdivisions. Simply, we are trying to make sure deviations from the current list are documented and join the list, rather than be an RA's judgement call.
I assume you are more interested in deviations from the list of allowed information sources, where more research judgement may be required to assess their suitability. This list is now pretty well defined for our existing business, which is to say that new entries are typically for a newly encountered jurisdiction and/or entity type for our customer base. As you have seen above and in other disclosures there is increased focus on Compliance - rather than Validation - approving such changes. We also benefit from DigiCert's larger experience in this area, and they have now made a version of their JOI reference public.
We also have interest from customers in LEI, and feel there is commonality of interests between LEI and EV in clearly identifying entities in different jurisdictions around the world. While I agree that we need to get the naming consistent - but the relying party desire is to make sure they can identify the right company.

Comment 7

[Wayne Thayer]

Ryan: any further questions?

Comment 8

[Ryan Sleevi]

I personally don't feel that I have a better understanding of QuoVadis' operations from when I asked in Comment #3. The answers in Comment #4 reveal it's still a "Trust the Validation Agent" process, and the answers in Comment #6 is "Well, we don't really have rules, because we think we've covered everything we have, but we're thinking about making rules in the future.". If these seem like uncharitable summaries, it's a reflection of a lack of understanding and clarity about the process and approvals.

The level of detail in this Incident Report does not, in my mind, help avoid or prevent similar issues, at QuoVadis or in the CA ecosystem. So I consider this a very disappointing response, but I also suspect we're unlikely to get further information that might help improve. I think that, when weighing the total of incidents of QuoVadis, we should consider this incident concluded, but I would not say it has a positive resolution.

Comment 10

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.