1589047 - QuoVadis: Incorrect EV jurisdiction of incorporation information

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Corey Bonnell]

On September 19, 2019, I sent a Certificate Problem Report to QuoVadis about incorrect jurisdiction of incorporation information in https://crt.sh/?id=1443925771&opt=ocsp. The certificate was revoked and the replacement (https://crt.sh/?id=1916867185) was issued on September 24.

I did not see an Incident Report created for this mis-issuance, so I'm creating that here.

Comment 1

[Stephen Davidson]

Requesting that this be combined with https://bugzilla.mozilla.org/show_bug.cgi?id=1581234 which deals with this issue.

Comment 2

[Ryan Sleevi]

Please do not close bugs directly until they've been reviewed.

The statement in Comment #0 of Bug 1581234 stated that, as of 2019-09-13, the issue had been fixed, and as of 2019-09-12, the revocations had been completed. This certificate was detected and revoked after the fact, as it had a relevant subdivision - but that was incorrect for the jurisidction of incorporation.

It also appears to be a different underlying root cause than the bug identified. At minimum, understanding this particular issue, and how it relates to the separate root cause called out in that bug, is, I think, crucial for understanding here. As it stands, the related certificate is not a valid jurisdiction of incorporation, even if it's a valid subdivision, and that seems fundamentally different.

Comment 3

[Jeremy Rowley]

I don't think the jurisdiction issues of Quovadis are completely remediated. For example, I don't think they've scanned their OV certs. Nor have they looked at their registration numbers. I was hoping they'd used the registration number patterns we're working on the scan their database of certs to identify any mis-isssuances.

I'm also interested in hearing what the root cause on this certificate was. Looks like the difference is that the jurisdiction was verified at the state/province level, not the locality level? Does this mean the root cause is bad source information where the jurisdiction is being pulled from?

Comment 4

[Ryan Sleevi]

I agree that it's unclear the root cause.

I will highlight the underlying issue though, so it may help in furthering a root cause analysis, and why I think it's different:

To the best of my knowledge, the source of authorities in Switzerland that can meet the definition of the EVGs are not operated at the locality level, thus the inclusion of jurisdictionLocalityName is not correct.

The serialNumber included ( CHE-200.595.965 ) matches against the Handelsregister of Basel-Stadt canton. However, even then, there's a legitimate question about whether the replacement certificate is correct. In Switzerland, the Unique Business Identification Number (UID) is maintained at the Federal level by the Federal Statistical Office, which is the form used here for the serialNumber.

This is why, for example, GLEIF - amusingly to whom this certificate was issued - recognizes Swiss registration at a jurisdictionCountryName level, rather than per Canton.

Bug 1581234 analyzes the use of different (localized) jurisdiction names, but Basel is a valid locality within Switzerland, and so that does not seem to be related. However, at issue here is the underlying data source used to validate the registration, and whether/how it complies with the EV guidelines. So there's definitely a question of "bad source information", which seems orthogonal to Bug 1581234 and is more in line with the set of issues DigiCert is working through.

Comment 5

[Stephen Davidson]

Historically QuoVadis has operated its own certificate management system (“Trustlink”) and PKI from datacenters located in Bermuda, Switzerland, and the Netherlands. In addition to managing the issuance of TLS certificates, Trustlink is used to manage other digital certificate types such as SMIME, Qualified, authentication, and private trust certificates.

The Trustlink system has checks that enforce technical standards such as the Baseline Requirements; however, QuoVadis has experienced some quality issues when trying to scale TLS validation using processes that are largely manual.

As with other acquisitions, such as the Symantec brands, DigiCert’s intent is to consolidate issuance platforms to its CertCentral platform. QuoVadis will benefit from DigiCert’s significant investment in validation tools, which include guided validation paths, automated checks, and pre-issuance linting. However, we are second in priority queue for consolidation, pending shut-down of all the legacy Symantec systems. DigiCert has told us this will happen in Apr 2020, and they will commence migration of Trustlink shortly after. Planning has already begun for this transfer with the migration paths for TLS being identified.

Like the Symantec migration, the full integration will be done in multiple phases focusing on different customer segments. In its earliest phases, the migration will focus on large enterprise customers or consortia which are not geographically sensitive. In its later stages, DigiCert’s roadmap is to provide a regional version of CertCentral, such that validation data and certain PKI operations can be operated in-region such as in the EU or Switzerland, which is respectful of the needs of customers who are geographically sensitive. This geographic sensitivity is the largest hurdle in migration as DigiCert data is currently stored only in the US. Figuring out the date when that can be accomplished is the primary obstacle to provide a shut-down date for the Trustlink system for TLS.

We have already started consolidating into a single validation team operating under DigiCert’s Dublin based validation group, and are adopting DigiCert’s validation training, standards, and methodologies. Although QuoVadis is still treated separately by DigiCert, we are trying to implement their practices and procedures, including their requirements around incident disclosure and revocation, well before the integration.

As TLS accounts transition from Trustlink to CertCentral, we will check existing organizational details and certificate profiles against DigiCert’s validation platform and revalidate when necessary.

Comment 6

[Ryan Sleevi]

I may be missing things, but I don't see an explanation as to root cause, captured in Comment #4, which was requested in Comment #2.

Comment 7

[Brenda Bernal]

Ryan: The root cause has several aspects:

• The reliance on people to put the correct information in the appropriate fields. In this case, a small number of validation staff in the QuoVadis EU operating centers did not properly interpret what is required for the JOI fields.
• Much of the validation process was manual, concluding in data entry of the JOI fields by validation agents into the certificate management system, with limited technical enforcement in the JOI fields. Due to this gap, the system did not block issuance when there were errors entered in the JOI fields by a small number of validation staff in these offices.
• There was insufficient focus on the outcome of certificate structure such as the JOI fields with the internal audits, which explains a bit of why these were not detected earlier.

Comment 8

[Stephen Davidson]

Hi Ryan: We are conducting a review of the Subject DN settings used in QuoVadis EV certificates including JOI. We will provide an update on our progress by Dec 15.

Comment 9

[Stephen Davidson]

As noted, we are conducting a detailed investigation of our EV TLS. Thus far we have revoked and replaced more than 450 certificates, with the same JOI root cause as the initial certificate identified here. The investigation continues; we will provide an update before the end of the year. If possible, I'd like to provide the crt.sh links at that time.

Comment 10

[Stephen Davidson]

We have now revoked and replaced 2,258 EV certificates, with the same JOI root cause as the initial certificate identified here. Our investigation of EV JOI issues continues.

Comment 11

[Stephen Davidson]

We have now revoked and replaced 5,530 EV certificates, with the same JOI root cause. Our investigation of EV JOI issues continues and we believe we will conclude the work by the end of January; we will post thumbprints of the revoked certificates at the conclusion of the process.

Comment 12

[Ryan Sleevi]

Comment #7 suggests it was a small number of validation staff. The original scope was 450 certificates, which has now expanded to 5,530 certificates (presumably, in cumulative total, and these are not totals-since-last-update).

While I appreciate that QuoVadis is continuing to examine its past issuance, and mitigate the issues, and I'm hopeful that we'll see resolution within the next week, I think one of the things that really bears emphasizing is a hopefully thoughtful analysis about the classes and types of problems faced here that lead to this many misisused EV certificates. This seems like a key systemic failure, even with the volume of EV certificates issued, and it seems essential for the ecosystem to understand what went wrong, where we got lucky, and where things went well.

Comment 13

[Stephen Davidson]

Hello Ryan: I made an error combining spreadsheets in calculating the update in comment 11. The correct count at that time was 5,328. The final numbers will be ~5,500. I will upload the hashes early next week.

The information provided earlier in the bug holds true in explaining the root cause, and in describing the approach we have taken to improve our validation processes as well as to systematically re-check the JOI settings for every EV account in our systems. That related work is the cause for this phased investigation.

The JOI issues were concentrated in several QV vetting locations which, given the nature of our enterprise focus, amplified the effect. For example, based on a nearly complete investigation, 95% of the problem certificates were to entities in only four countries (NL, CH, DE, and US). Just a handful of customer accounts ended up including more than 50% of the problem certificates.

The issues were the improper inclusion of JOI_locality and JOI_province information when EVG section 9.2.4 required only JOI_country, or the inclusion of JOI_locality when only JOI_province and JOI_country were appropriate. In the process of correcting these errors we have also sought to adopt ISO 3166-2 naming conventions in TLS geographic fields.

We also identified a flaw in our previous approach to internal audit, which focused on the process of “how” the information in an EV certificate is verified and approved, with insufficient attention to “what” the final certificate looks like. This has been remedied by building up our inhouse internal audit and analytics team focused on validation procedures and outcomes, as well as error checking on the cert content.

Comment 14

[Stephen Davidson]

Attachment: Bug1589047 Revoked TLS Certificates for JOI

SHA256 for revoked TLS Certificates for JOI Bug 1589047

Comment 15

[Stephen Davidson]

This remediation is complete; we request the bug be closed.

Comment 16

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.