1493760 - QuoVadis: improper countryName format

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Stephen Davidson]

On September 20-21, QuoVadis issued 4 certificates with “Germany” instead of “DE” in the countryName and jurisdictionCountryName.  These certificates have been replaced and will be revoked.

https://crt.sh/?id=774083119&opt=cablint
https://crt.sh/?id=770045338&opt=cablint 
https://crt.sh/?id=770016929&opt=cablint 
https://crt.sh/?id=769978075&opt=cablint

In the past, QuoVadis root signed external subCAs operated by Fiducia & GAD IT AG (VR IDENT EV SSL CA 2016 and VR IDENT SSL CA 2016).  In order to improve controls, these were replaced on Sept 20 with a managed PKI using dedicated issuing CAs (VR IDENT EV SSL CA 2018 and VR IDENT SSL CA 2018).

As part of the transition, the previously validated Organisation name details were bulk imported into our Managed PKI interface.  It appears that certain Organisations in that file included the text country name rather than ISO code.  Although filters exist in the Managed PKI interface to catch issues such as this, in this case they were not effective, and the issue was not visible in our Managed PKI interface.  Similarly, the CA software processed the request.

The Managed PKI interface overwrites Subject information in CSRs with the approved templates, which in this case, was incorrect.  Unfortunately Organisations used in testing were not affected by the issue.
The certificates were identified on Sept 21 by QuoVadis via post issuance linting, and the CA was immediately taken offline until the issue could be identified and remediated.  

The bulk import of Organisation details used in this case is an uncommon and manual operation.  The issue was also discussed with the underlying CA software vendor, which assisted QuoVadis to make improvements to the process and the filters in its Managed PKI interface, which are now in place and the CA returned to normal operation.

Comment 1

[Wayne Thayer]

Thank you for reporting this incident Stephen.  In order to be useful as a learning device, the incident report needs more detail. For instance:

* Am I correct to understand this as a problem with the certificate profiles?
* How did you detect the problem?
* Can you explain what the processes are that should have prevented this issue, and how they failed?
* Why was this not found in testing prior to moving the change to production?
* Is pre-issuance linting in place? If so, why didn't it catch this? If not, why not?
* What are the improvements that were made to Managed PKI interface? What other improvements have been made to processes, testing, etc. to prevent this sort of failure in the future?

Comment 2

[Stephen Davidson]

> * Am I correct to understand this as a problem with the certificate profiles?
No.  Our Managed PKI imposes checks for a variety of BR and other standards requirements.  Typically Subject DN information in submitted CSRs is overwritten by validated Org templates held in the Managed PKI. As part of the transition to a new CA platform, the information populating these templates was imported via a seldom used manual process prior to re-validation.  Certain records in the file showed the countryName and jurisdictionCountryName as “Germany” instead of “GERMANY” and were not converted to the required “DE” format in the underlying template.  Our usual input and upload methods would have corrected the case sensitivity.

> * How did you detect the problem?
Post issuance linting using Zlint.

> * Can you explain what the processes are that should have prevented this issue, and how they failed?
The data in the input file was accurate – but it was misunderstood that the manual import method used in this situation evaded case sensitivity checks in the Managed PKI.  Our “normal” input and bulk upload methods would have caught it.

> * Why was this not found in testing prior to moving the change to production?
The Orgs used in our testing were not affected by the case sensitivity issue (which was also not visible in the reports of Org details within our Managed PKI admin interface).

> * Is pre-issuance linting in place? If so, why didn't it catch this? If not, why not?
Pre-linting will require significant changes to our Managed PKI and CA implementation.  We are in discussions with our CA software provider regarding ways to accelerate our adoption of pre-linting.  At this time, we cannot provide a date.  As a stopgap, we are tightening our use of post-issuance linting.

> * What are the improvements that were made to Managed PKI interface? What other improvements have been made to processes, testing, etc. to prevent this sort of failure in the future?
The enforcement mechanisms of our Managed PKI are obviously an ongoing focus for us, and we are generally satisfied with their effectiveness. 
A code review is being made for the case sensitivity issue behind this miss-issuance.  
This type of large manual import – which is very rare, and only occurred as part of a transition from a root signing to a managed CA – will undergo closer inspection (more eyes) for formatting before import.

Comment 3

[Stephen Davidson]

Confirming that the four certificates were revoked.