- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
We were alerted to the issue by the MDSP post entitled “SECURITY RELEVANT FOR CAs: The curious case of the Dangerous Delegated Responder Cert”. We followed the discussion since inception and have been working to get an accurate view of what needs to be done for remediation since the topic was first posted.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
1/2 July 2020 – Awareness of Sleevi’s post on MDSP and this bug, emergency incident calls convened with QV and DigiCert to determine scope and extent of the issue.
2 July 2020 – Tim Hollebeek posts DigiCert + QuoVadis opinion on the issue.
3/4/5 July 2020 – Identification of affected sub CAs, setting of revocation priorities, discussions with DigiCert, PKIoverheid, affected external subCA customers, as well as regulatory supervisory authorities. Planning for replacement CAs. Arranged key ceremonies and starting naming documents to get key ceremonies prepared and executed.
3 July 2020 – Revocation of initial 4 sub CAs: QuoVadis QVRCA1G1 SSL ICA, QuoVadis QVRCA1G3 SSL ICA, QuoVadis QVRCA3G1 SSL ICA, QuoVadis QVRCA3G3 SSL ICA.
6/7/8 July 2020 – Key ceremonies for creation of replacement CAs. Details are provided in subsequent comment 3 and comment 4. Contacted customers explaining the situation.
8 July 2020 -- Revocation of 3 external sub CAs: DarkMatter Assured CA, DarkMatter High Assurance CA, DarkMatter Secure CA.
- Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
QuoVadis ceased using the id-kp-OCSPSigning EKU in its new ICAs in 2019.
In addition, the CAs listed as “Reissued” no longer have the id-kp-OCSPSigning EKU and will move into production by COB 8 July 2020. See comment 3 and notes. We recognise that we need to destroy the key material of the affected CAs and are working towards that goal.
- In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
For ease of tracking and discussion, QuoVadis split the affected CAs as follows.
The earliest certificate was issued on 13 January 2015 and the latest on 19 May 2019.
We are prioritizing the replacement and destruction of the External Sub CAs as they operate outside our infrastructure. However, we are working towards replacement and destruction of all the affected CAs, including those operated by QuoVadis. The general plan is to:
- Immediately revoke CAs with low customer impact (completed).
- Reissue all QV-operated CAs without the id-kp-OCSPSigning EKU (completed by end 8 July).
- Create new CAs with new keys.
- Revoke and/or destroy keys for the “legacy” affected CAs.
In comment 3 and comment 4 CAs marked “Reissued” use the same key as the previous cert; these will later be decommissioned and destroyed the same as the impacted CAs. CAs marked “Revoked” are revoked but do not yet have keys destroyed.
We don’t necessarily need to distribute the “Reissued” CAs in all cases; they were created where necessary to replace pinned certs or certs on hardware devices that cannot rapidly be replaced with other CAs. A couple issues causing delays are:
- The CAs are non-TLS and involve large numbers of end entity certificates on tokens which require coordination with customers, regulatory agencies, or the EUTL.
- We are using this an opportunity to migrate from the QuoVadis-legacy infrastructure to DigiCert systems, which is anticipated to improve performance in other areas such as approval workflow and validation process. We think this is better than providing a replacement on the legacy infrastructure, and are basing revocation dates on this process.
We believe some CAs may require a period up to six months before key destruction but we are going to be terminating CAs at regular stages throughout the period. We will update this bug regularly to provide accurate dates.
- In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
See comment 3 and comment 4.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
At the time the CAs were built, the QuoVadis understanding was that EKU chaining was required including for id-kp-OCSPSigning. We were not aware at the time of potential issues with the use of the EKU.
QuoVadis recognises the issue and is replacing the certificates. QuoVadis will not create new ICAs with the id-kp-OCSPSigning EKU going forward.
- List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
We are revoking and replacing all QV CAs that include the id-kp-OCSPSigning EKU and performing key destruction. Details are provided in subsequent comment 3 and comment 4
We will regularly update this bug with details as we progress.
We have opened Bug 1651553 acknowledging that not all affected certificates were revoked within 7 days.