Open Bug 1649938 Opened 1 month ago Updated 2 days ago

QuoVadis: Incorrect OCSP Delegated Responder Certificate

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: ryan.sleevi, Assigned: stephen.davidson)

Details

(Whiteboard: [ca-compliance])

The following was originally reported to m.d.s.p. at https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg13493.html

QuoVadis has issued one or more OCSP Delegated Responders, as defined within RFC 6960, Section 2.6 and Section 4.2.2.2, without including the id-pkix-ocsp-nocheck response, as required by the Baseline Requirements, Version 1, Section 13.2.5 through Version 1.7.0, Section 4.9.9

Example certificate: https://crt.sh/?id=549505562

Please provide an incident report, including the timeline for revocation.

We acknowledge this problem report and are investigating prior to filing a response.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We were alerted to the issue by the MDSP post entitled “SECURITY RELEVANT FOR CAs: The curious case of the Dangerous Delegated Responder Cert”. We followed the discussion since inception and have been working to get an accurate view of what needs to be done for remediation since the topic was first posted.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

1/2 July 2020 – Awareness of Sleevi’s post on MDSP and this bug, emergency incident calls convened with QV and DigiCert to determine scope and extent of the issue.

2 July 2020 – Tim Hollebeek posts DigiCert + QuoVadis opinion on the issue.

3/4/5 July 2020 – Identification of affected sub CAs, setting of revocation priorities, discussions with DigiCert, PKIoverheid, affected external subCA customers, as well as regulatory supervisory authorities. Planning for replacement CAs. Arranged key ceremonies and starting naming documents to get key ceremonies prepared and executed.

3 July 2020 – Revocation of initial 4 sub CAs: QuoVadis QVRCA1G1 SSL ICA, QuoVadis QVRCA1G3 SSL ICA, QuoVadis QVRCA3G1 SSL ICA, QuoVadis QVRCA3G3 SSL ICA.

6/7/8 July 2020 – Key ceremonies for creation of replacement CAs. Details are provided in subsequent comment 3 and comment 4. Contacted customers explaining the situation.

8 July 2020 -- Revocation of 3 external sub CAs: DarkMatter Assured CA, DarkMatter High Assurance CA, DarkMatter Secure CA.

  1. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

QuoVadis ceased using the id-kp-OCSPSigning EKU in its new ICAs in 2019.

In addition, the CAs listed as “Reissued” no longer have the id-kp-OCSPSigning EKU and will move into production by COB 8 July 2020. See comment 3 and notes. We recognise that we need to destroy the key material of the affected CAs and are working towards that goal.

  1. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

For ease of tracking and discussion, QuoVadis split the affected CAs as follows.

The earliest certificate was issued on 13 January 2015 and the latest on 19 May 2019.
We are prioritizing the replacement and destruction of the External Sub CAs as they operate outside our infrastructure. However, we are working towards replacement and destruction of all the affected CAs, including those operated by QuoVadis. The general plan is to:

  1. Immediately revoke CAs with low customer impact (completed).
  2. Reissue all QV-operated CAs without the id-kp-OCSPSigning EKU (completed by end 8 July).
  3. Create new CAs with new keys.
  4. Revoke and/or destroy keys for the “legacy” affected CAs.

In comment 3 and comment 4 CAs marked “Reissued” use the same key as the previous cert; these will later be decommissioned and destroyed the same as the impacted CAs. CAs marked “Revoked” are revoked but do not yet have keys destroyed.

We don’t necessarily need to distribute the “Reissued” CAs in all cases; they were created where necessary to replace pinned certs or certs on hardware devices that cannot rapidly be replaced with other CAs. A couple issues causing delays are:

  • The CAs are non-TLS and involve large numbers of end entity certificates on tokens which require coordination with customers, regulatory agencies, or the EUTL.
  • We are using this an opportunity to migrate from the QuoVadis-legacy infrastructure to DigiCert systems, which is anticipated to improve performance in other areas such as approval workflow and validation process. We think this is better than providing a replacement on the legacy infrastructure, and are basing revocation dates on this process.

We believe some CAs may require a period up to six months before key destruction but we are going to be terminating CAs at regular stages throughout the period. We will update this bug regularly to provide accurate dates.

  1. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See comment 3 and comment 4.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    At the time the CAs were built, the QuoVadis understanding was that EKU chaining was required including for id-kp-OCSPSigning. We were not aware at the time of potential issues with the use of the EKU.

QuoVadis recognises the issue and is replacing the certificates. QuoVadis will not create new ICAs with the id-kp-OCSPSigning EKU going forward.

  1. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We are revoking and replacing all QV CAs that include the id-kp-OCSPSigning EKU and performing key destruction. Details are provided in subsequent comment 3 and comment 4

We will regularly update this bug with details as we progress.

We have opened Bug 1651553 acknowledging that not all affected certificates were revoked within 7 days.

QV-operated CAs

Name Type Status Link to original cert
QuoVadis Europe Advanced CA G1 nonTLS Reissued https://crt.sh/?q=8E5CEA1D1012521C27A4E11270702BC7C9A1156F135073D6CA4E42EEE826249E
QuoVadis Europe SSL CA G1 TLS Reissued https://crt.sh/?q=DC8B2DEE50DD478AB135CAC269CEA68851557A9129ABCD98DF5213B23DBFB3CD
QuoVadis EV SSL ICA G1 TLS Reissued https://crt.sh/?q=3FE8BE392A08684B99F497E618C7DDF5A02A4289BF9D08E595045931BFBA814F
QuoVadis EV SSL ICA G3 TLS Reissued https://crt.sh/?q=F18442BEDF70B4D15211356C72B659332BED03FFD3BBA7AFAAABE6DE9D723002
QuoVadis Qualified Web ICA G1 TLS Reissued/Q https://crt.sh/?q=04ECBA8F92BFF7458C4A5E7C69261FC7E2EF52D5AF54FBDD92B17141BBE0651F
QuoVadis Qualified Web ICA G1 TLS Reissued/Q https://crt.sh/?q=C02D8A30ED69B2F864ED8FB1A63A3E7255288920CA294BDCA30F63898FB9195C
QuoVadis QVRCA1G1 SSL ICA TLS Revoked https://crt.sh/?q=9058D5065F8F3E9A63AAFE5CE89A764470B0DEF9DCF3B9EC7D0587FAB88FEBA0
QuoVadis QVRCA1G3 SSL ICA TLS Revoked https://crt.sh/?q=51587C867BF66F35ECE554A08E0A41C13B8BBD9B59D262D204A70309A672BEE6
QuoVadis QVRCA3G1 SSL ICA TLS Revoked https://crt.sh/?q=8D99217FF82D60E4DF59BE8A1121625CDFFC4F22F21E1263A1DF06D2DC0B540E
QuoVadis QVRCA3G3 SSL ICA TLS Revoked https://crt.sh/?q=764A0D84D5552CD5872C73464F37F02175CDA70588102B13ADA2A0199FC403E9
QuoVadis Swiss Advanced CA G3 nonTLS Reissued https://crt.sh/?q=31A7C579F24D5562CDB203FA15A9F2C3FF5DC1F2E6BF7C0BD95FBCB0114C8D21
QuoVadis Swiss Regulated CA G1 nonTLS Reissued https://crt.sh/?q=53570D05E40DC62F9DF91D15AFB015B56AAA680C20CC168E6D57D946C9CE2684
QuoVadis Swiss Regulated CA G2 nonTLS Reissued https://crt.sh/?q=9390714B1D909FA3D70DDC7681B38F07ED4E6356CB5C71915D1BDCD48FE335F8
HydrantID EV SSL ICA G1 TLS Reissued https://crt.sh/?q=80FDE428212AF0CA0AC531EEE6ED2DF3D3C2A4557DFCE857070FC947922E9B24
itsme Sign Issuing CA G1 nonTLS Reissued/Q https://crt.sh/?q=F640E5643C40C1F329E100438E28C957691AFA8A53E405A326F7AFEB70C23BC1
VR IDENT EV SSL CA 2018 TLS Reissued https://crt.sh/?q=BF39A4241F42D522368944B3DC53ED9EAA5AC7735E242E0627C0DD5BBA714484
VR IDENT SSL CA 2018 TLS Reissued https://crt.sh/?q=502C7A870341D7BE67DB26DBFACF9647AFD89A854BF8812FA4EF70C356EC13E5

Replacement of the CAs marked “Reissued/Q” will occur after the new Issuing CA is distributed on the relevant country Trusted List (EUTL). The process has already been started.
Note: Our planning is still in flight for the revocation and key destruction of the QV-operated CAs. Our plans at this stage are:
By Aug 31

  • QuoVadis Swiss Regulated CA G2 (nonTLS)
  • QuoVadis Qualified Web ICA G1 (TLS)
  • VR IDENT EV SSL CA 2018 (TLS)

By Sept 30

  • QuoVadis Europe SSL CA G1 (TLS)
  • QuoVadis EV SSL ICA G1 (TLS)

By Oct 31

  • itsme Sign Issuing CA G1 (nonTLS)
  • VR IDENT SSL CA 2018 (TLS)
  • QuoVadis Swiss Regulated CA G1 (nonTLS)

By Dec 31

  • HydrantID EV SSL ICA G1 (TLS)
  • QuoVadis Swiss Advanced CA G3 (nonTLS)
  • QuoVadis Europe Advanced CA G1 (nonTLS)
  • QuoVadis EV SSL ICA G3 (TLS)

External Sub CAs

Name Type Status Link to original cert
DarkMatter Assured CA nonTLS Revoked https://crt.sh/?q=D8888F4A84F74C974DFFB573A1BF5BBBACD1713B905096F8EB015062BF396C4D
DarkMatter High Assurance CA TLS Revoked https://crt.sh/?q=3AE699D94E8FEBDACB86D4F90D40903333478E65E0655C432451197E33FA07F2
DarkMatter Secure CA TLS Revoked https://crt.sh/?q=A25A19546819D048000EF9C6577C4BCD8D2155B1E4346A4599D6C8B79799D4A1
LLB Root CA public v3 nonTLS New CA created https://crt.sh/?q=FD4C2B993E0356E90D4F9FBD2361DEF1A498378CEECEB92DD76FE0AD7E2C7B16
LLB Root CA public v4 nonTLS New CA created https://crt.sh/?q=3A59C1A60A69FE57A98DAB376736275AF0913FD1F65B92B71B7E9A7A1EB440FE
Siemens Issuing CA EE Auth 2016 nonTLS Reissued https://crt.sh/?q=5972B9BD471017D5F32705BEE915703626575F8B270A39C1C312C7F9246C10D4
Siemens Issuing CA EE Auth 2016 nonTLS Reissued https://crt.sh/?q=940D2F212A2A39CC84BD42D0F6DC4F7BA4C477E7A5A9922C96B9F5EC14E4A6C8
Siemens Issuing CA EE Enc 2016 nonTLS Reissued https://crt.sh/?q=ABF3803CD2939E26803E52280A81F67C46C3E0EE75FCDBB1E30FB03A321ACFAD
Siemens Issuing CA EE Network Smartcard Auth 2016 nonTLS Reissued https://crt.sh/?q=37D220A6C7522799021191349C183F917BE1BE8626CF926B0FD6E0A8681EE031
Siemens Issuing CA Internet Code Signing 2016 nonTLS Reissued https://crt.sh/?q=1B046535378E07D10ACDAA24EEFCE20420BB9A596114EB475CA696357753E925
Siemens Issuing CA Internet Server 2017 TLS Reissued https://crt.sh/?q=7D33AE618CD62553377D253D2EBCA285D84E98A924D89F98D4BE4FEE31F92AA8
Siemens Issuing CA Medium Strength Authentication 2016 nonTLS Reissued https://crt.sh/?q=42AB4D9F1809454EBEC245D8DB06FF61AA8289B05A263DFEE9662DAC91666043
Siemens Issuing CA Multi Purpose 2016 nonTLS Reissued https://crt.sh/?q=05BFB6605D48516A571BAF9A7FF75376130470DA5EE7FF684C2672EAA0C0C8AD

Notes:

  • DarkMatter have engaged KPMG (WebTrust auditors) to witness key destruction by July 31.
  • Technically constrained LLB expected to be revoked with key destruction by July 18 (witness subject to lockdown protocols). Rather than reissue, a new CA with a new key was created.
  • Siemens:
    • Siemens Issuing CA Internet Server 2017 to be revoked with key destruction by October 31
    • Remaining Siemens nonTLS CAs have issued large numbers of certificate on tokens to employees and business counterparties. Siemens are investigating timelines for revocation and reissuance of these end entity certificates. These CAs have ceased issuance. We will update by July 15 with a plan for revocation and key destruction.

QuoVadis continues to advance on its plans as described in Comments 3 and 4.

QV-operated CAs

Name Type Last Status Update
QuoVadis Europe Advanced CA G1 nonTLS Reissued New CA created: QuoVadis Europe Advanced CA G2
QuoVadis Europe SSL CA G1 TLS Reissued New CA created: QuoVadis Europe SSL CA G2
QuoVadis EV SSL ICA G1 TLS Reissued Transition planning underway, see Comment 3
QuoVadis EV SSL ICA G3 TLS Reissued Transition planning underway, see Comment 3
QuoVadis Qualified Web ICA G1 TLS Reissued/Q New CA created: QuoVadis Qualified Web CA G2
QuoVadis Qualified Web ICA G1 TLS Reissued/Q Revoked
QuoVadis QVRCA1G1 SSL ICA TLS Revoked NA
QuoVadis QVRCA1G3 SSL ICA TLS Revoked NA
QuoVadis QVRCA3G1 SSL ICA TLS Revoked NA
QuoVadis QVRCA3G3 SSL ICA TLS Revoked NA
QuoVadis Swiss Advanced CA G3 nonTLS Reissued New CA created: QuoVadis Swiss Advanced CA G4
QuoVadis Swiss Regulated CA G1 nonTLS Reissued New CA to be created week of 7/27: QuoVadis Swiss Regulated CA G3
QuoVadis Swiss Regulated CA G2 nonTLS Reissued New CA to be created week of 7/27: QuoVadis Swiss Regulated CA G3
HydrantID EV SSL ICA G1 TLS Reissued New CA created: HydrantID EV SSL CA G2
itsme Sign Issuing CA G1 nonTLS Reissued/Q Transition planning underway, see Comment 3
VR IDENT EV SSL CA 2018 TLS Reissued New CA created: VR IDENT EV SSL CA 2020
VR IDENT SSL CA 2018 TLS Reissued New CA created: VR IDENT SSL CA 2020

External Sub CAs

  • DarkMatter CAs, revoked, key destruction planned for completion by Aug 7
  • LLB CAs, revoked, key destruction planned for completion by July 29
  • Siemens CAs, transition planning underway, will provide update by July 31

For the QuoVadis-operated issuing CAs and Siemens TLS CAs we continue on the schedule outlined in Comment 3. It is targeted that the ~500,000 end entity certificates will be replaced and the 17 issuing CAs revoked/keys destroyed by Dec 31.

For the external issuing CAs, key destruction has occurred for DarkMatter and LLB. The DarkMatter CAs are now "oneCRL" equivalent in Mozilla, Microsoft, and Google. Apple will similarly remove trust from the DarkMatter CAs. An update on final plan is pending for the nonTLS Siemens CAs, which involve ~850,000 end entity certificates, the majority of which are on tokens.

You need to log in before you can comment on or make changes to this bug.