Open Bug 1581597 Opened 3 months ago Updated 13 days ago

QuoVadis: Unconstrained CAs missing audits

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: s.davidson, Assigned: s.davidson)

Details

(Whiteboard: [ca-compliance])

Attachments

(4 files)

15.27 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
9.93 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
87.35 KB, application/pdf
Details
64.22 KB, application/pdf
Details
  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 10-September, we received an e-mail from the Chrome Root Authority program with the subject “Baseline Requirements Audit Status” requesting confirmation of Baseline Requirements disclosures.

We found 18 ICA certificates that although technically capable of TLS issuance (due to lack of EKUs) are not actually capable of TLS issuance as they are not provided TLS certificate profiles nor TLS workflows in our certificate management system. These ICAs do not issue TLS certificates. These ICAs are older, before QuoVadis commenced using explicit EKU in line with changing industry expectations. These ICAs are listed in the table below.

All these ICAs have always been disclosed in the annual WebTrust for CAs audit, which is conducted concurrently with the WebTrust audits for BRs and EVG. The complete population of QuoVadis issued certificates in the audit period is provided to the WebTrust auditors.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
  • 10-September-2019: received an e-mail from the Chrome Root Authority program
  • 10-September-2019: Performed investigation on EKUs and WebTrust CA scope
  • 13-September-2019: We met with the WebTrust auditors to finalize the plan to amend the WTBR report for the 1-January-2018 to 31-December-2018 audit period.
  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

N/A - no TLS server certificates have been issued from these CAs.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

N/A - no TLS server certificates have been issued from these CAs.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

QuoVadis did not include these unconstrained CAs in our most recent WTBR report. These CAs were however included in the WebTrust Principles and Criteria for Certification Authorities (WTCA) report.

Three of these impacted ICAs are also included in the scope of QuoVadis ETSI audits and explicitly named in the ETSI EN 319 411-1 / ETSI EN 319 411-2 certificates. The Baseline Requirements are referenced in “Audit Criteria” section. Since these 3 ICAs don’t issue TLS, the policies that are relevant to these CAs include NCP, NCP+, QCP-n, QCP-n-qscd, QCP-l, QCP-l-qscd. These ETSI reports are provided to our European Supervisory bodies and are posted on our website, but have not previously been uploaded to CCADB. For the remainder of the ICAs, there was an oversight of the intent of the Mozilla policy for covering them in a BR audit even though these ICAs were purpose built for other than TLS issuance.

https://www.quovadisglobal.com/~/media/Files/Files_Global/QuoVadis_ETS_030.ashx
https://www.quovadisglobal.com/~/media/Files/Files_Global/QuoVadis_ETS_010.ashx

No TLS certificates were issued from these CAs.

Common Name crt.sh Note
HIN Health Info Net CA https://crt.sh/?q=479F4E101F380691201A34CBADBC09E5F0523B35EE3839EB4B14332481EF8463
HydrantID Client ICA https://crt.sh/?q=2EEE91CC892A16CCB7320CDED2AE4948C052345D6B24E214C24EB93932D10DD9
QuoVadis Belgium Issuing CA G1 https://crt.sh/?q=27EBACD86DD3BF86143DA4342861031A57CF3FA414D40A86E669C3F4F1D8CF24
QuoVadis Belgium Issuing CA G2 https://crt.sh/?q=D90B40132306D1094608B1B9A2F6A9E23B45FE121FEF514A1C9DF70A815AD95C Disclosed in ETSI certificate
QuoVadis ElDI-V CA G1 https://crt.sh/?q=393E95D3AE5233A04FEFE058BA8F445132D30E4362D5F7259061392716B34D2C
QuoVadis EU Issuing Certification Authority https://crt.sh/?q=EC50E7E17D3802811C8B6567148CED68BBB1BD79EDDC61DBD298CEA5BA0FB862 Revoked
QuoVadis EU Issuing Certification Authority G2 https://crt.sh/?q=EC3F940A48EF7CBCEA4142F735A5DF2976DB38183D9033C76B78E25F8F53EB5B
QuoVadis EU Issuing Certification Authority G3 https://crt.sh/?q=ADDFFA6FD0809A54A9F0B31FD25F74BF7F2D7AE11C80FD99DAA0FB603A65CD0E Disclosed in ETSI certificate
QuoVadis EU Issuing Certification Authority G4 https://crt.sh/?q=0DD818228990D83FCE9F9DCA7B5CC44ED318EDD16399987EA893877EA52DE11E Disclosed in ETSI certificate
QuoVadis Internal CA G1 https://crt.sh/?q=6B8973A0DBADA29988C5DC06CBCEF049BE770604F8A7436D817FAC3A9710F481
QuoVadis Issuing CA 3 G3 https://crt.sh/?q=C12DD0347C0D4AA25D3986E0499740C5363A6B7EC32A49C5D18B9D56B075E368 Revoked
QuoVadis Issuing CA G3 https://crt.sh/?q=15CE38976716DCB35AA7B35FC168EBBB3BC2EC4696A8C795FC5C48457140E0A7
QuoVadis Issuing CA G4 https://crt.sh/?q=DA3BC81005FDBB853D681A7E942661AEBA23789211525EAF52221F28514C09CB
QuoVadis Personal Signing Service CA G1 https://crt.sh/?q=F94C931373F850FF3D7DB5FB20AC04EC2F812CAEC9BD17E32DB6DCAE2269104D
QuoVadis SuisseID Advanced CA https://crt.sh/?q=5DDAB0A802D83893AC0EDF9B30A620411B1A74A8B7D411A6A7AD7DC46EB1C8C8
QuoVadis SuisseID Qualified CA https://crt.sh/?q=5B7017B80F97C621AF1163B04BEBAFD2F932A42B85F4B9FEC71B38609F564922
QuoVadis Swiss Advanced CA https://crt.sh/?q=235C96A2E2DA557B904E90F3A0CAA57EABB4BDB5F401969DA8C282F60839568F
QuoVadis Swiss Advanced CA G2 https://crt.sh/?q=5044F65E1042CD380B0B9997E4283358F0DEEF7873DA72EFDB6F02474AE37EBE
  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

A decision was made to only list ICAs in the WebTrust for BR assertion that actually issued TLS. As such the 18 ICAs – which were confirmed not to issue TLS in the audit – were not listed in the WebTrust for BR’s report. That decision, in retrospect, was incorrect. In part, that decision was based on legacy ambiguity in the requirements to call out unconstrained ICAs in the WebTrust for BR’s report.

Many of these ICAs are old legacy ICAs that were created in the period 2009 to 2014.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
  • We are working with our WebTrust assessors to amend our most recent WTBR audit to explicitly disclose these impacted ICAs. The audit work has been completed and EY are in the reporting stages. We will update this disclosure when that is complete.
  • We are in the process of performing a review of all QuoVadis ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with https://bugzilla.mozilla.org/show_bug.cgi?id=1563573

We request that the ICAs in the attachment "QuoVadis_OneCRL_16Sept2019" be added to OneCRL.

Assignee: wthayer → s.davidson
Flags: needinfo?(kwilson)
Summary: QuoVadis: Unconstrained CAs → QuoVadis: Unconstrained CAs missing audits
Whiteboard: [ca-compliance]
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Stephen, Please check the attached spreadsheet to make sure it contains the correct items to add to OneCRL.

Flags: needinfo?(kwilson) → needinfo?(s.davidson)

Confirmed: the spreadsheet contains the correct items to add to OneCRL.
Thank you.

Flags: needinfo?(s.davidson)

(In reply to Stephen Davidson from comment #2)

Confirmed: the spreadsheet contains the correct items to add to OneCRL.

Those records in the CCADB are set as "Ready to Add" to OneCRL, so they will be added as part of a future batch of updates to OneCRL.

I think the remaining item for this bug is status on:

We are in the process of performing a review of all QuoVadis ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with https://bugzilla.mozilla.org/show_bug.cgi?id=1563573

We performed a review of all QuoVadis ICAs and reconciled them to the following 2018 WebTrust reports:
• WebTrust for CAs
• WebTrust for Baseline Requirements
• WebTrust for EV

Based on this reconciliation, there were no further additions to the 2018 WebTrust for Baseline Requirements report needed other than the 18 CAs already discussed above. EY are in the process of re-issuing the 2018 WebTrust for Baseline Requirements report. Target date for updated report is Friday 27th September.

This reconciliation exercise found one EV ICA that was missing from disclosure in the 2018 WebTrust for EV report (VR IDENT EV SSL CA 2018). This ICA was included in the scope of the EY EV testing and was disclosed in the 2018 WebTrust for CAs and 2018 WebTrust for Baseline Requirements reports but omitted in error from the 2018 WebTrust for EV report. EY are in the process of re-issuing the WebTrust for EV report to correct this. Target date for updated report is Friday 27th September.

QuoVadis will work closely with our auditors going forward to help ensure that CAs are disclosed appropriately in future audits.

Barry: Is the updated audit report available?

Flags: needinfo?(karoshi-)

EY reissued the 2018 WebTrust for Baseline Requirements report for QuoVadis, effective 13 Sept 2019, naming the 18 CAs in Appendix A Section II “CAs for which no SSL/TLS certificates were issued during 2018”.

In addition, EY reissued the 2018 WebTrust for EV report for QuoVadis, effective 23 Sept 2019, to include VR IDENT EV SSL CA 2018 which had been omitted in error from the original report.

The reports are attached, or may be reached from https://www.quovadisglobal.com/AboutUs/Accreditations.aspx

Flags: needinfo?(karoshi-)

The following CAs have now been revoked:

Common Name crt.sh Note
QuoVadis EU Issuing Certification Authority https://crt.sh/?q=EC50E7E17D3802811C8B6567148CED68BBB1BD79EDDC61DBD298CEA5BA0FB862 Revoked
QuoVadis EU Issuing Certification Authority G2 https://crt.sh/?q=EC3F940A48EF7CBCEA4142F735A5DF2976DB38183D9033C76B78E25F8F53EB5B Revoked
QuoVadis Internal CA G1 https://crt.sh/?q=6B8973A0DBADA29988C5DC06CBCEF049BE770604F8A7436D817FAC3A9710F481 Revoked
QuoVadis Issuing CA 3 G3 https://crt.sh/?q=C12DD0347C0D4AA25D3986E0499740C5363A6B7EC32A49C5D18B9D56B075E368 Revoked
QuoVadis Issuing CA G3 https://crt.sh/?q=15CE38976716DCB35AA7B35FC168EBBB3BC2EC4696A8C795FC5C48457140E0A7 Revoked
QuoVadis Swiss Advanced CA https://crt.sh/?q=235C96A2E2DA557B904E90F3A0CAA57EABB4BDB5F401969DA8C282F60839568F Revoked

We reiterate that all 18 certificates were disclosed in the WebTrust for CA reports (and 3 were also in ETSI reports) but not disclosed in the WebTrust for BR reports as they did not issue TLS. In addition to the update of the 2018 WebTrust for BR to disclose the CAs, we have asked EY to provide an appropriate report for the years 2014-2017 for the remaining CAs. We are working with EY to define the report format and timeline.

(Note: This comment should not be inteprreted as personal or professional acceptance of Comment #9).

As the proposed remediation is supposedly to provide greater assurance to relying parties, it would seem important to ensure from EY:

  1. Why this information was not include in the relevant audit report. QuoVadis has been tasked with including this information since April 2017, so either QuoVadis failed to properly engage their auditor (thus calling into question the proposed remediation) or EY failed to properly follow the engaged expectations (thus calling into question reports from the auditor). This is particularly relevant, given that the WebTrust Task Force's Illustrative Reporting, finalized in September 2017 but circulated in draft form well before then, demonstrated how to properly include the relevant CAs within the scope of the reporting.

  2. Under what provisions of the applicable AICPA Professional Standards, AT-C standards for attestation engagements, the modifications to the existing reports and/or the issuance of new reports, are permitted and retroactively so? The proper use of reports remains critical to ensuring the necessary requirements are met; the proposed next steps provide no evidence that such information would be proper, or the proper use and interpretation of such a report, and thus may easily lead to such a report being misinterpreted as providing a degree of assurance it cannot and does not provide. Understanding the relevant professional standards is essential to ensuring that such a report is used consistently.

Regardless, the proposed remediation steps will not retroactively correct the non-compliance. While I understand Mozilla has taken the view that the use of OneCRL is an appropriate short-term mitigation for risk, other root programs view this as a significant matter of non-compliance, with an expectation of revocation, as has been done by other CAs in other issues. QuoVadis' approach, by comparison, leaves the broader ecosystem at substantially greater risk, and remains indistinguishable from a stalling tactic that might allow the exploitation of that non-compliance to expose users to risk. Regardless of QuoVadis' explanation of benign mistake, it is indistinguishable from malice based on the information provided, and would be wholly inappropriate a path to take in truly malicious situations. Similarly, the absence of concrete timelines and the lack of substantive action is deeply concerning and suggests this issue is not taken with the seriousness it deserves.

To note the following other bits that raise serious concerns with this proposal.

The 2018 report, for the period 2017-01-01 through 2017-12-31, issued 2018-03-29, , and the 2017 report, for the period 2016-01-01 through 2016-12-31, issued 2017-03-31, states it was performed by:

EY Bermuda Ltd, 3 Bermudiana Road, Hamilton HM08, Bermuda, P.O. Box HM 463, Hamilton HM BX, Bermuda

The 2019 report, for the period 2018-01-01 through 2018-12-31, issued 2019-03-29, states it was performed by:

Ernst & Young LLP, 1775 Tysons Blvd, McLean, VA 22102

The latter was recently subject to an auditor compliance issue, Bug 1582596

The following is an update on our efforts to remedy the non-BR disclosure of these CAs.

  1. The 18 CAs were disclosed in our WebTrust for CAs report and were included in the Webtrust for BR procedures but not disclosed in the report as they did not issue TLS. As previously reported in the bug, our auditors Ernst & Young LLP have restated their 2018 WebTrust for BR report to disclose the 18 CAs in Appendix II. This may be found via the BR site seal located at https://www.quovadisglobal.com/AboutUs/Accreditations
  2. As reported in the bug, we have so far revoked 6 of the Issuing CAs:
    • QuoVadis EU Issuing Certification Authority
    • QuoVadis EU Issuing Certification Authority G2
    • QuoVadis Issuing CA 3 G3
    • QuoVadis Issuing CA G3
    • QuoVadis Internal CA G1
    • QuoVadis Swiss Advanced CA
  3. We have requested the Issuing CAs be added to OneCRL (or equivalent such as crlSet) to negate their ability to be used to issue TLS certificates. This has already occurred in Mozilla and Microsoft.
  4. We have retained Ernst & Young LLP to provide an AT105 attestation report for the years 2014-2017 for 12 unrevoked CAs. The objective of the report is to confirm that the assertion that the Issuing CAs were in scope for WebTrust for BR and had not issued TLS certificates for the years ending December 31, 2014 through 2017 inclusive. The recipients of the report are limited to Apple, Chrome, Microsoft, and Mozilla. The timeline has been delayed by the fact that EY has audit engagements with an entity that recently invested in DigiCert, as well as with CAs operated by several browsers, requiring a more extensive independence check by EY than previously anticipated. We will provide another update when the delivery date of the report is defined.
  5. As reported in the bug, the Issuing CAs in question are used to issue certificates to individuals (both in personal contexts as well as corporate contexts) as well as uses such as e-Seals, and many are issued on smartcards and crypto-tokens. Six of the CAs had previously ceased issuance; the other 6 will cease as soon as replacement CAs can be provided. We have been actively seeking to assist customers in replacing end entity certificates that may be affected. We intend to revoke all 12 certificates, with the following proposed schedule.
Common Name Estimated Revoke Date Notes
QuoVadis Personal Signing Service CA G1 31 Oct 2019 Ceased issuance Nov 2018. Disabled in Certificate Management System (CMS).
QuoVadis EU Issuing Certification Authority G4 30 Nov 2019 On EU Trusted List. Disclosed in ETSI certificate.
QuoVadis ElDI-V CA G1 30 Nov 2019 Ceased issuance Mar 2019. Disabled in CMS. Smartcards and HSMs.
HydrantID Client ICA 31 Jan 2020 Replacement CA created Oct 2019. Large manual effort to replace certs.
QuoVadis Belgium Issuing CA G1 31 Jan 2020 Ceased issuance Nov 2018. Issuance disabled in CMS. Only 3 certs remain: revocation complicated as certs have been pinned in critical govt health application.
QuoVadis SuisseID Qualified CA 31 Dec 2019 Ceased issuance in 2017. Disabled in CMS. Signing certificates only. Large manual effort to replace; smartcards. Complicated as SuisseID program has ceased and no new cards are being issued.
QuoVadis SuisseID Advanced CA 31 Dec 2019 Ceased issuance Aug 2018. Disabled in CMS. Authentication certificates only. Comments as above.
QuoVadis EU Issuing Certification Authority G3 31 Jan 2020 On EU Trusted List. Disclosed in ETSI certificate. Approx 1,400 certs to reissue on smartcards.
QuoVadis Belgium Issuing CA G2 31 Jan 2020 On EU Trusted List. Disclosed in ETSI certificate. Plan to renew with EKUs then revoke the original.
HIN Health Info Net CA 31 Dec 2019 Ceased issuance May 2017. Disabled in CMS. Replacement CA already exists; scheduling reissue of affected certs.
QuoVadis Issuing CA G4 31 Mar 2020 Replacement G5 CA has been created. Approx 10,000 certs to reissue.
QuoVadis Swiss Advanced CA G2 31 Mar 2020 Replacement G3 CA has been created. Approx 60,000 certs to reissue heavily weighted in banking sector.

We are working with the end users to replace certificate and revoke these ICAs as quickly as possible based on feedback received, understanding the impact caused to end users by such revocations. We will update this schedule with firmer dates as they become available. We welcome your feedback.

Stephen:

Thanks for the update, including the clear timeline to revocation and the challenges being faced, which focuses on addressing the overall and underlying compliance issue. This appears to fit within the framework described for addressing these issues of non-compliance, and appears to be a reasonable timeframe based on the evidence. Note: This is not an exception to the BRs or the BRs' required timeframe to revocation, and is seen as an incident that fits within the overall picture of the CA.

With respect to your proposed inclusion of a report: presumably you meant AT-C 105 (and not AT 105), given SSAE18. It's unclear the type of attestation report being suggested, but it sounds like an Agreed Upon Procedures report, given the limited recipients. While this is useful in helping QuoVadis obtain an independent assessment of its system, from the point of view of relying parties and browsers, it's not useful as a tool for increasing confidence. This is due to the inherent limits in such reports, including how the procedures are crafted and the responsibilities with respect to the engaging party. In the past, while CAs have suggested the use of AUP reports (e.g. Turktrust and WoSign), these haven't been used to address the underlying issues. So if the goal is for improved compliance or assurance for browsers, this doesn't necessarily help. However, fit within your overall picture of planned remediation, it is useful in helping QuoVadis obtain the independent assurance, based on the procedures it determines.

I think this means that the necessary pieces from the Incident Report have now been provided. While Comment #0 discusses the work to ensure no future lapses in audits, it seems useful to discuss what operational changes or changes to hierarchies and PKI design may be done to ensure no future delays in revocation of intermediates, in the event those audit controls fail.

Flags: needinfo?(s.davidson)

As described in Comment 12, QuoVadis Personal Signing Service CA G1 has been revoked on 31 October: Cessation of Operation.
https://crt.sh/?q=F94C931373F850FF3D7DB5FB20AC04EC2F812CAEC9BD17E32DB6DCAE2269104D

Flags: needinfo?(s.davidson)

As described in Comment 12, the following CAs were revoked on 29 November:

QuoVadis EU Issuing Certification Authority G4
https://crt.sh/?q=0dd818228990d83fce9f9dca7b5cc44ed318edd16399987ea893877ea52de11e
superseded

QuoVadis ElDI-V CA G1
https://crt.sh/?q=393E95D3AE5233A04FEFE058BA8F445132D30E4362D5F7259061392716B34D2C
cessationOfOperation

You need to log in before you can comment on or make changes to this bug.