QuoVadis: Unconstrained CAs missing audits
Categories
(NSS :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: s.davidson, Assigned: s.davidson)
Details
(Whiteboard: [ca-compliance])
Attachments
(4 files)
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On 10-September, we received an e-mail from the Chrome Root Authority program with the subject “Baseline Requirements Audit Status” requesting confirmation of Baseline Requirements disclosures.
We found 18 ICA certificates that although technically capable of TLS issuance (due to lack of EKUs) are not actually capable of TLS issuance as they are not provided TLS certificate profiles nor TLS workflows in our certificate management system. These ICAs do not issue TLS certificates. These ICAs are older, before QuoVadis commenced using explicit EKU in line with changing industry expectations. These ICAs are listed in the table below.
All these ICAs have always been disclosed in the annual WebTrust for CAs audit, which is conducted concurrently with the WebTrust audits for BRs and EVG. The complete population of QuoVadis issued certificates in the audit period is provided to the WebTrust auditors.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
- 10-September-2019: received an e-mail from the Chrome Root Authority program
- 10-September-2019: Performed investigation on EKUs and WebTrust CA scope
- 13-September-2019: We met with the WebTrust auditors to finalize the plan to amend the WTBR report for the 1-January-2018 to 31-December-2018 audit period.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
N/A - no TLS server certificates have been issued from these CAs.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
N/A - no TLS server certificates have been issued from these CAs.
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
QuoVadis did not include these unconstrained CAs in our most recent WTBR report. These CAs were however included in the WebTrust Principles and Criteria for Certification Authorities (WTCA) report.
Three of these impacted ICAs are also included in the scope of QuoVadis ETSI audits and explicitly named in the ETSI EN 319 411-1 / ETSI EN 319 411-2 certificates. The Baseline Requirements are referenced in “Audit Criteria” section. Since these 3 ICAs don’t issue TLS, the policies that are relevant to these CAs include NCP, NCP+, QCP-n, QCP-n-qscd, QCP-l, QCP-l-qscd. These ETSI reports are provided to our European Supervisory bodies and are posted on our website, but have not previously been uploaded to CCADB. For the remainder of the ICAs, there was an oversight of the intent of the Mozilla policy for covering them in a BR audit even though these ICAs were purpose built for other than TLS issuance.
https://www.quovadisglobal.com/~/media/Files/Files_Global/QuoVadis_ETS_030.ashx
https://www.quovadisglobal.com/~/media/Files/Files_Global/QuoVadis_ETS_010.ashx
No TLS certificates were issued from these CAs.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
A decision was made to only list ICAs in the WebTrust for BR assertion that actually issued TLS. As such the 18 ICAs – which were confirmed not to issue TLS in the audit – were not listed in the WebTrust for BR’s report. That decision, in retrospect, was incorrect. In part, that decision was based on legacy ambiguity in the requirements to call out unconstrained ICAs in the WebTrust for BR’s report.
Many of these ICAs are old legacy ICAs that were created in the period 2009 to 2014.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
- We are working with our WebTrust assessors to amend our most recent WTBR audit to explicitly disclose these impacted ICAs. The audit work has been completed and EY are in the reporting stages. We will update this disclosure when that is complete.
- We are in the process of performing a review of all QuoVadis ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with https://bugzilla.mozilla.org/show_bug.cgi?id=1563573
We request that the ICAs in the attachment "QuoVadis_OneCRL_16Sept2019" be added to OneCRL.
|
||
Updated•3 months ago
|
|
||
Comment 1•3 months ago
|
||
Stephen, Please check the attached spreadsheet to make sure it contains the correct items to add to OneCRL.
|
Assignee | |
Comment 2•3 months ago
|
||
Confirmed: the spreadsheet contains the correct items to add to OneCRL.
Thank you.
|
|
||
Comment 3•3 months ago
|
||
(In reply to Stephen Davidson from comment #2)
Confirmed: the spreadsheet contains the correct items to add to OneCRL.
Those records in the CCADB are set as "Ready to Add" to OneCRL, so they will be added as part of a future batch of updates to OneCRL.
I think the remaining item for this bug is status on:
We are in the process of performing a review of all QuoVadis ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with https://bugzilla.mozilla.org/show_bug.cgi?id=1563573
|
||
Comment 4•3 months ago
|
||
We performed a review of all QuoVadis ICAs and reconciled them to the following 2018 WebTrust reports:
• WebTrust for CAs
• WebTrust for Baseline Requirements
• WebTrust for EV
Based on this reconciliation, there were no further additions to the 2018 WebTrust for Baseline Requirements report needed other than the 18 CAs already discussed above. EY are in the process of re-issuing the 2018 WebTrust for Baseline Requirements report. Target date for updated report is Friday 27th September.
This reconciliation exercise found one EV ICA that was missing from disclosure in the 2018 WebTrust for EV report (VR IDENT EV SSL CA 2018). This ICA was included in the scope of the EY EV testing and was disclosed in the 2018 WebTrust for CAs and 2018 WebTrust for Baseline Requirements reports but omitted in error from the 2018 WebTrust for EV report. EY are in the process of re-issuing the WebTrust for EV report to correct this. Target date for updated report is Friday 27th September.
QuoVadis will work closely with our auditors going forward to help ensure that CAs are disclosed appropriately in future audits.
|
|
Assignee | |
Comment 6•2 months ago
|
||
|
|
Assignee | |
Comment 7•2 months ago
|
||
|
|
||
Comment 8•2 months ago
|
||
EY reissued the 2018 WebTrust for Baseline Requirements report for QuoVadis, effective 13 Sept 2019, naming the 18 CAs in Appendix A Section II “CAs for which no SSL/TLS certificates were issued during 2018”.
In addition, EY reissued the 2018 WebTrust for EV report for QuoVadis, effective 23 Sept 2019, to include VR IDENT EV SSL CA 2018 which had been omitted in error from the original report.
The reports are attached, or may be reached from https://www.quovadisglobal.com/AboutUs/Accreditations.aspx
|
|
Assignee | |
Comment 9•2 months ago
|
||
The following CAs have now been revoked:
| Common Name | crt.sh | Note |
|---|---|---|
| QuoVadis EU Issuing Certification Authority | https://crt.sh/?q=EC50E7E17D3802811C8B6567148CED68BBB1BD79EDDC61DBD298CEA5BA0FB862 | Revoked |
| QuoVadis EU Issuing Certification Authority G2 | https://crt.sh/?q=EC3F940A48EF7CBCEA4142F735A5DF2976DB38183D9033C76B78E25F8F53EB5B | Revoked |
| QuoVadis Internal CA G1 | https://crt.sh/?q=6B8973A0DBADA29988C5DC06CBCEF049BE770604F8A7436D817FAC3A9710F481 | Revoked |
| QuoVadis Issuing CA 3 G3 | https://crt.sh/?q=C12DD0347C0D4AA25D3986E0499740C5363A6B7EC32A49C5D18B9D56B075E368 | Revoked |
| QuoVadis Issuing CA G3 | https://crt.sh/?q=15CE38976716DCB35AA7B35FC168EBBB3BC2EC4696A8C795FC5C48457140E0A7 | Revoked |
| QuoVadis Swiss Advanced CA | https://crt.sh/?q=235C96A2E2DA557B904E90F3A0CAA57EABB4BDB5F401969DA8C282F60839568F | Revoked |
We reiterate that all 18 certificates were disclosed in the WebTrust for CA reports (and 3 were also in ETSI reports) but not disclosed in the WebTrust for BR reports as they did not issue TLS. In addition to the update of the 2018 WebTrust for BR to disclose the CAs, we have asked EY to provide an appropriate report for the years 2014-2017 for the remaining CAs. We are working with EY to define the report format and timeline.
|
||
Comment 10•2 months ago
|
||
(Note: This comment should not be inteprreted as personal or professional acceptance of Comment #9).
As the proposed remediation is supposedly to provide greater assurance to relying parties, it would seem important to ensure from EY:
-
Why this information was not include in the relevant audit report. QuoVadis has been tasked with including this information since April 2017, so either QuoVadis failed to properly engage their auditor (thus calling into question the proposed remediation) or EY failed to properly follow the engaged expectations (thus calling into question reports from the auditor). This is particularly relevant, given that the WebTrust Task Force's Illustrative Reporting, finalized in September 2017 but circulated in draft form well before then, demonstrated how to properly include the relevant CAs within the scope of the reporting.
-
Under what provisions of the applicable AICPA Professional Standards, AT-C standards for attestation engagements, the modifications to the existing reports and/or the issuance of new reports, are permitted and retroactively so? The proper use of reports remains critical to ensuring the necessary requirements are met; the proposed next steps provide no evidence that such information would be proper, or the proper use and interpretation of such a report, and thus may easily lead to such a report being misinterpreted as providing a degree of assurance it cannot and does not provide. Understanding the relevant professional standards is essential to ensuring that such a report is used consistently.
Regardless, the proposed remediation steps will not retroactively correct the non-compliance. While I understand Mozilla has taken the view that the use of OneCRL is an appropriate short-term mitigation for risk, other root programs view this as a significant matter of non-compliance, with an expectation of revocation, as has been done by other CAs in other issues. QuoVadis' approach, by comparison, leaves the broader ecosystem at substantially greater risk, and remains indistinguishable from a stalling tactic that might allow the exploitation of that non-compliance to expose users to risk. Regardless of QuoVadis' explanation of benign mistake, it is indistinguishable from malice based on the information provided, and would be wholly inappropriate a path to take in truly malicious situations. Similarly, the absence of concrete timelines and the lack of substantive action is deeply concerning and suggests this issue is not taken with the seriousness it deserves.
|
|
||
Comment 11•2 months ago
|
||
To note the following other bits that raise serious concerns with this proposal.
The 2018 report, for the period 2017-01-01 through 2017-12-31, issued 2018-03-29, , and the 2017 report, for the period 2016-01-01 through 2016-12-31, issued 2017-03-31, states it was performed by:
EY Bermuda Ltd, 3 Bermudiana Road, Hamilton HM08, Bermuda, P.O. Box HM 463, Hamilton HM BX, Bermuda
The 2019 report, for the period 2018-01-01 through 2018-12-31, issued 2019-03-29, states it was performed by:
Ernst & Young LLP, 1775 Tysons Blvd, McLean, VA 22102
The latter was recently subject to an auditor compliance issue, Bug 1582596
|
|
Assignee | |
Comment 12•2 months ago
|
||
The following is an update on our efforts to remedy the non-BR disclosure of these CAs.
- The 18 CAs were disclosed in our WebTrust for CAs report and were included in the Webtrust for BR procedures but not disclosed in the report as they did not issue TLS. As previously reported in the bug, our auditors Ernst & Young LLP have restated their 2018 WebTrust for BR report to disclose the 18 CAs in Appendix II. This may be found via the BR site seal located at https://www.quovadisglobal.com/AboutUs/Accreditations
- As reported in the bug, we have so far revoked 6 of the Issuing CAs:
• QuoVadis EU Issuing Certification Authority
• QuoVadis EU Issuing Certification Authority G2
• QuoVadis Issuing CA 3 G3
• QuoVadis Issuing CA G3
• QuoVadis Internal CA G1
• QuoVadis Swiss Advanced CA - We have requested the Issuing CAs be added to OneCRL (or equivalent such as crlSet) to negate their ability to be used to issue TLS certificates. This has already occurred in Mozilla and Microsoft.
- We have retained Ernst & Young LLP to provide an AT105 attestation report for the years 2014-2017 for 12 unrevoked CAs. The objective of the report is to confirm that the assertion that the Issuing CAs were in scope for WebTrust for BR and had not issued TLS certificates for the years ending December 31, 2014 through 2017 inclusive. The recipients of the report are limited to Apple, Chrome, Microsoft, and Mozilla. The timeline has been delayed by the fact that EY has audit engagements with an entity that recently invested in DigiCert, as well as with CAs operated by several browsers, requiring a more extensive independence check by EY than previously anticipated. We will provide another update when the delivery date of the report is defined.
- As reported in the bug, the Issuing CAs in question are used to issue certificates to individuals (both in personal contexts as well as corporate contexts) as well as uses such as e-Seals, and many are issued on smartcards and crypto-tokens. Six of the CAs had previously ceased issuance; the other 6 will cease as soon as replacement CAs can be provided. We have been actively seeking to assist customers in replacing end entity certificates that may be affected. We intend to revoke all 12 certificates, with the following proposed schedule.
| Common Name | Estimated Revoke Date | Notes |
|---|---|---|
| QuoVadis Personal Signing Service CA G1 | 31 Oct 2019 | Ceased issuance Nov 2018. Disabled in Certificate Management System (CMS). |
| QuoVadis EU Issuing Certification Authority G4 | 30 Nov 2019 | On EU Trusted List. Disclosed in ETSI certificate. |
| QuoVadis ElDI-V CA G1 | 30 Nov 2019 | Ceased issuance Mar 2019. Disabled in CMS. Smartcards and HSMs. |
| HydrantID Client ICA | 31 Jan 2020 | Replacement CA created Oct 2019. Large manual effort to replace certs. |
| QuoVadis Belgium Issuing CA G1 | 31 Jan 2020 | Ceased issuance Nov 2018. Issuance disabled in CMS. Only 3 certs remain: revocation complicated as certs have been pinned in critical govt health application. |
| QuoVadis SuisseID Qualified CA | 31 Dec 2019 | Ceased issuance in 2017. Disabled in CMS. Signing certificates only. Large manual effort to replace; smartcards. Complicated as SuisseID program has ceased and no new cards are being issued. |
| QuoVadis SuisseID Advanced CA | 31 Dec 2019 | Ceased issuance Aug 2018. Disabled in CMS. Authentication certificates only. Comments as above. |
| QuoVadis EU Issuing Certification Authority G3 | 31 Jan 2020 | On EU Trusted List. Disclosed in ETSI certificate. Approx 1,400 certs to reissue on smartcards. |
| QuoVadis Belgium Issuing CA G2 | 31 Jan 2020 | On EU Trusted List. Disclosed in ETSI certificate. Plan to renew with EKUs then revoke the original. |
| HIN Health Info Net CA | 31 Dec 2019 | Ceased issuance May 2017. Disabled in CMS. Replacement CA already exists; scheduling reissue of affected certs. |
| QuoVadis Issuing CA G4 | 31 Mar 2020 | Replacement G5 CA has been created. Approx 10,000 certs to reissue. |
| QuoVadis Swiss Advanced CA G2 | 31 Mar 2020 | Replacement G3 CA has been created. Approx 60,000 certs to reissue heavily weighted in banking sector. |
We are working with the end users to replace certificate and revoke these ICAs as quickly as possible based on feedback received, understanding the impact caused to end users by such revocations. We will update this schedule with firmer dates as they become available. We welcome your feedback.
|
|
||
Comment 13•2 months ago
|
||
Stephen:
Thanks for the update, including the clear timeline to revocation and the challenges being faced, which focuses on addressing the overall and underlying compliance issue. This appears to fit within the framework described for addressing these issues of non-compliance, and appears to be a reasonable timeframe based on the evidence. Note: This is not an exception to the BRs or the BRs' required timeframe to revocation, and is seen as an incident that fits within the overall picture of the CA.
With respect to your proposed inclusion of a report: presumably you meant AT-C 105 (and not AT 105), given SSAE18. It's unclear the type of attestation report being suggested, but it sounds like an Agreed Upon Procedures report, given the limited recipients. While this is useful in helping QuoVadis obtain an independent assessment of its system, from the point of view of relying parties and browsers, it's not useful as a tool for increasing confidence. This is due to the inherent limits in such reports, including how the procedures are crafted and the responsibilities with respect to the engaging party. In the past, while CAs have suggested the use of AUP reports (e.g. Turktrust and WoSign), these haven't been used to address the underlying issues. So if the goal is for improved compliance or assurance for browsers, this doesn't necessarily help. However, fit within your overall picture of planned remediation, it is useful in helping QuoVadis obtain the independent assurance, based on the procedures it determines.
I think this means that the necessary pieces from the Incident Report have now been provided. While Comment #0 discusses the work to ensure no future lapses in audits, it seems useful to discuss what operational changes or changes to hierarchies and PKI design may be done to ensure no future delays in revocation of intermediates, in the event those audit controls fail.
|
|
Assignee | |
Comment 14•Last month
|
||
As described in Comment 12, QuoVadis Personal Signing Service CA G1 has been revoked on 31 October: Cessation of Operation.
https://crt.sh/?q=F94C931373F850FF3D7DB5FB20AC04EC2F812CAEC9BD17E32DB6DCAE2269104D
|
|
Assignee | |
Comment 15•13 days ago
|
||
As described in Comment 12, the following CAs were revoked on 29 November:
QuoVadis EU Issuing Certification Authority G4
https://crt.sh/?q=0dd818228990d83fce9f9dca7b5cc44ed318edd16399987ea893877ea52de11e
superseded
QuoVadis ElDI-V CA G1
https://crt.sh/?q=393E95D3AE5233A04FEFE058BA8F445132D30E4362D5F7259061392716B34D2C
cessationOfOperation
Description
•