Closed Bug 1581597 Opened 5 years ago Closed 4 years ago

QuoVadis: Unconstrained CAs missing audits

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: stephen.davidson, Assigned: stephen.davidson)

Details

(Whiteboard: [ca-compliance] [ca-revocation-delay] [covid-19])

Attachments

(5 files)

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 10-September, we received an e-mail from the Chrome Root Authority program with the subject “Baseline Requirements Audit Status” requesting confirmation of Baseline Requirements disclosures.

We found 18 ICA certificates that although technically capable of TLS issuance (due to lack of EKUs) are not actually capable of TLS issuance as they are not provided TLS certificate profiles nor TLS workflows in our certificate management system. These ICAs do not issue TLS certificates. These ICAs are older, before QuoVadis commenced using explicit EKU in line with changing industry expectations. These ICAs are listed in the table below.

All these ICAs have always been disclosed in the annual WebTrust for CAs audit, which is conducted concurrently with the WebTrust audits for BRs and EVG. The complete population of QuoVadis issued certificates in the audit period is provided to the WebTrust auditors.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
  • 10-September-2019: received an e-mail from the Chrome Root Authority program
  • 10-September-2019: Performed investigation on EKUs and WebTrust CA scope
  • 13-September-2019: We met with the WebTrust auditors to finalize the plan to amend the WTBR report for the 1-January-2018 to 31-December-2018 audit period.
  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

N/A - no TLS server certificates have been issued from these CAs.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

N/A - no TLS server certificates have been issued from these CAs.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

QuoVadis did not include these unconstrained CAs in our most recent WTBR report. These CAs were however included in the WebTrust Principles and Criteria for Certification Authorities (WTCA) report.

Three of these impacted ICAs are also included in the scope of QuoVadis ETSI audits and explicitly named in the ETSI EN 319 411-1 / ETSI EN 319 411-2 certificates. The Baseline Requirements are referenced in “Audit Criteria” section. Since these 3 ICAs don’t issue TLS, the policies that are relevant to these CAs include NCP, NCP+, QCP-n, QCP-n-qscd, QCP-l, QCP-l-qscd. These ETSI reports are provided to our European Supervisory bodies and are posted on our website, but have not previously been uploaded to CCADB. For the remainder of the ICAs, there was an oversight of the intent of the Mozilla policy for covering them in a BR audit even though these ICAs were purpose built for other than TLS issuance.

https://www.quovadisglobal.com/~/media/Files/Files_Global/QuoVadis_ETS_030.ashx
https://www.quovadisglobal.com/~/media/Files/Files_Global/QuoVadis_ETS_010.ashx

No TLS certificates were issued from these CAs.

Common Name crt.sh Note
HIN Health Info Net CA https://crt.sh/?q=479F4E101F380691201A34CBADBC09E5F0523B35EE3839EB4B14332481EF8463
HydrantID Client ICA https://crt.sh/?q=2EEE91CC892A16CCB7320CDED2AE4948C052345D6B24E214C24EB93932D10DD9
QuoVadis Belgium Issuing CA G1 https://crt.sh/?q=27EBACD86DD3BF86143DA4342861031A57CF3FA414D40A86E669C3F4F1D8CF24
QuoVadis Belgium Issuing CA G2 https://crt.sh/?q=D90B40132306D1094608B1B9A2F6A9E23B45FE121FEF514A1C9DF70A815AD95C Disclosed in ETSI certificate
QuoVadis ElDI-V CA G1 https://crt.sh/?q=393E95D3AE5233A04FEFE058BA8F445132D30E4362D5F7259061392716B34D2C
QuoVadis EU Issuing Certification Authority https://crt.sh/?q=EC50E7E17D3802811C8B6567148CED68BBB1BD79EDDC61DBD298CEA5BA0FB862 Revoked
QuoVadis EU Issuing Certification Authority G2 https://crt.sh/?q=EC3F940A48EF7CBCEA4142F735A5DF2976DB38183D9033C76B78E25F8F53EB5B
QuoVadis EU Issuing Certification Authority G3 https://crt.sh/?q=ADDFFA6FD0809A54A9F0B31FD25F74BF7F2D7AE11C80FD99DAA0FB603A65CD0E Disclosed in ETSI certificate
QuoVadis EU Issuing Certification Authority G4 https://crt.sh/?q=0DD818228990D83FCE9F9DCA7B5CC44ED318EDD16399987EA893877EA52DE11E Disclosed in ETSI certificate
QuoVadis Internal CA G1 https://crt.sh/?q=6B8973A0DBADA29988C5DC06CBCEF049BE770604F8A7436D817FAC3A9710F481
QuoVadis Issuing CA 3 G3 https://crt.sh/?q=C12DD0347C0D4AA25D3986E0499740C5363A6B7EC32A49C5D18B9D56B075E368 Revoked
QuoVadis Issuing CA G3 https://crt.sh/?q=15CE38976716DCB35AA7B35FC168EBBB3BC2EC4696A8C795FC5C48457140E0A7
QuoVadis Issuing CA G4 https://crt.sh/?q=DA3BC81005FDBB853D681A7E942661AEBA23789211525EAF52221F28514C09CB
QuoVadis Personal Signing Service CA G1 https://crt.sh/?q=F94C931373F850FF3D7DB5FB20AC04EC2F812CAEC9BD17E32DB6DCAE2269104D
QuoVadis SuisseID Advanced CA https://crt.sh/?q=5DDAB0A802D83893AC0EDF9B30A620411B1A74A8B7D411A6A7AD7DC46EB1C8C8
QuoVadis SuisseID Qualified CA https://crt.sh/?q=5B7017B80F97C621AF1163B04BEBAFD2F932A42B85F4B9FEC71B38609F564922
QuoVadis Swiss Advanced CA https://crt.sh/?q=235C96A2E2DA557B904E90F3A0CAA57EABB4BDB5F401969DA8C282F60839568F
QuoVadis Swiss Advanced CA G2 https://crt.sh/?q=5044F65E1042CD380B0B9997E4283358F0DEEF7873DA72EFDB6F02474AE37EBE
  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

A decision was made to only list ICAs in the WebTrust for BR assertion that actually issued TLS. As such the 18 ICAs – which were confirmed not to issue TLS in the audit – were not listed in the WebTrust for BR’s report. That decision, in retrospect, was incorrect. In part, that decision was based on legacy ambiguity in the requirements to call out unconstrained ICAs in the WebTrust for BR’s report.

Many of these ICAs are old legacy ICAs that were created in the period 2009 to 2014.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
  • We are working with our WebTrust assessors to amend our most recent WTBR audit to explicitly disclose these impacted ICAs. The audit work has been completed and EY are in the reporting stages. We will update this disclosure when that is complete.
  • We are in the process of performing a review of all QuoVadis ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with https://bugzilla.mozilla.org/show_bug.cgi?id=1563573

We request that the ICAs in the attachment "QuoVadis_OneCRL_16Sept2019" be added to OneCRL.

Assignee: wthayer → s.davidson
Flags: needinfo?(kwilson)
Summary: QuoVadis: Unconstrained CAs → QuoVadis: Unconstrained CAs missing audits
Whiteboard: [ca-compliance]
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Stephen, Please check the attached spreadsheet to make sure it contains the correct items to add to OneCRL.

Flags: needinfo?(kwilson) → needinfo?(s.davidson)

Confirmed: the spreadsheet contains the correct items to add to OneCRL.
Thank you.

Flags: needinfo?(s.davidson)

(In reply to Stephen Davidson from comment #2)

Confirmed: the spreadsheet contains the correct items to add to OneCRL.

Those records in the CCADB are set as "Ready to Add" to OneCRL, so they will be added as part of a future batch of updates to OneCRL.

I think the remaining item for this bug is status on:

We are in the process of performing a review of all QuoVadis ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with https://bugzilla.mozilla.org/show_bug.cgi?id=1563573

We performed a review of all QuoVadis ICAs and reconciled them to the following 2018 WebTrust reports:
• WebTrust for CAs
• WebTrust for Baseline Requirements
• WebTrust for EV

Based on this reconciliation, there were no further additions to the 2018 WebTrust for Baseline Requirements report needed other than the 18 CAs already discussed above. EY are in the process of re-issuing the 2018 WebTrust for Baseline Requirements report. Target date for updated report is Friday 27th September.

This reconciliation exercise found one EV ICA that was missing from disclosure in the 2018 WebTrust for EV report (VR IDENT EV SSL CA 2018). This ICA was included in the scope of the EY EV testing and was disclosed in the 2018 WebTrust for CAs and 2018 WebTrust for Baseline Requirements reports but omitted in error from the 2018 WebTrust for EV report. EY are in the process of re-issuing the WebTrust for EV report to correct this. Target date for updated report is Friday 27th September.

QuoVadis will work closely with our auditors going forward to help ensure that CAs are disclosed appropriately in future audits.

Barry: Is the updated audit report available?

Flags: needinfo?(karoshi-)

EY reissued the 2018 WebTrust for Baseline Requirements report for QuoVadis, effective 13 Sept 2019, naming the 18 CAs in Appendix A Section II “CAs for which no SSL/TLS certificates were issued during 2018”.

In addition, EY reissued the 2018 WebTrust for EV report for QuoVadis, effective 23 Sept 2019, to include VR IDENT EV SSL CA 2018 which had been omitted in error from the original report.

The reports are attached, or may be reached from https://www.quovadisglobal.com/AboutUs/Accreditations.aspx

Flags: needinfo?(karoshi-)

The following CAs have now been revoked:

Common Name crt.sh Note
QuoVadis EU Issuing Certification Authority https://crt.sh/?q=EC50E7E17D3802811C8B6567148CED68BBB1BD79EDDC61DBD298CEA5BA0FB862 Revoked
QuoVadis EU Issuing Certification Authority G2 https://crt.sh/?q=EC3F940A48EF7CBCEA4142F735A5DF2976DB38183D9033C76B78E25F8F53EB5B Revoked
QuoVadis Internal CA G1 https://crt.sh/?q=6B8973A0DBADA29988C5DC06CBCEF049BE770604F8A7436D817FAC3A9710F481 Revoked
QuoVadis Issuing CA 3 G3 https://crt.sh/?q=C12DD0347C0D4AA25D3986E0499740C5363A6B7EC32A49C5D18B9D56B075E368 Revoked
QuoVadis Issuing CA G3 https://crt.sh/?q=15CE38976716DCB35AA7B35FC168EBBB3BC2EC4696A8C795FC5C48457140E0A7 Revoked
QuoVadis Swiss Advanced CA https://crt.sh/?q=235C96A2E2DA557B904E90F3A0CAA57EABB4BDB5F401969DA8C282F60839568F Revoked

We reiterate that all 18 certificates were disclosed in the WebTrust for CA reports (and 3 were also in ETSI reports) but not disclosed in the WebTrust for BR reports as they did not issue TLS. In addition to the update of the 2018 WebTrust for BR to disclose the CAs, we have asked EY to provide an appropriate report for the years 2014-2017 for the remaining CAs. We are working with EY to define the report format and timeline.

(Note: This comment should not be inteprreted as personal or professional acceptance of Comment #9).

As the proposed remediation is supposedly to provide greater assurance to relying parties, it would seem important to ensure from EY:

  1. Why this information was not include in the relevant audit report. QuoVadis has been tasked with including this information since April 2017, so either QuoVadis failed to properly engage their auditor (thus calling into question the proposed remediation) or EY failed to properly follow the engaged expectations (thus calling into question reports from the auditor). This is particularly relevant, given that the WebTrust Task Force's Illustrative Reporting, finalized in September 2017 but circulated in draft form well before then, demonstrated how to properly include the relevant CAs within the scope of the reporting.

  2. Under what provisions of the applicable AICPA Professional Standards, AT-C standards for attestation engagements, the modifications to the existing reports and/or the issuance of new reports, are permitted and retroactively so? The proper use of reports remains critical to ensuring the necessary requirements are met; the proposed next steps provide no evidence that such information would be proper, or the proper use and interpretation of such a report, and thus may easily lead to such a report being misinterpreted as providing a degree of assurance it cannot and does not provide. Understanding the relevant professional standards is essential to ensuring that such a report is used consistently.

Regardless, the proposed remediation steps will not retroactively correct the non-compliance. While I understand Mozilla has taken the view that the use of OneCRL is an appropriate short-term mitigation for risk, other root programs view this as a significant matter of non-compliance, with an expectation of revocation, as has been done by other CAs in other issues. QuoVadis' approach, by comparison, leaves the broader ecosystem at substantially greater risk, and remains indistinguishable from a stalling tactic that might allow the exploitation of that non-compliance to expose users to risk. Regardless of QuoVadis' explanation of benign mistake, it is indistinguishable from malice based on the information provided, and would be wholly inappropriate a path to take in truly malicious situations. Similarly, the absence of concrete timelines and the lack of substantive action is deeply concerning and suggests this issue is not taken with the seriousness it deserves.

To note the following other bits that raise serious concerns with this proposal.

The 2018 report, for the period 2017-01-01 through 2017-12-31, issued 2018-03-29, , and the 2017 report, for the period 2016-01-01 through 2016-12-31, issued 2017-03-31, states it was performed by:

EY Bermuda Ltd, 3 Bermudiana Road, Hamilton HM08, Bermuda, P.O. Box HM 463, Hamilton HM BX, Bermuda

The 2019 report, for the period 2018-01-01 through 2018-12-31, issued 2019-03-29, states it was performed by:

Ernst & Young LLP, 1775 Tysons Blvd, McLean, VA 22102

The latter was recently subject to an auditor compliance issue, Bug 1582596

The following is an update on our efforts to remedy the non-BR disclosure of these CAs.

  1. The 18 CAs were disclosed in our WebTrust for CAs report and were included in the Webtrust for BR procedures but not disclosed in the report as they did not issue TLS. As previously reported in the bug, our auditors Ernst & Young LLP have restated their 2018 WebTrust for BR report to disclose the 18 CAs in Appendix II. This may be found via the BR site seal located at https://www.quovadisglobal.com/AboutUs/Accreditations
  2. As reported in the bug, we have so far revoked 6 of the Issuing CAs:
    • QuoVadis EU Issuing Certification Authority
    • QuoVadis EU Issuing Certification Authority G2
    • QuoVadis Issuing CA 3 G3
    • QuoVadis Issuing CA G3
    • QuoVadis Internal CA G1
    • QuoVadis Swiss Advanced CA
  3. We have requested the Issuing CAs be added to OneCRL (or equivalent such as crlSet) to negate their ability to be used to issue TLS certificates. This has already occurred in Mozilla and Microsoft.
  4. We have retained Ernst & Young LLP to provide an AT105 attestation report for the years 2014-2017 for 12 unrevoked CAs. The objective of the report is to confirm that the assertion that the Issuing CAs were in scope for WebTrust for BR and had not issued TLS certificates for the years ending December 31, 2014 through 2017 inclusive. The recipients of the report are limited to Apple, Chrome, Microsoft, and Mozilla. The timeline has been delayed by the fact that EY has audit engagements with an entity that recently invested in DigiCert, as well as with CAs operated by several browsers, requiring a more extensive independence check by EY than previously anticipated. We will provide another update when the delivery date of the report is defined.
  5. As reported in the bug, the Issuing CAs in question are used to issue certificates to individuals (both in personal contexts as well as corporate contexts) as well as uses such as e-Seals, and many are issued on smartcards and crypto-tokens. Six of the CAs had previously ceased issuance; the other 6 will cease as soon as replacement CAs can be provided. We have been actively seeking to assist customers in replacing end entity certificates that may be affected. We intend to revoke all 12 certificates, with the following proposed schedule.
Common Name Estimated Revoke Date Notes
QuoVadis Personal Signing Service CA G1 31 Oct 2019 Ceased issuance Nov 2018. Disabled in Certificate Management System (CMS).
QuoVadis EU Issuing Certification Authority G4 30 Nov 2019 On EU Trusted List. Disclosed in ETSI certificate.
QuoVadis ElDI-V CA G1 30 Nov 2019 Ceased issuance Mar 2019. Disabled in CMS. Smartcards and HSMs.
HydrantID Client ICA 31 Jan 2020 Replacement CA created Oct 2019. Large manual effort to replace certs.
QuoVadis Belgium Issuing CA G1 31 Jan 2020 Ceased issuance Nov 2018. Issuance disabled in CMS. Only 3 certs remain: revocation complicated as certs have been pinned in critical govt health application.
QuoVadis SuisseID Qualified CA 31 Dec 2019 Ceased issuance in 2017. Disabled in CMS. Signing certificates only. Large manual effort to replace; smartcards. Complicated as SuisseID program has ceased and no new cards are being issued.
QuoVadis SuisseID Advanced CA 31 Dec 2019 Ceased issuance Aug 2018. Disabled in CMS. Authentication certificates only. Comments as above.
QuoVadis EU Issuing Certification Authority G3 31 Jan 2020 On EU Trusted List. Disclosed in ETSI certificate. Approx 1,400 certs to reissue on smartcards.
QuoVadis Belgium Issuing CA G2 31 Jan 2020 On EU Trusted List. Disclosed in ETSI certificate. Plan to renew with EKUs then revoke the original.
HIN Health Info Net CA 31 Dec 2019 Ceased issuance May 2017. Disabled in CMS. Replacement CA already exists; scheduling reissue of affected certs.
QuoVadis Issuing CA G4 31 Mar 2020 Replacement G5 CA has been created. Approx 10,000 certs to reissue.
QuoVadis Swiss Advanced CA G2 31 Mar 2020 Replacement G3 CA has been created. Approx 60,000 certs to reissue heavily weighted in banking sector.

We are working with the end users to replace certificate and revoke these ICAs as quickly as possible based on feedback received, understanding the impact caused to end users by such revocations. We will update this schedule with firmer dates as they become available. We welcome your feedback.

Stephen:

Thanks for the update, including the clear timeline to revocation and the challenges being faced, which focuses on addressing the overall and underlying compliance issue. This appears to fit within the framework described for addressing these issues of non-compliance, and appears to be a reasonable timeframe based on the evidence. Note: This is not an exception to the BRs or the BRs' required timeframe to revocation, and is seen as an incident that fits within the overall picture of the CA.

With respect to your proposed inclusion of a report: presumably you meant AT-C 105 (and not AT 105), given SSAE18. It's unclear the type of attestation report being suggested, but it sounds like an Agreed Upon Procedures report, given the limited recipients. While this is useful in helping QuoVadis obtain an independent assessment of its system, from the point of view of relying parties and browsers, it's not useful as a tool for increasing confidence. This is due to the inherent limits in such reports, including how the procedures are crafted and the responsibilities with respect to the engaging party. In the past, while CAs have suggested the use of AUP reports (e.g. Turktrust and WoSign), these haven't been used to address the underlying issues. So if the goal is for improved compliance or assurance for browsers, this doesn't necessarily help. However, fit within your overall picture of planned remediation, it is useful in helping QuoVadis obtain the independent assurance, based on the procedures it determines.

I think this means that the necessary pieces from the Incident Report have now been provided. While Comment #0 discusses the work to ensure no future lapses in audits, it seems useful to discuss what operational changes or changes to hierarchies and PKI design may be done to ensure no future delays in revocation of intermediates, in the event those audit controls fail.

Flags: needinfo?(s.davidson)

As described in Comment 12, QuoVadis Personal Signing Service CA G1 has been revoked on 31 October: Cessation of Operation.
https://crt.sh/?q=F94C931373F850FF3D7DB5FB20AC04EC2F812CAEC9BD17E32DB6DCAE2269104D

Flags: needinfo?(s.davidson)

As described in Comment 12, the following CAs were revoked on 29 November:

QuoVadis EU Issuing Certification Authority G4
https://crt.sh/?q=0dd818228990d83fce9f9dca7b5cc44ed318edd16399987ea893877ea52de11e
superseded

QuoVadis ElDI-V CA G1
https://crt.sh/?q=393E95D3AE5233A04FEFE058BA8F445132D30E4362D5F7259061392716B34D2C
cessationOfOperation

Short update:

  • The fieldwork on the AT 105 audit is well advanced. Timeline update will be provided as soon as possible.
  • We are on track to revoke an additional 3 CAs by December 31, bringing the total to 12.

As described in Comment 12, the following CA was revoked on 27 December:

HIN Health Info Net CA
https://crt.sh/?q=479F4E101F380691201A34CBADBC09E5F0523B35EE3839EB4B14332481EF8463
superseded

As described in Comment 12, the following CAs were revoked on 30 December:

QuoVadis SuisseID Advanced CA
https://crt.sh/?q=5DDAB0A802D83893AC0EDF9B30A620411B1A74A8B7D411A6A7AD7DC46EB1C8C8
cessationOfOperation

QuoVadis SuisseID Qualified CA
https://crt.sh/?q=5B7017B80F97C621AF1163B04BEBAFD2F932A42B85F4B9FEC71B38609F564922
cessationOfOperation

As described in [Comment 12] (https://bugzilla.mozilla.org/show_bug.cgi?id=1581597#c12), the following CAs were revoked on 31 January 2020:

QuoVadis EU Issuing Certification Authority G3
https://crt.sh/?q=ADDFFA6FD0809A54A9F0B31FD25F74BF7F2D7AE11C80FD99DAA0FB603A65CD0E
cessationOfOperation

HydrantID Client ICA
https://crt.sh/?q=2EEE91CC892A16CCB7320CDED2AE4948C052345D6B24E214C24EB93932D10DD9
cessationOfOperation

QuoVadis Belgium Issuing CA G2
https://crt.sh/?q=D90B40132306D1094608B1B9A2F6A9E23B45FE121FEF514A1C9DF70A815AD95C
superseded

QuoVadis Belgium Issuing CA G1
https://crt.sh/?q=27EBACD86DD3BF86143DA4342861031A57CF3FA414D40A86E669C3F4F1D8CF24
superseded

As such, 10 of the 12 CAs have now been revoked. The two remaining CAs require the replacement of a large number of end entity certificates and are scheduled for revocation on 31 March 2020. Unless there is significant activity to report in the interim, this bug will not be updated until then.

Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 31-March 2020

We are making steady progress on in replacing the ~60,000 end entity certificates under the remaining two subCAs. We are on track to revoke QuoVadis Swiss Advanced CA G2 on 31 Mar 2020.

However, under QuoVadis Issuing CA G4 we have issued nine Time-stamping Authority (TSA) certificates. If we revoke this subCA, any signature time-stamped through this hierarchy will be rendered invalid. All nine of the certificates expire on 17 Mar 2021, which is the ValidTo end-of-life of the subCA.

Thus our plan for QuoVadis Issuing CA G4 is now as follows:
• To revoke all remaining end entity certificates issued by the subCA with the exception of the nine TSA certificates
• Pre-generating CRL through expiry of the subCA
• Witnessed destruction of the subCA key pair

(In reply to Stephen Davidson from comment #20)

We are making steady progress on in replacing the ~60,000 end entity certificates under the remaining two subCAs. We are on track to revoke QuoVadis Swiss Advanced CA G2 on 31 Mar 2020.

However, under QuoVadis Issuing CA G4 we have issued nine Time-stamping Authority (TSA) certificates. If we revoke this subCA, any signature time-stamped through this hierarchy will be rendered invalid. All nine of the certificates expire on 17 Mar 2021, which is the ValidTo end-of-life of the subCA.

Thus our plan for QuoVadis Issuing CA G4 is now as follows:
• To revoke all remaining end entity certificates issued by the subCA with the exception of the nine TSA certificates
• Pre-generating CRL through expiry of the subCA
• Witnessed destruction of the subCA key pair

Thanks for the update. I can understand this is tricky, and the answers here are less clear. In terms of pre-generating CRLs, what would happen if one of the TSA certificates was compromised? Would you then revoke Issuing CA G4, since you'd be unable to issue a CRL for it?

Flags: needinfo?(s.davidson)

In terms of pre-generating CRLs, what would happen if one of the TSA certificates was compromised? Would you then revoke Issuing CA G4, since you'd be unable to issue a CRL for it?

If one of the TSA certificates was compromised and would otherwise require revocation, we would revoke QVICAG4.

Flags: needinfo?(s.davidson)

Wayne: In terms of next steps, it sounds like

  • 2020-03-31: The final planned revocation

And it sounds like potentially restricting or removing trust in QVICAG4 (which is only used for TSA, which as I understand Firefox/Mozilla products do not implement), either now or then, would work.

Setting explicit n-i for explicit review/confirmation of the plan in Comment #20 / Comment #22

Flags: needinfo?(wthayer)

The CAs were added to the Mozilla OneCRL last year following our initial request in this bug.
The subsequent revocations have been an additional step.

Based on comment #24, I don't believe any further action in regard to QVICAG4 is required.

Leaving this bug open pending the final planned revocation.

Flags: needinfo?(wthayer)

As reported earlier, we planned the revocation of QuoVadis Swiss Advanced CA G2 and the audited key destruction of QuoVadis Issuing CA G4 for 31 Mar 2020. Those actions will not be able to occur on that date due to unprecedented disruptions related to COVID-19. Given the unknown duration of these disruptions and government controls, we have set a new date of 29 April 2020. We will update here if circumstances allow an earlier date.

While our overall continuity plans have adapted, and our facilities remain secure and operational, this exercise faces challenges including government restrictions on movement and group gatherings, and the fact that team members are in mandated self-isolation following potential exposure to the virus.

For the same reason, our external auditor is not currently engaging in on-site work. Their participation is important given the non-routine circumstances of this revoke/key destruction.

Whiteboard: [ca-compliance] - Next Update - 31-March 2020 → [ca-compliance] [delayed-revocation-ca] [covid-19] - Next Update - 29-April 2020

As reported in Comment 12, QuoVadis retained Ernst & Young LLP to provide an AT105 attestation report for the years 2014-2017 for 12 unrevoked CAs.

Dated April 7, 2020, the resulting attestation report confirmed management’s assertion that the Issuing CAs identified in comment 12 were included in the WebTrust for BR procedures and had not issued TLS-capable certificates for the years ending December 31, 2014 through 2017 inclusive.

TLS-capable is defined as when a certificate lacked an Extended Key Usage, or included either an ‘anyExtendedKeyUsage’ or ‘id-kp-serverAuth’ extension.

The Ernst & Young attestation report is intended solely for the information and use of QuoVadis and Browser root distribution programs, and will be provided to Mozilla representatives at this time.

As reported in Comment 26, government-imposed lockdowns have delayed our ability complete the final two Issuing CA terminations with an external auditor present. The lockdown has now been extended until May 2, with potential phased reintroduction thereafter of business activities. We will update when we are able to schedule external auditors to witness the Issuing CA key destruction. Revocation of the relevant end entity certificates from the Issuing CAs is complete.

(In reply to Stephen Davidson from comment #27)

The Ernst & Young attestation report is intended solely for the information and use of QuoVadis and Browser root distribution programs, and will be provided to Mozilla representatives at this time.

Received via email. The attestation report said that during the time period (December 31, 2014 through 2017 inclusive) the listed ICAs did not issue TLS certificates.

(In reply to Stephen Davidson from comment #28)

As reported in Comment 26, government-imposed lockdowns have delayed our ability complete the final two Issuing CA terminations with an external auditor present. The lockdown has now been extended until May 2, with potential phased reintroduction thereafter of business activities. We will update when we are able to schedule external auditors to witness the Issuing CA key destruction. Revocation of the relevant end entity certificates from the Issuing CAs is complete.

I'll set the next update to May 9 for now.

Whiteboard: [ca-compliance] [delayed-revocation-ca] [covid-19] - Next Update - 29-April 2020 → [ca-compliance] [delayed-revocation-ca] [covid-19] - Next Update - May 9, 2020

The circumstances have not changed, with Government restrictions preventing external auditors from being physically present during the CA terminations. We expect a Government update on ~May 25 regarding next stages in opening up office operations. We continue to work ahead with the auditors on the appropriate form of the procedure, assertion, and auditor letter.

Whiteboard: [ca-compliance] [delayed-revocation-ca] [covid-19] - Next Update - May 9, 2020 → [ca-compliance] [delayed-revocation-ca] [covid-19] - Next Update - 25-May 2020

Access restrictions remain unchanged. We will update again during the week of June 8.

I'm explicitly not setting Needs-Info.

Right now, there's not enough information on this bug to justify the continued delay. In line with https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay , and the discussions around impacts related to Covid-19, I think we want to understand more details here.

I think it's worth reading https://bugzilla.mozilla.org/show_bug.cgi?id=1591005#c31 through to https://bugzilla.mozilla.org/show_bug.cgi?id=1591005#c37 and looking at how to provide the necessary level of information to help us understand the facts around these CAs.

Flags: needinfo?(stephen.davidson)
Whiteboard: [ca-compliance] [delayed-revocation-ca] [covid-19] - Next Update - 25-May 2020 → [ca-compliance] [delayed-revocation-ca] [covid-19]

The official COVID access restrictions have now been lifted, and plans are proceeding to complete the revocation and key destruction of these final two ICAs during the week of June 22.

Flags: needinfo?(stephen.davidson)

Effective today June 25, QuoVadis Swiss Advanced CA G2 has been revoked. Thank you for your patience through the unexpected COVID-19 delays.
https://crt.sh/?q=5044F65E1042CD380B0B9997E4283358F0DEEF7873DA72EFDB6F02474AE37EBE

Also today, a key destruction ceremony was completed on QuoVadis Issuing CA G4 as described in Comment 20.
https://crt.sh/?q=DA3BC81005FDBB853D681A7E942661AEBA23789211525EAF52221F28514C09CB

Validation of QuoVadis’ performance of its key destruction procedures was conducted by Ernst & Young in accordance with relevant criteria including the CA Key Destruction Criterion 4.6 of WebTrust for CAs (v2.2). The Ernst & Young attestation report is intended solely for the information and use of QuoVadis and Browser root distribution programs, and will be provided to Mozilla representatives when complete.

QuoVadis Issuing CA G4 expires on 31 March 2021. Even though the private key of the CA has been destroyed, it will be named it the relevant WebTrust reports for QuoVadis. In the interim, how should this CA be “tagged” in CCADB?

The confirmation of the audit report concludes the termination of the 18 CAs affected by this bug.

As described in Comment 20, QuoVadis retained Ernst & Young LLP to validate QuoVadis’ performance of the key destruction procedures for 'QuoVadis Issuing CA G4' in accordance with the QuoVadis CP/CPS and relevant criteria including the CA Key Destruction Criterion 4.6 of WebTrust for CAs (v2.2).

Dated July 13, 2020, the resulting report confirmed management’s assertion that the CA was destroyed in accordance with the criteria.

The Ernst & Young attestation report is intended solely for the information and use of QuoVadis and Browser root distribution programs, and has now been provided to Mozilla representatives.

Ben: Does Mozilla plan to share this report, as part of providing transparency and assurance to relying parties? While the CA is contractually limited in its distribution, as I understand it, Mozilla is not. I think that might be sufficient to close this out?

Flags: needinfo?(bwilson)

This attachment is subject to a restricted use by Mozilla and other internet browser root distribution programs and is not intended to be, and should not be, used by anyone else.

Flags: needinfo?(bwilson)

Thanks Ben. As mentioned in Comment #36, I think that's sufficient to close this out, as if my tracking in Comment #23 was correct, this should be the last CA.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED

I would like to point out that QuoVadis suggested in comment #26 that an audited key destruction would occur:

As reported earlier, we planned the revocation of QuoVadis Swiss Advanced CA G2 and the audited key destruction of QuoVadis Issuing CA G4 for 31 Mar 2020. Those actions will not be able to occur on that date due to unprecedented disruptions related to COVID-19. Given the unknown duration of these disruptions and government controls, we have set a new date of 29 April 2020.

The EY statement available on https://bug1581597.bmoattachments.org/attachment.cgi?id=9166900 does not reflect this. The statement from EY says:

Scope: We have examined management of QuoVadis Limited’s (“QuoVadis”) assertion ...
Auditor’s Opinion: In our opinion, management’s assertion, as referred to above, ...
followed by a

It seems to me that the only assertion the auditor makes in this statement is that, in his opinion, the management assertion from QuoVadis states that the CA listed are destroyed. To me it seems that they have not witnessed the key destruction, and I think the EY statement is intentionally misleading.

Flags: needinfo?(bwilson)

QuoVadis confirms that EY was present in person for the key destruction procedures. As described above, the key destruction was delayed during the period of COVID restrictions until the various parties could attend the procedures in person.

The E&Y audit statement was that they performed an examination that included "physical observation of all procedures performed during the CA destruction process to ensure that the procedures actually performed on June 25, 2020 were in accordance with the Key Destruction script(s) for the Subject Matter."

Flags: needinfo?(bwilson)
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-ca] [covid-19] → [ca-compliance] [ca-revocation-delay] [covid-19]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: