Closed Bug 1591005 Opened 2 years ago Closed 8 months ago

GlobalSign: ICAs in CCADB, without EKU extension are listed in WTCA report but not in WTBR report

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: arvid.vermote, Assigned: arvid.vermote)

Details

(Whiteboard: [ca-compliance] [delayed-revocation-ca] Next Update 2021-03-15)

Attachments

(9 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36

Steps to reproduce:

On 16-October, when we reviewed the items listed in our CCADB task "Check failed Audit Letter Validation (ALV) results".

Actual results:

We found there are 30 ICA certificates that are technically capable of TLS issuance (due to lack of EKUs) which were not included in our SSLBR and EVSSL audit reports.

Expected results:

The respective ICA certificates should have been included in the SSLBR and EVSSL audit reports (see attached table).

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On 16-October, when we reviewed the items listed in our CCADB task "Check failed Audit Letter Validation (ALV) results", we found there are 30 ICA certificates, that although technically capable of TLS issuance (due to lack of EKUs) are not actually capable of TLS issuance as they are not provided TLS certificate profiles nor TLS workflows in our certificate management system. These ICAs do not issue TLS certificates. These ICAs are older, before GlobalSign commenced using explicit EKU in line with changing industry expectations. These ICAs are listed in the table below.

All these ICAs were disclosed in CCADB, and have always been disclosed in the annual WebTrust for CAs audit, which is conducted concurrently with the WebTrust audits for BRs and EVG. The complete population of GlobalSign issued certificates in the audit period is provided to the WebTrust auditors.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  • 16-October-2019: discover 30 ICA in CCADB without EKU but not intended for TLS issuance are listed in WebTrust audit for CAs audit report, but not listed in WebTrust audit for BR report.
  • 17-October-2019: Performed investigation on EKUs and WebTrust CA scope
  • 18-October-2019: We met with the WebTrust auditors to work out the plan to amend the WTBR report for the 1-April-2018 to 31-March-2019 audit period.
  • 23-October-2019: Further workshops with auditor to revalidate the EE certs issued from these impacted ICAs to confirm none of them are TLS EE certs. We expect the revalidation to be completed by within around 3 weeks.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
N/A - no TLS server certificates have been issued from these CAs.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
N/A - no TLS server certificates have been issued from these CAs.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

GlobalSign did not include these unconstrained CAs in our most recent WTBR report. These CAs were however included in the WebTrust Principles and Criteria for Certification Authorities (WTCA) report. All of them were disclosed in CCADB. These ICAs missing in the WTBR report was an oversight of the intent of the Mozilla policy for covering them in a BR audit even though these ICAs were purposely built not for TLS issuance.
No TLS certificates were issued from these CAs.

Please refer to the file attached to the initial bug report for an overview of the affected ICA.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The mistake was due to our false understanding of the meaning of technical capability of ICAs.

In Mozilla root policy up to v2.4 (Publication date: February 28, 2017), Audit Requirements #2 states,
2. CA operations relating to issuance of certificates capable of being used for SSL-enabled servers must also conform to the latest version of the CA/Browser Forum Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates.

Then Mozilla root policy since v2.4.1 (Publication date: March 31, 2017), 3.1.2 Required Audits states,
3.1.2.1 WebTrust
If being audited to the WebTrust criteria, the following audits are required (see section 3.1.1 for specific version numbers):
•For the SSL trust bit, a CA and all subordinate CAs technically capable of issuing server certificates must have all of the following:
◦WebTrust for CAs
◦WebTrust for CAs - SSL Baseline with Network Security
◦WebTrust for CAs - EV SSL (if applying for EV recognition)
•For the email trust bit, a CA and all subordinate CAs technically capable of issuing email certificates must have all of the following:
◦WebTrust for CAs

While in the Mozilla 'April 2017 CA Communication' survey, ACTION 3: UPDATED MOZILLA CA CERTIFICATE POLICY stated,
Versions 2.4 and 2.4.1 of Mozilla's CA Certificate Policy have been published. Differences between versions 2.4 and 2.3 (published Dec 2016), and between versions 2.4 and 2.2 (published July 2013) may be viewed on Github. Version 2.4.1 contains exactly the same normative requirements as version 2.4 but has been completely reorganized. Please review version 2.4.1 of Mozilla's CA Certificate Policy and confirm that your CA complies with it.

So when we read the 3.1.2.1 WebTrust audit requirements, we thought the trust bit refers to the Mozilla trusted root store setting, and “technically capable of issuing server certificates” still has the same mean as in v2.4, “CA operations relating to issuance of certificates capable of being used for SSL-enabled servers”, and both were referring to issuing system control in all of operational, procedural, technical aspects.

Another factor enforced our wrong interpretation regarding purpose or capability of ICAs, was actual via reading of Microsoft policy. Microsoft policy has been had the separation of purposes quite some time ago. In Sept 2016, before creating a bundle of ICAs, we specifically arranged a meeting with Microsoft root policy team to clarify their policy about the purpose and separation of ICAs. Microsoft advised us that, the conclusion, is pretty simple. You must put EKUs in end entities and you may put them in intermediates. Intermediates can therefore also have AnyEKU (or no EKU) but end entities cannot have this as the risk is a code signing certificate that could be used for SSL (or Visa Versa) if the root is marked for SSL and no EKUs are present in the chain. Old and new roots are really treated the same, the only difference being that new roots are specifically marked for certain EKUs where as old roots were not, so that’s why they originally separated in the root program text to rule 11 and 12.

This reinforced our wrong understanding that the procedure control will also be a deciding factor when looking at the capability of an ICA. Because of this false impression we had, we didn‘t realised that the Mozzila policy ver 2.4.1 audit requirement only based on EKUs, without considering the technical and procedural controls we put in on these ICAs.

We think the new ALV feature provided in CCADB now is very useful to communicate and clarify our wrong understanding about this Mozilla audit expectation.

7. List of steps your CA is taking to resolve the situation and ensure such issuace will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

  • We are now working with our WebTrust auditor to revalidate the EE certs issued from these impacted ICAs to confirm none of them are TLS EE certs. We expect the revalidation to be completed by within around 3 weeks.
  • When this revalidation work is completed, we will work with our WebTrust auditor to amend our most recent WTBR audit to explicitly disclose these impacted ICAs. We will update this disclosure when that is complete, which should be shortly after the completion of revalidation work.
  • Although not reflected in the ALV report of CCADB, we went ahead and further requested the WebTrust auditor to review all the ICAs in the WTBR report and amend the WTEVSSL report for the same period, if any ICAs are technically capable of issuing EV EE certs due to unconstrained policy OIDs.
  • We are in the process of performing a review of all GlobalSign ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with Mozilla Root Program Policy 3.1.2.1 WebTrust.
Assignee: wthayer → arvid.vermote
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance]

I'm trying to reconcile and understand some of the statements here.

Audit Scope

The statement is that they were included within the scope of the WTBR audit, and yet, in terms of remediation, GlobalSign is planning to deliver to their WT auditors the set of certificates they issued. This naturally leads to a conclusion that this infrastructure was not in scope of the WTBR audit, as otherwise, the WT auditor would and should have performed evaluation of controls, at the time of the audit, regarding WTBR compliance.

Put differently: If you have to disclose the certificates you issued to your auditor, in order to establish no TLS issuance, it means the auditor did not, at the time of the WTBR audit, actually examine that no TLS issuance happened.

Changes to Mozilla Requirements

It's extremely helpful and encouraging that GlobalSign reviewed the changes to Mozilla Policy in order to understand where the confusion was introduced. However, what I'm concerned is how limited the the review of the past CA Communications is.

Mozilla communicated expectations several times during that same time period, GlobalSign responded to the November 2017 communication, the April 2017 communication, and the September 2018 communication.

I can understand that, once a systemic misunderstanding was introduced - despite clear communication from Mozilla about the policy expectations as to what "technically capable" meant (and how it was not supported by GlobalSign) - no subsequent reviews identified this significant divergence from expectation.

Proposed Remediations

  • You should discuss with all Root Programs you participate in this proposed remediation. Other root programs have consistently rejected these proposals, due to the lack of assurance they provide, and required that, as required by the Baseline Requirements, these certificates be revoked. Your remediation plan does not provide a clear timeline for revocation, which is the Baseline Requirements expectation.
  • The WebTrust illustrative guidance, published in 2017, has included guidance for how WebTrust auditors can and should disclose the scope of the audit. The omission of this information, while the inclusion of other information, raises serious questions about whether the amended report provides adequate assurance.
  • None of this remediation plan seems to address the systemic issues that lead to the failure to comply with Mozilla Policy, and the subsequent misstatements about compliance made to Mozilla and the community. For example, Mozilla Policy Section 5.3 clearly defines what technical capability is, and so it's unclear how that could be selectively ignored.
    • It would seem that any evaluation would need to go back to at least policy 2.4.1, as it calls into question all of GlobalSign's operations since then, and all new policies adopted.
    • It would also seem to call into question all of GlobalSign's responses to CA communications since then.
Flags: needinfo?(arvid.vermote)

The respective ICA and leaf certificates were provided to the auditor as part of the raw population which the auditor uses to perform tests of effectiveness based on their audit (sampling and) testing methodology. These ICA are managed in the same realm and infrastructure as all other ICA in scope of our audits.

The reason they were omitted from the SSLBR report is because of an error (based on our described misinterpretation of the root policies) in the way we compiled the scope for our assertion in the different SOC3 WebTrust reports, not because these ICA or supporting infrastructure were not subject to audit.

When we discussed the issue at hand with our auditor they stated they wanted to again look at the respective populations from the period under audit to ensure no TLS certificates had been issued from these ICA. This is what we meant when using the word “revalidation”.

Since our initial post we’ve had further internal discussions and apart from, once available, disclosing the rectified reports, we would like to request for all identified ICA to be added to oneCRL. (as they are not used to issue TLS certificates)

Flags: needinfo?(arvid.vermote)

Can you please indicate the timelines that you'll be revoking these certificates?

Flags: needinfo?(arvid.vermote)

Again, can you please provide a timeline for prompt revocation, as the Baseline Requirements require?

Flags: needinfo?(douglas.beattie)

Please find attached an updated overview of the affected ICAs and their current plan of action:

  • Three (3) ICAs will be revoked on November 20 2019
  • Six (6) ICAs have been created after the start of the period-under-audit of our latest cycle (April 1 2018) and will be covered by the updated report as soon as made available by our auditor
  • Twenty-one (21) ICAs are being discussed with the relevant product owners and impacted clients for replacement as, although they are not used to issue TLS certificates, a significant number of leaf certificates (mainly AATL, SMIME and Time-stamping certificates) is chained towards them and immediate revocation will result in a significant operational impact towards our customers

We will provide another update on November 21 2019 with more solid plans for the 21 ICA mentioned above. We expect to do another round of revocation in the last week of November or the first week of December, the exact date of that revocation ceremony will also be shared on November 21 2019.

In the meanwhile we request Mozilla and the other WebPKI root programs to add the affected ICA to OneCRL or equivalent root program revocation mechanisms.

Flags: needinfo?(arvid.vermote)

We will also be submitting a separate incident report for not revoking these ICA within the timeframe required by the Baseline Requirements.

(In reply to Arvid Vermote from comment #6)

In the meanwhile we request Mozilla and the other WebPKI root programs to add the affected ICA to OneCRL or equivalent root program revocation mechanisms.

Arvid, Please clarify which intermediate certs should be added to OneCRL. i.e. Should all of the intermediate certs listed in the attachment to Comment #7 be added to OneCRL?

(In reply to Kathleen Wilson from comment #9)

(In reply to Arvid Vermote from comment #6)

In the meanwhile we request Mozilla and the other WebPKI root programs to add the affected ICA to OneCRL or equivalent root program revocation mechanisms.

Arvid, Please clarify which intermediate certs should be added to OneCRL. i.e. Should all of the intermediate certs listed in the attachment to Comment #7 be added to OneCRL?

Hi Kathleen, yes - all 30 attached intermediate certs should be added to OneCLR. Thanks, Arvid.

(In reply to Arvid Vermote from comment #10)

Hi Kathleen, yes - all 30 attached intermediate certs should be added to OneCLR. Thanks, Arvid.

Arvid, The attached spreadsheet shows the certs that have been set to "Ready to Add" to OneCRL. Please check the list to confirm it is correct.

(In reply to Kathleen Wilson from comment #11)

Created attachment 9110127 [details]
GlobalSign-OneCRL-Request-Nov2019.xls

(In reply to Arvid Vermote from comment #10)

Hi Kathleen, yes - all 30 attached intermediate certs should be added to OneCLR. Thanks, Arvid.

Arvid, The attached spreadsheet shows the certs that have been set to "Ready to Add" to OneCRL. Please check the list to confirm it is correct.

Hi Kathleen, I confirm this spreadsheet is correct. Thank you.

As communicated earlier three (3) ICAs have been revoked on November 20 2019. We will do another batch of revocations on December 4 2019. So far we have confirmed five (5) ICAs to be revoked on December 4 2019 with another potential four (4) given we can timely replace leaf certificates and ensure no impact. Please refer to the newly attached file for the updated status of all affected ICA.

We will provide a next update on November 25 2019.

The audit report should follow later this week and we are further identifying and planning a next batch of revocations on December 4 2019. We will provide another update on November 30.

Attached the audit report covering the ICA discussed in this ticket during the period April 1 2018 to March 31 2019.

A total of eight (8) ICA will be revoked during the key ceremony of December 4 2019. Please find attached an updated overview that reflects this information.

We are further analyzing and defining the timelines of the fourteen (14) remaining ICA and will provide a new update on December 6 2019.

Kathleen: I've tagged this as a Delayed Revocation bug, even though it also captures the BR omission. Am I correct in understanding that this works for you, for bugs like this (i.e. those triggered by ALV)? We have a few more existing bugs that I debated tagging, so wanted to run past you.

Flags: needinfo?(kwilson)
Whiteboard: [ca-compliance] → [ca-compliance] [delayed-revocation-ca]

(In reply to Ryan Sleevi from comment #19)

Kathleen: I've tagged this as a Delayed Revocation bug, even though it also captures the BR omission. Am I correct in understanding that this works for you, for bugs like this (i.e. those triggered by ALV)? We have a few more existing bugs that I debated tagging, so wanted to run past you.

Yes, this works for Mozilla. We are OK with a bug being used for both an initial BR compliance problem (such as BR audit omission) and a delayed revocation (missing the BR revocation deadlines). Also, when the problematic certs are added to OneCRL (so considered from then on to be out of scope of the BRs for Mozilla), I think we should still add "[delayed-revocation-ca] " to the whiteboard.

Flags: needinfo?(kwilson)

We revoked seven (7) ICA on the 4th of November. One of the planned revocations has been postponed to the next key ceremony due to a large subset of leafs not timely being migrated. We are further working with all stakeholders to define a final timing for all ICA, this is taking the time due to the fact that roughly 500 thousand entities have leafs under the remaining ICA. We will provide our timelines before December 20 2019.

(In reply to Arvid Vermote from comment #21)

We revoked seven (7) ICA on the 4th of November. One of the planned revocations has been postponed to the next key ceremony due to a large subset of leafs not timely being migrated. We are further working with all stakeholders to define a final timing for all ICA, this is taking the time due to the fact that roughly 500 thousand entities have leafs under the remaining ICA. We will provide our timelines before December 20 2019.

Correction - we revoked seven (7) ICA on the 4th of December 2019.

Whiteboard: [ca-compliance] [delayed-revocation-ca] → [ca-compliance] [delayed-revocation-ca] - Next Update 20-December 2019

Attached an update on the current status of the different ICA. For the timestamping related ICA we will pursue key destruction as these ICA have been used to issue the GlobalSign-hosted timestamping authority (TSA) certificates. Timestamps used for providing signatures with certain GlobalSign products such as code signing and document/AATL signing rely on the validity of these TSA certificates. If we revoke these ICA any signature time-stamped through this hierarchy will be rendered invalid.

Arvid: Thank you for the update. For the time stamping CAs that GlobalSign [I presume] doesn't want to revoke but has/will instead destroy the private keys, can you provide verification of the destruction?

Whiteboard: [ca-compliance] [delayed-revocation-ca] - Next Update 20-December 2019 → [ca-compliance] [delayed-revocation-ca]

Hi Wayne. Two of the time stamping CAs have been destroyed January 2016 already, and evidences of the destruction ceremony have been captured according to our standard key management procedures. For the two that we plan to destroy March 2020 we will ask a qualified auditor to witness the destruction and create an ISAE3000 report of the witnessed activities. Thank you.

By now we have addressed seventeen (17) out of thirty (30) affected ICA:

Common Name Action Date
GlobalSign Timestamping CA - R3 Key destruction January 26 2016
GlobalSign Timestamping CA Key destruction January 26 2016
SignTrust Domain Verification CA - G2 Revocation November 20 2019
Touchtech Intermediate CA Revocation November 20 2019
SignTrust Domain Verification CA - SHA256 - G2 Revocation November 20 2019
GlobalSign CA for AATL - SHA384 - G4 Audit coverage - created after March 31 2018 November 25 2019
GlobalSign TSA CA for AATL Audit coverage - created after March 31 2018 November 25 2019
GlobalSign Timestamping CA - SHA384 - G4 Audit coverage - created after March 31 2018 November 25 2019
GlobalSign Partners Timestamping CA - SHA384 - G4 Audit coverage - created after March 31 2018 November 25 2019
GlobalSign Partners TSA CA for AATL Audit coverage - created after March 31 2018 November 25 2019
GlobalSign PersonalSign Partners CA - G2 (SHA256 B2A...D0A) Revocation December 4 2019
GlobalSign PersonalSign Partners CA - G2 (SHA256 236...C1A) Revocation December 4 2019
JCAN Public CA0 - G3 Revocation December 4 2019
GlobalSign CA for AATL - SHA256 - G2 (SHA256 3AA...4A1) Revocation December 4 2019
GlobalSign CA 2 for AATL (SHA256 752...441) Revocation December 4 2019
GlobalSign CA 3 for AATL (SHA256 AB6...179) Revocation December 4 2019
JCAN Public CA0 - G3 Revocation December 4 2019

The below table provides an overview of the remediation plans for the thirteen (13) out of thirty (30) affected ICA which have not been addressed yet.

Common Name Action Date Additional information
GlobalSign PersonalSign Partners CA - SHA256 - G2 (SHA256 C8F...5BB) Revocation February 19 2020 All replacement activities have been complete and ICA is ready to be revoked during the next key ceremony (February 19 2020).
GlobalSign Timestamping CA - G2 Key Destruction March 18 2020 Destroy the keys as this ICA has been used to issue the GlobalSign-hosted timestamping authority (TSA) certificates. Timestamps used for providing signatures with certain GlobalSign products such as code signing and document/AATL signing rely on the validity of these TSA certificates. If we revoke this ICA any signature time-stamped through this hierarchy will be rendered invalid.
GlobalSign Timestamping CA - SHA256 - G2 Key Destruction March 18 2020 Destroy the keys as this ICA has been used to issue the GlobalSign-hosted timestamping authority (TSA) certificates. Timestamps used for providing signatures with certain GlobalSign products such as code signing and document/AATL signing rely on the validity of these TSA certificates. If we revoke this ICA any signature time-stamped through this hierarchy will be rendered invalid.
GlobalSign PersonalSign 1 CA - G3 Revocation June 31 2020 The "GlobalSign PersonalSign 1 CA - G3", "GlobalSign PersonalSign 2 CA - G3" and "GlobalSign PersonalSign 3 CA - G3" have combined approximately 60.000 active leaf certificates. This includes certificates issued on hardware tokens.
GlobalSign PersonalSign 2 CA - G3 Revocation June 31 2020 The "GlobalSign PersonalSign 1 CA - G3", "GlobalSign PersonalSign 2 CA - G3" and "GlobalSign PersonalSign 3 CA - G3" have combined approximately 60.000 active leaf certificates. This includes certificates issued on hardware tokens.
GlobalSign PersonalSign 3 CA - G3 Revocation June 31 2020 The "GlobalSign PersonalSign 1 CA - G3", "GlobalSign PersonalSign 2 CA - G3" and "GlobalSign PersonalSign 3 CA - G3" have combined approximately 60.000 active leaf certificates. This includes certificates issued on hardware tokens.
GlobalSign PersonalSign Partners CA - SHA256 - G2 (SHA256 118...B5B) Revocation September 30 2020 This ICA has subordinate ICA chained to it that effectively sign leaf "PersonalSign" certificates. These subordinate ICA are dedicated per customer (but hosted within GlobalSign's environment) to provide machine and client authentication leaf certificates. We expect to complete all these replacement activities by June 30 2020. There is one other subordinate ICA, "NAESB Issuing CA - SHA384 - G3", which is also directly affected by this incident and for which addressing of leaf certificates is estimated to take until September 30 2020 - which is also the date at which we are planning to revoke the subject ICA ("GlobalSign PersonalSign Partners CA - SHA256 - G2").
NAESB Issuing CA - SHA384 - G3 Revocation September 30 2020 25.000 certificates issued to Wholesale Electric participants under the North American Energy Standards Board (NAESB) PKI standard. It will take time for the energy grid to load the replacement ICA in all applications and devices and then replace all leaf certificates.
JCAN Sub Root CA0 Revocation December 31 2020 Has one subordinate ICA left: the "JCAN Public CA1 - G3" also affected by this incident. As soon as that one is actioned we will revoke this ICA.
JCAN Public CA1 - G3 Revocation December 31 2020 This ICA has 40.000 active leaf certificates issued to a variety of medical devices in Japanese hospitals and for some of these devices replacement is not trivial and will require physical maintenance of the devices.
GlobalSign CA 2 for AATL Revocation December 31 2020 The "GlobalSign CA 2 for AATL" and "GlobalSign CA 3 for AATL" have combined approximately 600.000 active leaf certificates. This includes certificates issued on hardware tokens.
GlobalSign CA 3 for AATL Revocation December 31 2020 The "GlobalSign CA 2 for AATL" and "GlobalSign CA 3 for AATL" have combined approximately 600.000 active leaf certificates. This includes certificates issued on hardware tokens.
GlobalSign CA for AATL - SHA256 - G2 (SHA256 AA8...F34) Key Destruction January 20 2021 This ICA has around 100 subordinate certificates in three categories: a) OCSP responder certificates, b) AATL TSA certificates, c) subordinate ICA that effectively sign leaf AATL certificates. Included in c) are the "GlobalSign CA 2 for AATL" and "GlobalSign CA 3 for AATL" which are also impacted by the incident. There are approximately 700.000 active leaf certificates issued by the ICA under c). After the leaf certificates under c) are addressed we will revoke all c) subordinate ICA but not revoke the subject ICA ("GlobalSign CA for AATL - SHA256 - G2") because of the AATL TSA certificates, which have been used to provide Timestamps for document signatures provided by any leaf of the subject ICA. If we revoke the subject ICA any signature time-stamped through this hierarchy will be rendered invalid. The subject ICA keys will hence be destroyed after all subordinate ICAs are revoked.

In the discussion related to Bug 1599788 we further detailed the following initiatives to prevent the issue at hand of occurring in the future.

There are two initiatives we are taking to address both this and Bug 1591005

a) Hierarchy hygiene exercise (to prevent revocation of non-TLS products affecting the speed with which we can take action for TLS ICA / hierarchies) were

  • short term we clean up and address the ICA discussed in Bug 1591005
  • mid-term we will replace any other compliant TLS-capable ICA not effectively issuing TLS leafs with a ICA effectively constrained to the boundaries of the product it issues
  • long-term we plan to have a next generation of roots were WebPKI roots are separated from other uses

b) Rotation of TLS ICA: in the future we are planning to rotate TLS ICA frequently so the amount of leafs that chains to them is contained, which will allow more timely re-issuance of certificates and subsequently timely revocation of their affected ICA.

We are currently finalizing the long-term plan and will share this with the browsers as soon as we have sign-off from the CA Governance policy authority body.

As per the above committed timeline, the GlobalSign PersonalSign Partners CA - SHA256 - G2 (SHA256 C8F...5BB) has been revoked on February 19 2020.

The active keypairs of the GlobalSign Timestamping CA - G2 and GlobalSign Timestamping CA - SHA256 - G2 have been destroyed on the 18th of March 2020. The backup (inactive) copies of those keys are being destroyed in the upcoming weeks. The timing of destruction of one backup copy is still uncertain because of travel limitations due to the Covid-19 situation but is also expected to happen in the upcoming weeks. We will provide an update on this ticket as soon as the destruction of the backup copies is completed.

The destruction ceremony has been witnessed by a qualified auditor, as will the destruction of backup copies. We will share the ISAE3000 report on the key destruction as soon as available.

All backup copies of the GlobalSign Timestamping CA - G2 and GlobalSign Timestamping CA - SHA256 - G2 have been destroyed except a single copy which is a region where the COVID-19 impact is rendering it very difficult to perform a coordinated key destruction activity at the current time. We anticipate to be able to destroy this last copy by June, pending the easing of government imposed COVID-19 restrictions in the subject region.

In accordance to the timeline we are planning for the revocation of GlobalSign PersonalSign 1 CA - G3, GlobalSign PersonalSign 2 CA - G3 and GlobalSign PersonalSign 1 CA - G3 on the 31st of June 2020.

Arvid: I’m not sure Comment #30 provides sufficient detail here to meet the expectations around COVID-19 delays. I think an important concern is understanding more concrete details about the region and the government-imposed restrictions (e.g. references to relevant orders or directives) in order to better assess things.

While we are definitely sensitive to the concerns, there’s an expectation of greater transparency related to such delays as a way to offset the concerns about non-compliance, and help independently understand the decision making process and tradeoffs. Thanks!

Flags: needinfo?(arvid.vermote)

GlobalSign prefers to not publicly disclose the location of this set of key material as apart from the key pairs under discussion this location does also contain other active key pairs. The key pair that has not yet been destroyed is at a back-up key material location: it is an offline, incomplete set of material that cannot be unlocked or used in any way (additional material from other locations would need to be retrieved). It is stored in a bank vault and a collusion of GlobalSign board members is required in order to unlock the vault.

The government recommendation in the specific region is to avoid any non-essential, non-critical travel and maintain the necessary social distance. As per that recommendation, and the fact that we would put certain critical company members together and at increased risk, it is GlobalSign's assessment this outweighs the impact an incomplete, non-active set of key material existing for another few months could have to the WebPKI.

Flags: needinfo?(arvid.vermote)

I’m setting N-I for Kathleen and Ben, to make sure they’re aware of the discussion in Comment #32 as to whether the disclosure meets their expectations. I can think of scenarios where such a response could be used to disguise abuse, and so it’s understandably a risk tradeoff for UAs.

What controls around the activation key material exist such that if this location was already compromised, and such backup key out there, it would not be usable?

Flags: needinfo?(kwilson)
Flags: needinfo?(bwilson)

We do not want CAs publicly disclosing sensitive or confidential information, such as the exact location of private key material.

However, we have said the following in https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay
"Disclose each location (at the state/province level) ..."

Arvid, Is it problematic for you to disclose the state/province of the impacted key material?

Flags: needinfo?(kwilson)
Flags: needinfo?(douglas.beattie)
Flags: needinfo?(bwilson)
Flags: needinfo?(arvid.vermote)

Hi Kathleen - whilst we understand and fully support the need for transparency we are concerned on disclosing the subject location in the current discussion, since the earlier comments (primarily #32) in this ticket disclose the type and use of the facility, and furthermore already hint which GlobalSign employees would have access. I understand this is not the intent of https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay as in there it is specifically mentioned "The public audit statement does not need to identify the type of Facility.". If necessary I can share the regional details directly to Mozilla over e-mail.

We are currently trying to organize the destruction of the last copy of the keys in such a way that it is compliant with health & safety recommendations and does not put our employees at increased risk. According to the current plans and given no unexpected blockers we still expect the destruction of the last copy to happen in the upcoming weeks. I will confirm on this ticket once we have solidified timing or have any other update.

Flags: needinfo?(arvid.vermote)

(In reply to Arvid Vermote from comment #35)

Hi Kathleen - whilst we understand and fully support the need for transparency we are concerned on disclosing the subject location in the current discussion, since the earlier comments (primarily #32) in this ticket disclose the type and use of the facility, and furthermore already hint which GlobalSign employees would have access. I understand this is not the intent of https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay as in there it is specifically mentioned "The public audit statement does not need to identify the type of Facility.". If necessary I can share the regional details directly to Mozilla over e-mail.

Please do not send me sensitive/confidential information via email.

We are currently trying to organize the destruction of the last copy of the keys in such a way that it is compliant with health & safety recommendations and does not put our employees at increased risk. According to the current plans and given no unexpected blockers we still expect the destruction of the last copy to happen in the upcoming weeks. I will confirm on this ticket once we have solidified timing or have any other update.

Thanks for the update.

I don't have further questions.

I'm uncomfortable that the justification the CA is using to not to disclose the locality information (in Comment #35), as required, is that they disclosed other information in Comment #32 that is explicitly not required. While I definitely appreciate the transparency provided here, my concern is that this approach explicitly denies Relying Parties the opportunity to understand the type and nature of the restrictions and why they present a challenge, which is exactly the scenario we've been trying to avoid with these Covid-19 delays. That is, the exact scenario we now find ourselves in was exactly the kind of scenario that a CA could abuse to hide malfeasance, and I'm concerned at how quickly we reached the point of having to blindly trust in the honesty of the CA.

In terms of risk assessments, GlobalSign's responses (e.g. in Comment #35) effectively move us from an attempt to have an objective, risk based analysis that is consistent, to one that largely relies on how confident we are in GlobalSign itself. Again, while I appreciate the transparency in Comment #32, I'm concerned: both overall, and because questions like those in Comment #33 go unanswered.

To the extent it seems I'm picking on GlobalSign, I think it's important to plainly state the problems here, because it's important that any response here not be used as some misguided precedent for other CAs. This is an example of the kind of situation we're trying to avoid here.

We have destroyed the last key pair of the GlobalSign Timestamping CA - G2 and GlobalSign Timestamping CA - SHA256 - G2. The auditor is working on the ISAE3000 report related to the independent witnessing of the key destruction, we will share the report on this ticket as soon as we receive the last version. Meanwhile, in accordance to the timeline we are preparing for the revocation of GlobalSign PersonalSign 1 CA - G3, GlobalSign PersonalSign 2 CA - G3 and GlobalSign PersonalSign 1 CA - G3 on the 31st of June 2020.

As for the outstanding question in Comment 33 we would appreciate further elaboration as we are unsure what information is being asked for.

As per the plan, the "GlobalSign PersonalSign 1 CA - G3", "GlobalSign PersonalSign 2 CA - G3" and "GlobalSign PersonalSign 3 CA - G3" have been revoked on June 30 2020.

I somehow missed Comment #38.

The intent of the question in Comment #31, and broadly, related to "Covid-19" delays for CAs, is to understand the locality information of the delay. While the pandemic is global, the extent of restrictions and limitations vary on many regional/jurisdictional boundaries, typically related to the rate of spread/infection. As discussed during m.d.s.p discussions around Covid-19 delays, the goal is to avoid a situation where a CA claims, due to Covid-19, that a jurisdiction has restrictions when it doesn't actually have them, to avoid having to disclose an incident.

Whiteboard: [ca-compliance] [delayed-revocation-ca] → [ca-compliance] [delayed-revocation-ca] Next Update 1-Oct-2020
QA Contact: wthayer → bwilson

Attached the ISAE3000 report on the destruction of the GlobalSign Timestamping CA - G2 and GlobalSign Timestamping CA - SHA256 - G2 keys.

As per the plan the "GlobalSign PersonalSign Partners CA - SHA256 - G2" and "NAESB Issuing CA - SHA384 - G3" have been revoked on September 30 2020.

Whiteboard: [ca-compliance] [delayed-revocation-ca] Next Update 1-Oct-2020 → [ca-compliance] [delayed-revocation-ca] Next Update 4-Jan-2021

The "JCAN Sub Root CA0", "JCAN Public CA1 - G3", "GlobalSign CA 2 for AATL" and "GlobalSign CA 3 for AATL" have been revoked on December 31 2020.

Are there any remaining CA certificates that need to be revoked?

Flags: needinfo?(arvid.vermote)
Whiteboard: [ca-compliance] [delayed-revocation-ca] Next Update 4-Jan-2021 → [ca-compliance] [delayed-revocation-ca] Next Update 2021-01-15

Hi Ben - all CAs affected have been revoked, but there is one pending key destruction of the "GlobalSign CA for AATL - SHA256 - G2 (SHA256 AA8...F34)", which is planned for January 20 2021 for the copy loaded in issuance-capable environments and the month following January 20 2021 for the backup copies. After that activity we will provide the ISAE3000 independent key destruction witnessing report as soon as released by the auditor. Once that is done all remedial actions will be complete.

Flags: needinfo?(arvid.vermote)
Whiteboard: [ca-compliance] [delayed-revocation-ca] Next Update 2021-01-15 → [ca-compliance] [delayed-revocation-ca] Next Update 2021-02-15

All copies of the "GlobalSign CA for AATL - SHA256 - G2 (SHA256 AA8...F34)" key have been destroyed. Our qualified auditor is working on the ISAE3000 independent key destruction witnessing report, which we will attach to this ticket as soon as it is released by the auditor.

Whiteboard: [ca-compliance] [delayed-revocation-ca] Next Update 2021-02-15 → [ca-compliance] [delayed-revocation-ca] Next Update 2021-03-15

Our auditor informed us that the key destruction report is now undergoing final pre-issuance review and CPA approval.

Please find attached a key destruction cermony witnessing report including destruction of the key pair associated with the CA "GlobalSign CA for AATL - SHA256 - G2 (SHA256 AA8...F34)". This concludes the remedial actions for this issue, unless there are any further questions or concerns we believe this bug can be closed.

Flags: needinfo?(bwilson)

I'll schedule this for closure on or about 31-Mar-2021.

Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.