Open Bug 1591005 Opened 2 months ago Updated 6 days ago

GlobalSign: ICAs in CCADB, without EKU extension are listed in WTCA report but not in WTBR report

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: arvid.vermote, Assigned: arvid.vermote, NeedInfo)

Details

(Whiteboard: [ca-compliance] [delayed-revocation-ca] - Next Update 20-December 2019)

Attachments

(6 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36

Steps to reproduce:

On 16-October, when we reviewed the items listed in our CCADB task "Check failed Audit Letter Validation (ALV) results".

Actual results:

We found there are 30 ICA certificates that are technically capable of TLS issuance (due to lack of EKUs) which were not included in our SSLBR and EVSSL audit reports.

Expected results:

The respective ICA certificates should have been included in the SSLBR and EVSSL audit reports (see attached table).

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On 16-October, when we reviewed the items listed in our CCADB task "Check failed Audit Letter Validation (ALV) results", we found there are 30 ICA certificates, that although technically capable of TLS issuance (due to lack of EKUs) are not actually capable of TLS issuance as they are not provided TLS certificate profiles nor TLS workflows in our certificate management system. These ICAs do not issue TLS certificates. These ICAs are older, before GlobalSign commenced using explicit EKU in line with changing industry expectations. These ICAs are listed in the table below.

All these ICAs were disclosed in CCADB, and have always been disclosed in the annual WebTrust for CAs audit, which is conducted concurrently with the WebTrust audits for BRs and EVG. The complete population of GlobalSign issued certificates in the audit period is provided to the WebTrust auditors.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  • 16-October-2019: discover 30 ICA in CCADB without EKU but not intended for TLS issuance are listed in WebTrust audit for CAs audit report, but not listed in WebTrust audit for BR report.
  • 17-October-2019: Performed investigation on EKUs and WebTrust CA scope
  • 18-October-2019: We met with the WebTrust auditors to work out the plan to amend the WTBR report for the 1-April-2018 to 31-March-2019 audit period.
  • 23-October-2019: Further workshops with auditor to revalidate the EE certs issued from these impacted ICAs to confirm none of them are TLS EE certs. We expect the revalidation to be completed by within around 3 weeks.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
N/A - no TLS server certificates have been issued from these CAs.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
N/A - no TLS server certificates have been issued from these CAs.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

GlobalSign did not include these unconstrained CAs in our most recent WTBR report. These CAs were however included in the WebTrust Principles and Criteria for Certification Authorities (WTCA) report. All of them were disclosed in CCADB. These ICAs missing in the WTBR report was an oversight of the intent of the Mozilla policy for covering them in a BR audit even though these ICAs were purposely built not for TLS issuance.
No TLS certificates were issued from these CAs.

Please refer to the file attached to the initial bug report for an overview of the affected ICA.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The mistake was due to our false understanding of the meaning of technical capability of ICAs.

In Mozilla root policy up to v2.4 (Publication date: February 28, 2017), Audit Requirements #2 states,
2. CA operations relating to issuance of certificates capable of being used for SSL-enabled servers must also conform to the latest version of the CA/Browser Forum Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates.

Then Mozilla root policy since v2.4.1 (Publication date: March 31, 2017), 3.1.2 Required Audits states,
3.1.2.1 WebTrust
If being audited to the WebTrust criteria, the following audits are required (see section 3.1.1 for specific version numbers):
•For the SSL trust bit, a CA and all subordinate CAs technically capable of issuing server certificates must have all of the following:
◦WebTrust for CAs
◦WebTrust for CAs - SSL Baseline with Network Security
◦WebTrust for CAs - EV SSL (if applying for EV recognition)
•For the email trust bit, a CA and all subordinate CAs technically capable of issuing email certificates must have all of the following:
◦WebTrust for CAs

While in the Mozilla 'April 2017 CA Communication' survey, ACTION 3: UPDATED MOZILLA CA CERTIFICATE POLICY stated,
Versions 2.4 and 2.4.1 of Mozilla's CA Certificate Policy have been published. Differences between versions 2.4 and 2.3 (published Dec 2016), and between versions 2.4 and 2.2 (published July 2013) may be viewed on Github. Version 2.4.1 contains exactly the same normative requirements as version 2.4 but has been completely reorganized. Please review version 2.4.1 of Mozilla's CA Certificate Policy and confirm that your CA complies with it.

So when we read the 3.1.2.1 WebTrust audit requirements, we thought the trust bit refers to the Mozilla trusted root store setting, and “technically capable of issuing server certificates” still has the same mean as in v2.4, “CA operations relating to issuance of certificates capable of being used for SSL-enabled servers”, and both were referring to issuing system control in all of operational, procedural, technical aspects.

Another factor enforced our wrong interpretation regarding purpose or capability of ICAs, was actual via reading of Microsoft policy. Microsoft policy has been had the separation of purposes quite some time ago. In Sept 2016, before creating a bundle of ICAs, we specifically arranged a meeting with Microsoft root policy team to clarify their policy about the purpose and separation of ICAs. Microsoft advised us that, the conclusion, is pretty simple. You must put EKUs in end entities and you may put them in intermediates. Intermediates can therefore also have AnyEKU (or no EKU) but end entities cannot have this as the risk is a code signing certificate that could be used for SSL (or Visa Versa) if the root is marked for SSL and no EKUs are present in the chain. Old and new roots are really treated the same, the only difference being that new roots are specifically marked for certain EKUs where as old roots were not, so that’s why they originally separated in the root program text to rule 11 and 12.

This reinforced our wrong understanding that the procedure control will also be a deciding factor when looking at the capability of an ICA. Because of this false impression we had, we didn‘t realised that the Mozzila policy ver 2.4.1 audit requirement only based on EKUs, without considering the technical and procedural controls we put in on these ICAs.

We think the new ALV feature provided in CCADB now is very useful to communicate and clarify our wrong understanding about this Mozilla audit expectation.

7. List of steps your CA is taking to resolve the situation and ensure such issuace will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

  • We are now working with our WebTrust auditor to revalidate the EE certs issued from these impacted ICAs to confirm none of them are TLS EE certs. We expect the revalidation to be completed by within around 3 weeks.
  • When this revalidation work is completed, we will work with our WebTrust auditor to amend our most recent WTBR audit to explicitly disclose these impacted ICAs. We will update this disclosure when that is complete, which should be shortly after the completion of revalidation work.
  • Although not reflected in the ALV report of CCADB, we went ahead and further requested the WebTrust auditor to review all the ICAs in the WTBR report and amend the WTEVSSL report for the same period, if any ICAs are technically capable of issuing EV EE certs due to unconstrained policy OIDs.
  • We are in the process of performing a review of all GlobalSign ICAs to ensure that audit scoping is appropriate and have updated our processes to ensure that any future ICAs are named in appropriate audits per Mozilla’s requirements, in accordance with Mozilla Root Program Policy 3.1.2.1 WebTrust.
Assignee: wthayer → arvid.vermote
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance]

I'm trying to reconcile and understand some of the statements here.

Audit Scope

The statement is that they were included within the scope of the WTBR audit, and yet, in terms of remediation, GlobalSign is planning to deliver to their WT auditors the set of certificates they issued. This naturally leads to a conclusion that this infrastructure was not in scope of the WTBR audit, as otherwise, the WT auditor would and should have performed evaluation of controls, at the time of the audit, regarding WTBR compliance.

Put differently: If you have to disclose the certificates you issued to your auditor, in order to establish no TLS issuance, it means the auditor did not, at the time of the WTBR audit, actually examine that no TLS issuance happened.

Changes to Mozilla Requirements

It's extremely helpful and encouraging that GlobalSign reviewed the changes to Mozilla Policy in order to understand where the confusion was introduced. However, what I'm concerned is how limited the the review of the past CA Communications is.

Mozilla communicated expectations several times during that same time period, GlobalSign responded to the November 2017 communication, the April 2017 communication, and the September 2018 communication.

I can understand that, once a systemic misunderstanding was introduced - despite clear communication from Mozilla about the policy expectations as to what "technically capable" meant (and how it was not supported by GlobalSign) - no subsequent reviews identified this significant divergence from expectation.

Proposed Remediations

  • You should discuss with all Root Programs you participate in this proposed remediation. Other root programs have consistently rejected these proposals, due to the lack of assurance they provide, and required that, as required by the Baseline Requirements, these certificates be revoked. Your remediation plan does not provide a clear timeline for revocation, which is the Baseline Requirements expectation.
  • The WebTrust illustrative guidance, published in 2017, has included guidance for how WebTrust auditors can and should disclose the scope of the audit. The omission of this information, while the inclusion of other information, raises serious questions about whether the amended report provides adequate assurance.
  • None of this remediation plan seems to address the systemic issues that lead to the failure to comply with Mozilla Policy, and the subsequent misstatements about compliance made to Mozilla and the community. For example, Mozilla Policy Section 5.3 clearly defines what technical capability is, and so it's unclear how that could be selectively ignored.
    • It would seem that any evaluation would need to go back to at least policy 2.4.1, as it calls into question all of GlobalSign's operations since then, and all new policies adopted.
    • It would also seem to call into question all of GlobalSign's responses to CA communications since then.
Flags: needinfo?(arvid.vermote)

The respective ICA and leaf certificates were provided to the auditor as part of the raw population which the auditor uses to perform tests of effectiveness based on their audit (sampling and) testing methodology. These ICA are managed in the same realm and infrastructure as all other ICA in scope of our audits.

The reason they were omitted from the SSLBR report is because of an error (based on our described misinterpretation of the root policies) in the way we compiled the scope for our assertion in the different SOC3 WebTrust reports, not because these ICA or supporting infrastructure were not subject to audit.

When we discussed the issue at hand with our auditor they stated they wanted to again look at the respective populations from the period under audit to ensure no TLS certificates had been issued from these ICA. This is what we meant when using the word “revalidation”.

Since our initial post we’ve had further internal discussions and apart from, once available, disclosing the rectified reports, we would like to request for all identified ICA to be added to oneCRL. (as they are not used to issue TLS certificates)

Flags: needinfo?(arvid.vermote)

Can you please indicate the timelines that you'll be revoking these certificates?

Flags: needinfo?(arvid.vermote)

Again, can you please provide a timeline for prompt revocation, as the Baseline Requirements require?

Flags: needinfo?(douglas.beattie)

Please find attached an updated overview of the affected ICAs and their current plan of action:

  • Three (3) ICAs will be revoked on November 20 2019
  • Six (6) ICAs have been created after the start of the period-under-audit of our latest cycle (April 1 2018) and will be covered by the updated report as soon as made available by our auditor
  • Twenty-one (21) ICAs are being discussed with the relevant product owners and impacted clients for replacement as, although they are not used to issue TLS certificates, a significant number of leaf certificates (mainly AATL, SMIME and Time-stamping certificates) is chained towards them and immediate revocation will result in a significant operational impact towards our customers

We will provide another update on November 21 2019 with more solid plans for the 21 ICA mentioned above. We expect to do another round of revocation in the last week of November or the first week of December, the exact date of that revocation ceremony will also be shared on November 21 2019.

In the meanwhile we request Mozilla and the other WebPKI root programs to add the affected ICA to OneCRL or equivalent root program revocation mechanisms.

Flags: needinfo?(arvid.vermote)

We will also be submitting a separate incident report for not revoking these ICA within the timeframe required by the Baseline Requirements.

(In reply to Arvid Vermote from comment #6)

In the meanwhile we request Mozilla and the other WebPKI root programs to add the affected ICA to OneCRL or equivalent root program revocation mechanisms.

Arvid, Please clarify which intermediate certs should be added to OneCRL. i.e. Should all of the intermediate certs listed in the attachment to Comment #7 be added to OneCRL?

(In reply to Kathleen Wilson from comment #9)

(In reply to Arvid Vermote from comment #6)

In the meanwhile we request Mozilla and the other WebPKI root programs to add the affected ICA to OneCRL or equivalent root program revocation mechanisms.

Arvid, Please clarify which intermediate certs should be added to OneCRL. i.e. Should all of the intermediate certs listed in the attachment to Comment #7 be added to OneCRL?

Hi Kathleen, yes - all 30 attached intermediate certs should be added to OneCLR. Thanks, Arvid.

(In reply to Arvid Vermote from comment #10)

Hi Kathleen, yes - all 30 attached intermediate certs should be added to OneCLR. Thanks, Arvid.

Arvid, The attached spreadsheet shows the certs that have been set to "Ready to Add" to OneCRL. Please check the list to confirm it is correct.

(In reply to Kathleen Wilson from comment #11)

Created attachment 9110127 [details]
GlobalSign-OneCRL-Request-Nov2019.xls

(In reply to Arvid Vermote from comment #10)

Hi Kathleen, yes - all 30 attached intermediate certs should be added to OneCLR. Thanks, Arvid.

Arvid, The attached spreadsheet shows the certs that have been set to "Ready to Add" to OneCRL. Please check the list to confirm it is correct.

Hi Kathleen, I confirm this spreadsheet is correct. Thank you.

As communicated earlier three (3) ICAs have been revoked on November 20 2019. We will do another batch of revocations on December 4 2019. So far we have confirmed five (5) ICAs to be revoked on December 4 2019 with another potential four (4) given we can timely replace leaf certificates and ensure no impact. Please refer to the newly attached file for the updated status of all affected ICA.

We will provide a next update on November 25 2019.

The audit report should follow later this week and we are further identifying and planning a next batch of revocations on December 4 2019. We will provide another update on November 30.

Attached the audit report covering the ICA discussed in this ticket during the period April 1 2018 to March 31 2019.

A total of eight (8) ICA will be revoked during the key ceremony of December 4 2019. Please find attached an updated overview that reflects this information.

We are further analyzing and defining the timelines of the fourteen (14) remaining ICA and will provide a new update on December 6 2019.

Kathleen: I've tagged this as a Delayed Revocation bug, even though it also captures the BR omission. Am I correct in understanding that this works for you, for bugs like this (i.e. those triggered by ALV)? We have a few more existing bugs that I debated tagging, so wanted to run past you.

Flags: needinfo?(kwilson)
Whiteboard: [ca-compliance] → [ca-compliance] [delayed-revocation-ca]

(In reply to Ryan Sleevi from comment #19)

Kathleen: I've tagged this as a Delayed Revocation bug, even though it also captures the BR omission. Am I correct in understanding that this works for you, for bugs like this (i.e. those triggered by ALV)? We have a few more existing bugs that I debated tagging, so wanted to run past you.

Yes, this works for Mozilla. We are OK with a bug being used for both an initial BR compliance problem (such as BR audit omission) and a delayed revocation (missing the BR revocation deadlines). Also, when the problematic certs are added to OneCRL (so considered from then on to be out of scope of the BRs for Mozilla), I think we should still add "[delayed-revocation-ca] " to the whiteboard.

Flags: needinfo?(kwilson)

We revoked seven (7) ICA on the 4th of November. One of the planned revocations has been postponed to the next key ceremony due to a large subset of leafs not timely being migrated. We are further working with all stakeholders to define a final timing for all ICA, this is taking the time due to the fact that roughly 500 thousand entities have leafs under the remaining ICA. We will provide our timelines before December 20 2019.

(In reply to Arvid Vermote from comment #21)

We revoked seven (7) ICA on the 4th of November. One of the planned revocations has been postponed to the next key ceremony due to a large subset of leafs not timely being migrated. We are further working with all stakeholders to define a final timing for all ICA, this is taking the time due to the fact that roughly 500 thousand entities have leafs under the remaining ICA. We will provide our timelines before December 20 2019.

Correction - we revoked seven (7) ICA on the 4th of December 2019.

Whiteboard: [ca-compliance] [delayed-revocation-ca] → [ca-compliance] [delayed-revocation-ca] - Next Update 20-December 2019
You need to log in before you can comment on or make changes to this bug.