1598807 - IdenTrust: Undisclosed Unrevoked ICAs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

IdenTrust: On 11/8/2019 via feedback supplied in bug 1555231 (https://bugzilla.mozilla.org/show_bug.cgi?id=1588213) involving these 2 ICA’s, 12716314 (https://crt.sh/?id=12716314) and 12716094 (https://crt.sh/?id=12716094), we were made aware that these 2 ICA’s require revocation due to lack of disclosure in the annual WebTrust audit report.

IdenTrust has been aware since their creation that 2 subordinate CA’s (ICAs), 12716314 (https://crt.sh/?id=12716314) and 12716094 (https://crt.sh/?id=12716094) , were not technically constrained. In 2016, we evaluated these ICAs and the corresponding ecosystem (please review response to #3 for description of the ecosystem) for suitability to obtain a WebTrust audit. We determined that it was not possible to conduct WebTrust audit of relevant players in the ecosystem (our customer and their customers).

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   IdenTrust:

2016: Because these ICAs were not technically constrained, we established an Agreed Upon Procedures (AUP) audit for these ICA’s that was agreed upon with Mozilla. We included in the AUP audit procedures such as: (i) Additional Subordinate CAs are not permissible via path length constraint set to ‘0’; (ii) Prevent SSL/TLS certificate issuance via programmatic control; and (iii) Non-existence of SSL/TLS certificate within the population of issued certificates. We have audited the ICAs annually in accordance with AUP and have disclosed those audit reports to Mozilla.
11/08/2019: As a result of the concerns documented in bug 1555231 https://bugzilla.mozilla.org/show_bug.cgi?id=1588213), we agreed with Mozilla to place these ICAs on OneCRL and we began to investigate the audit reporting and disclosure issues for these ICAs.
11/15/2019: On this date we determined we needed to revoke these 2 ICAs and we began coordination with parties that will be impacted.
11/18/2019: Supplied to the Google-Chrome Root Authority Program a copy of the AUP report that was shared with Mozilla. The AUP report is available to other Root Store programs upon email request to [e-mail].
11/19/2019: Completed the operational impact and formulated a remediation/revocation plan.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   IdenTrust: We have not stopped issuance of end-entity certificates from these 2 ICAs due to the following:

The IdenTrust Customer using these ICAs have a global ecosystem consisting of their customer base with hundreds of thousands machine-level endpoints that have used several million end entity certificates to provide device level authentication and confidentiality. This ecosystem has been in place for over 20 years and was migrated to a certificate-based authentication security model in 2003. Though, the certificate-based ecosystem pre-dates the existence of the baseline requirements, it should be noted that this is a closed ecosystem with compensating controls and recurring audits.

The ICAs that serve as the foundation of trust for this ecosystem have never and will not in future issue SSL/TLS certificates. Programmatic controls on the ICAs do not allow the issuance of SSL/TLS certificates nor do the endpoints use certificates for SSL/TLS connections. Annual audits are conducted according to Agreed Upon Procedures (AUP) ensuring compliance required for IdenTrust Customer’s ecosystem. If requested by relevant Root Programs, the 2019 AUP audit report will be provided directly to the requesting Root Program.

In addition, each client endpoint is a hardware secure cryptographic device (SCD) manufactured under full IdenTrust Customer’s control. End entity certificates are provisioned into each SCD within a secure manufacturing environment. SCDs comply with FIPS 140-2 level 2 or higher standards to protect private keys and their usage. SCD manufacturing, provisioning and life cycle maintenance to include SCD destruction, is subject to independent external bi-annual audits. The customers within the ecosystem generate back-end certificates with private keys also protected within a FIPS 140-2 level 2 or higher standards compliant hardware crypto-module. Certificate enrollment for the back-end is constrained to strict multi-party Identification & Authentication (I&A) procedures which are controlled by the customer.

In an effort to mitigate any unlikely risk, the customer will establish a new, private infrastructure and migrate the ecosystem from the current public root of trust. A global migration strategy must be executed in accordance with best security practices. Due to the widespread deployment of the SCDs and the requirement for high availability and reliability, each environment must undergo thorough integration and testing. This will require a significant amount of time. As such, IdenTrust proposes the following risk mitigation plan and revocation timeline:

1. IdenTrust have requested Chrome Root Authority Program and Microsoft Root Program to place both ICAs on to the relevant revocation lists similar to OneCRL’s. Please note we have also jointly agreed to place both of these ICAs on Mozilla’s OneCRL.

2. Availability upon request of the 2019 and prior years AUP report to other relevant Root Programs.

3. Establishment of alternate, private infrastructure recycling the same private keys as both ICAs subordinated under a Private Root CA.
   i. Infrastructure creation Dec 1, 2019
   ii. Production setup & manufacturing qualification – Dec 31, 2019
   iii. Qualify, Deploy into new environments – March 31, 2020
   iv. Qualify, Deploy into installed base – May 30, 2020

4. Revoke both ICAs from current Public Trust infrastructure – May 30, 2020.

5. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   IdenTrust: 2 Subordinate ICA certificates issued on 2/5/2002.

6. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   IdenTrust: 12716314 (https://crt.sh/?id=12716314); 12716094 (https://crt.sh/?id=12716094)

7. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   IdenTrust: These 2 ICAs have been issuing device certificates since February 2002, which precedes Baseline Requirements. As Baseline Requirements were established IdenTrust realized that this PKI does not meet the requirements of B.R., but that there are mitigation measures that could be independently audited to demonstrate controls are maintained and that controls ensure these ICA’s are not issuing SSL/TLS certificates and have never issued SSL/TLS certificates. In 2016, an agreement was reached between IdenTrust, the IdenTrust Customer and the Mozilla Root Store to start supplying a separate annual AUP Report. We have continued to do so every year since.

In addition, since 2016 we have reported these ICAs as covered within the same scope as the parent. The parent is audited to WebTrust and BR standard. Most, but not all, of these control objectives are also applicable and audited for these ICAs. We included in the AUP audit additional procedures including: (i) Additional Subordinate CAs are not permissible via path length constraint set to ‘0’; (ii) Prevent SSL/TLS certificate issuance via programmatic control; and (iii) Non-existence of SSL/TLS certificate within the population of issued certificates.

We incorrectly believed the disclosure of the AUP report to Mozilla and disclosure of coverage under parent’s audit satisfied needed disclosure requirements. As such, we did not include these ICAs in our management assertion to our auditors. In hindsight, we should have approached all relevant Root Programs in 2016 to validate our disclosure approach.

IdenTrust was made aware of concerns with this approach in October 2019 as documented in bug 1555231 (https://bugzilla.mozilla.org/show_bug.cgi?id=1588213).

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
   IdenTrust: As outlined in item 3, IdenTrust will revoke both ICA’s by May 30, 2020 and is requesting relevant Root Store programs to place them immediately on corresponding Revocation or Disallowed list.

Outside of these ICAs and since adoption of Baseline Requirements, IdenTrust subjects all existing Root CAs and ICAs to Baseline Requirements and WebTrust audit annually (as required by BR using external WebTrust auditors) if they are distributed in Public Root Store(s) and are either unconstrained or are issuing SSL/TLS certificates. Also going forward IdenTrust has established procedural controls that disallows creation of unconstrained ICA from a publicly distributed Root CA without multiple levels of management approvals and approval of our Risk Management Committee. If an unconstrained ICA is approved and issued, it will be added to our audit engagement scope, audited and reported in accordance with Baseline Requirements.

Comment 1

[IdenTrust]

Please note that numerals 5, 6, and 7 are really numerals 4, 5, and 6.

Comment 2

[Nick Lamb (tialaramex)]

Note that above description mentions Bug 1588231 but it means (and correctly links immediately afterwards) Bug 1588213. I spent a few minutes wondering how this closed bug was somehow related, it isn't.

Comment 3

[Ryan Sleevi]

Can you confirm that the following:

i. Infrastructure creation Dec 1, 2019

Was successful?

I've set Next-Update assuming it is, and that an update will be provided for:

ii. Production setup & manufacturing qualification – Dec 31, 2019

Comment 4

[IdenTrust]

Related to Comment 2, we confirm that the bug # listed in the description posted should have been Bug 1588213 and apologize for any confusion this caused.

Related to Comment 3, we are confirming we have established the private infrastructure. We will provide next update on or prior to Dec 31, 2019.

i. Infrastructure creation Dec 1, 2019
Status: Completed

ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: In-Process

Comment 5

[IdenTrust]

We confirm completion of production setup and manufacturing qualification. We will provide next update on or prior to March 31, 2020.
i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed
iii. Qualify, Deploy into new environments – March 31, 2020
Status: In-Process

Comment 6

[Ryan Sleevi]

Thanks. I've set the Next-Update to be 31-Jan 2020. Let's have monthly checkins to make sure everything is progressing and on-track, which may be as simple as "In-Progress". Basically, if anything would cause delays, then please provide an update as soon as that is known.

Comment 7

[IdenTrust]

(In reply to Ryan Sleevi from comment #6)

Thanks. I've set the Next-Update to be 31-Jan 2020. Let's have monthly checkins to make sure everything is progressing and on-track, which may be as simple as "In-Progress". Basically, if anything would cause delays, then please provide an update as soon as that is known.

We'll do. Thanks

Comment 8

[IdenTrust]

As requested, we are providing monthly progress update. We are progressing and on track to complete our next milestone (as listed below) by March 31, 2020. We will provide next update on or prior to February 28, 2020.

i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed
iii. Qualify, Deploy into new environments – March 31, 2020
Status: In-Process

Comment 9

[IdenTrust]

As requested, we are providing monthly progress update. We are progressing and on track to complete our next milestone (as listed below) by March 31, 2020. We will provide next update on or prior to March 31, 2020.

i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed
iii. Qualify, Deploy into new environments – March 31, 2020
Status: Partially behind schedule due to material impact of corona virus; we have launched mitigating activities and are assessing impact. We will provide update at the next monthly progress report March 31, 2020.

Comment 10

[IdenTrust]

As requested, we are providing monthly progress update. We are behind schedule as shown below.
i. Establishment of alternate, private infrastructure with two new ICAs subordinated under a Private Root CA.
i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed.
iii. Qualify, Deploy into new environments – Jun 30, 2020 (was March 31, 2020)
Status: We are behind schedule. Though we created the infrastructure, qualified the manufacturing environment and prepared for deployment, the corona virus is inhibiting global deployment into new environments. Therefore, we will need to revise our forecasted completion of this milestone to Jun 30, 2020.
iv. Qualify, Deploy into installed base – Aug 30, 2020 (was May 30, 2020)
Status: Due to the Milestone iii. delay, we will be unable to qualify and deploy into the installed base by May 30, 2020 and will need to revise our forecasted completion of this milestone to Aug 30, 2020.
2. Revoke two current ICAs from Public Trust infrastructure – Aug 30, 2020 (was May 30, 2020)
Status: Due to the Milestone iii. delay, we will be unable to revoke both ICAs by May 30, 2020 and will need to revise our forecasted completion of this milestone to Aug 30, 2020.
We will provide next update on or prior to April 30, 2020.

Comment 11

[Ryan Sleevi]

Thanks for the update. I've tagged this with [covid-19], related to the discussion on mozilla.dev.security.policy. This seems to fit into the scenario discussed regarding access to the data centers.

Comment 12

[IdenTrust]

(In reply to Ryan Sleevi from comment #11)
We would like to clarify that the COVID-19 related delays lies with our customer being unable to proceed as planned due to supply chain disruptions and social distance protocols limiting qualification and deployment activities which are critical to transition their eco-system (partners, customers, manufacturing sites, etc..) to the private trust infrastructure (new Root CA and associated ICAs) – which must happen prior to revocation of the 2 ICAs, 12716314 (https://crt.sh/?id=12716314) and 12716094 (https://crt.sh/?id=12716094).

If we believe we(IdenTrust) have material risk to our annual audits or accessing our operational sites to perform routine tasks such as CA revocations, we will identify in a separate incident report.

Comment 13

[Ryan Sleevi]

Thanks for the clarification! That's very helpful!

We've tagged other issues where the customer-impact due to covid-19 has lead to the delay (e.g. Bug 1624658), so I think it's still a useful data point, since we are trying to be sensitive of the global nature of the pandemic.

The tag is largely to help our understanding and tracking, as well as to look back, systemically, once the immediate crisis is over, to examine what can be done to ensure the system is more robust in the face of future incidents.

Comment 14

[IdenTrust]

As requested, we are providing monthly progress update. We are behind schedule as shown below.
i. Establishment of alternate, private infrastructure with two new ICAs subordinated under a Private Root CA.
i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed.
iii. Qualify, Deploy into new environments – Jun 30, 2020
Status: On track to complete by Jun 30, 2020.
iv. Qualify, Deploy into installed base – Aug 30, 2020
Status: On track to complete by Aug 30, 2020.
2. Revoke two current ICAs from Public Trust infrastructure – Aug 30, 2020
Status: On track to complete by Aug 30, 2020.

Comment 15

[IdenTrust]

As requested, we are providing monthly progress update. We are behind schedule as shown below.
i. Establishment of alternate, private infrastructure with two new ICAs subordinated under a Private Root CA.
i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed.
iii. Qualify, Deploy into new environments – Jun 30, 2020
Status: On track to complete by Jun 30, 2020.
iv. Qualify, Deploy into installed base – Aug 30, 2020
Status: On track to complete by Aug 30, 2020.
2. Revoke two current ICAs from Public Trust infrastructure – Aug 30, 2020

Comment 16

[IdenTrust]

As requested, we are providing monthly progress update. We are behind schedule as shown below.
i. Establishment of alternate, private infrastructure with two new ICAs subordinated under a Private Root CA.
i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed.
iii. Qualify, Deploy into new environments – Jun 30, 2020
Status: Qualification complete, deployment behind schedule due to COVID-19 – to be complete July 31, 2020
iv. Qualify, Deploy into installed base – Aug 30, 2020
Status: On track to complete by Aug 30, 2020.
2. Revoke two current ICAs from Public Trust infrastructure – Aug 30, 2020
Status: On track to complete by Aug 30, 2020.

Comment 17

[IdenTrust]

As requested, we are providing monthly progress update. We are behind schedule as shown below.
i. Establishment of alternate, private infrastructure with two new ICAs subordinated under a Private Root CA.
i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed.
iii. Qualify, Deploy into new environments – July 31, 2020
Status: completed
iv. Qualify, Deploy into installed base – Aug 30, 2020
Status: On track to complete by Aug 30, 2020.
2. Revoke two current ICAs from Public Trust infrastructure – Aug 30, 2020
Status: On track to complete by Aug 30, 2020.

Comment 18

[Ben Wilson]

Can you provide a status update on this bug? Thanks.

Comment 19

[IdenTrust]

(In reply to Ben Wilson from comment #18)

Can you provide a status update on this bug? Thanks.
We are on track with the final steps of the communicated plan\timeline. We will provide a final status no later than August 31.

Comment 20

[IdenTrust]

This is our final status report as all items of our plan have been completed.
i. Establishment of alternate, private infrastructure with two new ICAs subordinated under a Private Root CA.
i. Infrastructure creation Dec 1, 2019
Status: Completed
ii. Production setup & manufacturing qualification – Dec 31, 2019
Status: Completed.
iii. Qualify, Deploy into new environments – July 31, 2020
Status: Completed
iv. Qualify, Deploy into installed base – Aug 30, 2020
Status: Completed.
2. Revoke two current ICAs from Public Trust infrastructure – Aug 30, 2020
Status: Completed.

Comment 21

[Ben Wilson]

I intend to close this bug on Monday, 7-Sept-2020, unless there are objections.