Closed Bug 1757247 Opened 2 years ago Closed 2 years ago

IdenTrust: Delay Revocation for EV SSL Certificates

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: roots, Assigned: roots)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36

Steps to reproduce:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

In bug https://bugzilla.mozilla.org/show_bug.cgi?id=1753287 we identified 584 active EV TLS certificates requiring revocation within 5 days.
As of 2022-02-25 not all of those certificates have been revoked.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2022-1-26 16:30 MST: Identified 584 active EV TLS certificates issued without disclosing the vetting source.
2022-1-28 9:42 MST: Sent out email communication to affected customer asking to replace/revoke their EV TLS as soon as possible but not later than 2022-02-27.

As of 2022-02-25 we have confirmed that 238 certificates have been revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, see bug https://bugzilla.mozilla.org/show_bug.cgi?id=1753287

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

584 certificates, issued between 2021-1-4 and 2022-1-26

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See attachment in bug https://bugzilla.mozilla.org/show_bug.cgi?id=1753287

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Some affected customers were unable to replace their certificates within the mandated revocation time as they employed manual methods for replacing certificates. We were faced with the decision to revoke mis-issued certificates that posed a severe impact on customers that were unable to replace them within 5 days. We asked the customers to replace them as quickly as possible, expressing urgency, with a replacement deadline of 2022-02-27. As of 2022-02-25, not all of the customers have replaced/revoked all of the mis-issued certificates. We aim to have all identified certificates revoked no later than 2022-02-27.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

As mentioned in bug https://bugzilla.mozilla.org/show_bug.cgi?id=1753287, we:

• Published the list of sources IdenTrust uses to vet EV organizations.
• Published updated Policy documents (CP&CPS Section 3.2.2) referencing the public location of the vetting sources used for EV organizations.
• Modified internal processes to update the publicly available list of sources IdenTrust uses to vet EV organizations and prior to issuance of new EV TLS certificates.

Additionally:
• We are aiming to eliminate certificate mis-issuance by augmenting our internal compliance and reviewing processes with external resources to identify areas that can be improved.
• We offer automation tools for a variety of platforms that have been offered to affected customers at no charge to help reduce the burden and timeline required for certificate replacement. We will continue to press for the adoption of these tools for both existing and new customers.

Assignee: bwilson → roots
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [delayed-revocation-leaf]
Summary: Delay Revocation for EV SSL Certificates → Identrust: Delay Revocation for EV SSL Certificates

The list of steps in step 7 for this bug need to address the causes of the delayed revocation, not the causes of the original issue.

While it is true that avoiding the initial incident would render revocation unnecessary, simply trying harder to avoid misissuance is not sufficient to address the problem of replacing certificates when they need to be replaced, which is unfortunately necessary from time to time due to a variety of reasons.

As of 2022-02-28, 261 certificates have been revoked. We continue to work with affected customers to replace and revoke the remaining certificates. We will provide our next update no later than 2022-03-02.

(In reply to Tim Hollebeek from comment #1)
In order to reduce the complexity, time and effort required to replace certificates, we continually encourage, educate and support customers in implementing automation tools to help reduce the burden and timeline required for certificate replacement. The support and encouragement includes providing automation tools to customers, direct engagement for using those tools, and technical education\guidance.

(In reply to IdenTrust from comment #2)
As of 2022-03-02, 486 certificates have been revoked. We continue to work with affected customers to replace and revoke the remaining certificates. We will provide our next update no later than 2022-03-04.

As of 2022-03-04, 488 certificates have been revoked. We continue to work with affected customers to replace and revoke the remaining certificates. We will provide our next update no later than 2022-03-07.

As of 2022-03-07, we have only one customer pending to replace the remaing certificates. We continue to work with this customer to revoke the remaining certificates. We will provide our next update no later than 2022-03-08.

As of 2022-03-08, we are still working with the last customer to replace/revoke the remaining certificates.
We will provide our next update no later than 2022-03-11.

I appreciate the update, but note that, as of yet, there has been no meaningful prevention plan identified by Identrust relative to this incident.

That is, the answers in Comment #0 are unrelated to this incident, and focus on preventing misissuance. While it's encouraging to think that proper controls can minimize misissuance, it does not however address the issue in this bug, which is the failure to timely revoke once misissuance has occurred. This was highlighted by Comment #1.

Comment #3 did not describe a remediation plan, but simply stated what Identrust does today. However, as noted by Comment #0 and shown by Comment #6, what it does today is insufficient.

Hopefully, prior to considering this issue closed, we will have a clear remediation plan, with timelines, about the concrete changes that Identrust is making to ensure no further delays for revocation in the future. It should show a reasonable relationship to the root causes, which are similarly lacking in detail here (e.g. Comment #6 does not provide details about why this particular customer has delays, nor does Comment #0 address this).

If you compare this report to the expectations at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report , you will see it's clearly deficient and fails to meet the minimum standard. For example, an examination of the requirements show an expectation that:

When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.

We don't know the Subscriber in Comment #6, nor the rationale provided here, so this clearly fails to meet the minimum standard.

Flags: needinfo?(roots)

Out of the 13 customers requiring revocation, we are down to one in the health care sector pending to revoke 96 certificates. This particular customer indicated that they only have a once per month change window in which to make system changes, like replacing a certificate; this is to accommodate a need to provide high availability of systems for patient care activities. As of 2022-03-11, this customer is on track to have the remaining certificates revoked by 2022-3-31.
We will keep updating this bug as new replacement/revocations take place.

In order to prevent timely revocation in the future, we are working closely with this and other customers to adopt our automation tools for certificate replacement.

As of 3-25-2022, we have one customer pending to revoke 87 certificates and is on track to revoke them by month-end. We will provide the next update by 3-31-2022.

As of 3-31-2022 at 19:00 MST all but 3 of the certificates referenced in this bug have been revoked.
Of those 3, the first one listed below is used in conjunction with mobile health applications and is in line to be revoked but the customer has requested two additional weeks to make sure there has been enough time for the replacement certificate to propagate down to the client applications:
https://search.censys.io/certificates/e956df6b76fb3d73500918d923d0c9aa3bf53a8512bbf2626a32ed2a220f4a9f

These two will be revoked no later than this weekend - April 2, 2022:
https://search.censys.io/certificates/4c65b5e793f5805ebbfb2bb81829031eb745fccaf1bf0ed8a0fe974ad44b3a3f
https://search.censys.io/certificates/b4d4487e675523eb64d73021844aaf2ea268c4895ac36bb2a7b47264d6cf35e1

As of 4-21-2022 all certificates on this incident have been revoked.
We have no further pending activities.

I will close this next Wed. 27-April-2022, unless there are further questions or issues to discuss.

Flags: needinfo?(roots) → needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Summary: Identrust: Delay Revocation for EV SSL Certificates → IdenTrust: Delay Revocation for EV SSL Certificates
Whiteboard: [ca-compliance] [delayed-revocation-leaf] → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.