Open Bug 1886532 Opened 1 month ago Updated 5 days ago

Entrust: Delayed revocation of EV TLS certificates with missing cPSuri

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: paul.vanbrouwershaven, Assigned: paul.vanbrouwershaven)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

Preliminary Incident Report

Summary

Entrust has issued EV TLS Certificate with a missing cPSuri as reported in #1883843

All affected certificates should have been revoked within 5 days after we were made aware of the incident.

This incident report purely focusses on the delayed revocation, other updates will be provided in #1883843.

Impact

All certificates affected by the original incident (#1883843) are also affected by this incident.

Timeline

All times are UTC.

2024-03-06:

  • 08:35 Publication of the original preliminary incident (#1883843) report.

2024-03-18:

  • 21:00 Scaling up and initial briefing support teams.
  • 21:40 We stopped issuing miss-issued certificates and fixed the EV certificate profile.
  • 23:00 We made a report available to all customers listing certificates impacted.

2024-03-19:

  • 05:00 All impacted customers have been requested by email that their certificates will be revoked and that they need to replace these as soon as possible.
  • Started to reach out to customer proactively to support them with the certificate reissuance.

Root Cause Analysis

We failed to revoke all affected certificates within 5 days due to the following:

  • We initiated the revocation process to late, because we got confused by the TLS EV profiles changes from SC-62v2 (the profiles ballot) in the TLS Baseline Requirements, see #1883843.
  • We expect further revocation delays, details with customer feedback to follow, initial feedback includes:
    • Uncapable to deal with high volumes of manual re-issuance
    • Lockdown because of end of the fiscal year/quarter
    • Lockdown due to easter holidays

Lessons Learned

What went well

  • The Entrust reporting capabilities allows us to send customers a daily reminder about certificates that are pending revocation.

What didn't go well

Where we got lucky

Action Items

In addition to the action items listed in #1883843 we have identified the following actions items.

Action Item Kind Due Date
Add support for ACME Renewal Information (ARI) in ACME Mitigate 2024-09-30
Work with customers to replace and revoked impacted certificates Mitigate TBC
Improve escalation procedures and documentation Mitigate 2024-05-31

Appendix

Details of affected certificates

See the original incident (#1883843).

Assignee: nobody → paul.vanbrouwershaven
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [leaf-revocation-delay]

We will be providing weekly updates on our progress until this issue is fully remediated.
• We are working with 944 customer accounts to revoke and re-issue 26,668 affected EV certificates. Here is a summary of our progress as of this posting:

o 3,830 of 26,668 certificates have been revoked or expired.
o 6,483 certificates have been re-issued with revocation pending.
o 166 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

• We have experienced significant pushback from customers on the feasibility of revocation timelines. The revocation for customers is in most cases manual and complex, involving multiple internal parties to ensure that the change does not create an adverse impact on web services and back-end processing.

• As a result, we have moved these customers into delayed revocation, particularly where their operations are critical to the web ecosystem (e.g., banks, payment networks, airlines, and government agencies).

• This issue has been prioritized at the highest levels within Entrust. We have hundreds of people across Entrust working on remediation—including our senior leadership as well as teams from Customer Support, Operations, Sales, Legal, Compliance, and Product Management, and we have been working hand in hand with executives at Global 2000 companies who are impacted. Our colleagues are working around the clock to support our customers, meet CA/B Forum expectations, and expedite revocation and re-issuance of affected

Summary: Entrust: Delayed revocation of TLS certificates → Entrust: Delayed revocation of EV TLS certificates with missing cPSuri

Update on the revocation progress:

  • 6,114 certificates have been revoked or expired.
  • 5,495 certificates have been re-issued with revocation pending.
  • 311 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Why is the number of 'certificates have been re-issued with revocation pending' gone down?

Not even 1/3 of 'customers' have taken action yet. This speaks to failure of Entrust, even with 'hundreds of people across Entrust working on remediation'.

We now see new bug of CRL failure.

Mozilla representatives and other rootprogram representatives: Can we begin a discussion about distrust of Entrust, please?
This number bugs, many failures and constant ignoring of guidelines and revoke deadlines.

Why must we continue to trust a failed CA?

In response to Comment #3, while we understand your concerns regarding the incidents involving Entrust, we need to take a balanced approach and not jump immediately to discussions of CA distrust. So far, we believe that Entrust has demonstrated responsiveness, a commitment to rectifying issues promptly, and an effort to attain full compliance with requirements. Therefore, with this approach (of having Entrust report on and remediate deficiencies), we hope to mitigate overall risk to the ecosystem. If there are more severe issues discovered in the future, then we would re-assess the situation and consider our available options, which might include removal from the root store. However, at this time, no such action is being considered.

So far, we believe that Entrust has demonstrated responsiveness, a commitment to rectifying issues promptly, and an effort to attain full compliance with requirements.

Is that before or after they were fighting with the community and kept knowingly misissuing certificates?

I would agree with you if these were simply mistakes. Every CA makes mistakes and it’s expected that when they’re called out on it, they rectify it. But the reaction entrust had in the intial cpsuri incident is not at all what I’d expect from a CA that “ has demonstrated responsiveness, a commitment to rectifying issues promptly, and an effort to attain full compliance with requirements”.

Can you please let the community know how does Mozilla reconcile the willful misissuance by Entrust with these statements? What is the line that a CA can’t cross?

Note, I’m not arguing for distrust here. But I do think that the statement you’ve made is somewhat reductive of the situation we’ve seen.

Thanks, Amir. Your comment #5 is noted.

Update on the revocation progress:

  • 7,499 certificates have been revoked or expired.
  • 5,361 certificates have been re-issued with revocation pending.
  • 401 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

We just realized that we used the title "Preliminary Incident Report" instead of "Incident Report", we have no other information to provide besides our weekly progress on revocation.

@Ben, can you change the title in comment #0?

I can only edit my own posts/comments. Your requested correction has been noted.

Update on the revocation progress:

  • 8,572 certificates have been revoked or expired.
  • 5,232 certificates have been re-issued with revocation pending.
  • 481 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Update on the revocation progress:

  • 10,013 certificates have been revoked or expired.
  • 4,883 certificates have been re-issued with revocation pending.
  • 558 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Update on the revocation progress:

13,053 certificates have been revoked or expired.
2,610 certificates have been re-issued with revocation pending.
731 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

You need to log in before you can comment on or make changes to this bug.