1886532 - Entrust: Delayed revocation of EV TLS certificates with missing cPSuri

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Paul van Brouwershaven]

Preliminary Incident Report

Summary

Entrust has issued EV TLS Certificate with a missing cPSuri as reported in #1883843

All affected certificates should have been revoked within 5 days after we were made aware of the incident.

This incident report purely focusses on the delayed revocation, other updates will be provided in #1883843.

Impact

All certificates affected by the original incident (#1883843) are also affected by this incident.

Timeline

All times are UTC.

2024-03-06:

• 08:35 Publication of the original preliminary incident (#1883843) report.

2024-03-18:

• 21:00 Scaling up and initial briefing support teams.
• 21:40 We stopped issuing miss-issued certificates and fixed the EV certificate profile.
• 23:00 We made a report available to all customers listing certificates impacted.

2024-03-19:

• 05:00 All impacted customers have been requested by email that their certificates will be revoked and that they need to replace these as soon as possible.
• Started to reach out to customer proactively to support them with the certificate reissuance.

Root Cause Analysis

We failed to revoke all affected certificates within 5 days due to the following:

• We initiated the revocation process to late, because we got confused by the TLS EV profiles changes from SC-62v2 (the profiles ballot) in the TLS Baseline Requirements, see #1883843.
• We expect further revocation delays, details with customer feedback to follow, initial feedback includes:
  • Uncapable to deal with high volumes of manual re-issuance
  • Lockdown because of end of the fiscal year/quarter
  • Lockdown due to easter holidays

Lessons Learned

What went well

• The Entrust reporting capabilities allows us to send customers a daily reminder about certificates that are pending revocation.

What didn't go well

Where we got lucky

Action Items

In addition to the action items listed in #1883843 we have identified the following actions items.

Action Item	Kind	Due Date
Add support for ACME Renewal Information (ARI) in ACME	Mitigate	2024-09-30
Work with customers to replace and revoked impacted certificates	Mitigate	TBC
Improve escalation procedures and documentation	Mitigate	2024-05-31

Appendix

Details of affected certificates

See the original incident (#1883843).

Comment 1

[Paul van Brouwershaven]

We will be providing weekly updates on our progress until this issue is fully remediated.
• We are working with 944 customer accounts to revoke and re-issue 26,668 affected EV certificates. Here is a summary of our progress as of this posting:

o 3,830 of 26,668 certificates have been revoked or expired.
o 6,483 certificates have been re-issued with revocation pending.
o 166 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

• We have experienced significant pushback from customers on the feasibility of revocation timelines. The revocation for customers is in most cases manual and complex, involving multiple internal parties to ensure that the change does not create an adverse impact on web services and back-end processing.

• As a result, we have moved these customers into delayed revocation, particularly where their operations are critical to the web ecosystem (e.g., banks, payment networks, airlines, and government agencies).

• This issue has been prioritized at the highest levels within Entrust. We have hundreds of people across Entrust working on remediation—including our senior leadership as well as teams from Customer Support, Operations, Sales, Legal, Compliance, and Product Management, and we have been working hand in hand with executives at Global 2000 companies who are impacted. Our colleagues are working around the clock to support our customers, meet CA/B Forum expectations, and expedite revocation and re-issuance of affected

Comment 2

[Paul van Brouwershaven]

Update on the revocation progress:

• 6,114 certificates have been revoked or expired.
• 5,495 certificates have been re-issued with revocation pending.
• 311 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Comment 3

[JR Moir]

Why is the number of 'certificates have been re-issued with revocation pending' gone down?

Not even 1/3 of 'customers' have taken action yet. This speaks to failure of Entrust, even with 'hundreds of people across Entrust working on remediation'.

We now see new bug of CRL failure.

Mozilla representatives and other rootprogram representatives: Can we begin a discussion about distrust of Entrust, please?
This number bugs, many failures and constant ignoring of guidelines and revoke deadlines.

Why must we continue to trust a failed CA?

Comment 4

[Ben Wilson]

In response to Comment #3, while we understand your concerns regarding the incidents involving Entrust, we need to take a balanced approach and not jump immediately to discussions of CA distrust. So far, we believe that Entrust has demonstrated responsiveness, a commitment to rectifying issues promptly, and an effort to attain full compliance with requirements. Therefore, with this approach (of having Entrust report on and remediate deficiencies), we hope to mitigate overall risk to the ecosystem. If there are more severe issues discovered in the future, then we would re-assess the situation and consider our available options, which might include removal from the root store. However, at this time, no such action is being considered.

Comment 5

[amir]

So far, we believe that Entrust has demonstrated responsiveness, a commitment to rectifying issues promptly, and an effort to attain full compliance with requirements.

Is that before or after they were fighting with the community and kept knowingly misissuing certificates?

I would agree with you if these were simply mistakes. Every CA makes mistakes and it’s expected that when they’re called out on it, they rectify it. But the reaction entrust had in the intial cpsuri incident is not at all what I’d expect from a CA that “ has demonstrated responsiveness, a commitment to rectifying issues promptly, and an effort to attain full compliance with requirements”.

Can you please let the community know how does Mozilla reconcile the willful misissuance by Entrust with these statements? What is the line that a CA can’t cross?

Note, I’m not arguing for distrust here. But I do think that the statement you’ve made is somewhat reductive of the situation we’ve seen.

Comment 6

[Ben Wilson]

Thanks, Amir. Your comment #5 is noted.

Comment 7

[Paul van Brouwershaven]

Update on the revocation progress:

• 7,499 certificates have been revoked or expired.
• 5,361 certificates have been re-issued with revocation pending.
• 401 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Comment 8

[Paul van Brouwershaven]

We just realized that we used the title "Preliminary Incident Report" instead of "Incident Report", we have no other information to provide besides our weekly progress on revocation.

@Ben, can you change the title in comment #0?

Comment 9

[Ben Wilson]

I can only edit my own posts/comments. Your requested correction has been noted.

Comment 10

[Paul van Brouwershaven]

Update on the revocation progress:

• 8,572 certificates have been revoked or expired.
• 5,232 certificates have been re-issued with revocation pending.
• 481 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Comment 11

[Paul van Brouwershaven]

Update on the revocation progress:

• 10,013 certificates have been revoked or expired.
• 4,883 certificates have been re-issued with revocation pending.
• 558 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Comment 12

[Bruce Morton]

Update on the revocation progress:

13,053 certificates have been revoked or expired.
2,610 certificates have been re-issued with revocation pending.
731 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Comment 13

[Paul van Brouwershaven]

Update on the revocation progress:

• 14,736 certificates have been revoked or expired.
• 2,130 certificates have been re-issued with revocation pending.
• 866 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Comment 14

[Wayne]

By my count at 14,736 certificates of 26,668 affected certificates we're at 55.26% revoked after 44 days. That is ... not good.

Do we have a detailed breakdown of affected subscribers and when revocation will actually occur?

What barriers exist for adding support for ACME Renewal Information before 2024-09-30? Let's Encrypt had a good article of Tailscale deploying in less than 2 days.

In total, it took just two Tailscale engineers less than two days to implement ARI.

Comment 15

[Paul van Brouwershaven]

(In reply to Wayne from comment #14)

By my count at 14,736 certificates of 26,668 affected certificates we're at 55.26% revoked after 44 days. That is ... not good.

Do we have a detailed breakdown of affected subscribers and when revocation will actually occur?

Thanks Wayne, we do plan to add this breakdown and will provide this information next week.

What barriers exist for adding support for ACME Renewal Information before 2024-09-30? Let's Encrypt had a good article of Tailscale deploying in less than 2 days.

In total, it took just two Tailscale engineers less than two days to implement ARI.

We have to implement ACME ARI into our CA platform and ensure its well tested.

Comment 16

[Paul van Brouwershaven]

Update on the revocation progress:

• 15,440 certificates have been revoked or expired.
• 2,000 certificates have been re-issued with revocation pending.
• 879 out of 944 customer accounts have fully remediated the issue (certificates re-issued and old certificates revoked).

Comment 17

[Bruce Morton]

We are working diligently with our customers to complete revocation of affected certificates. Over 95% of customers have completed revocation. We have 9,906 certificates remaining within the following four (4) industries: Financial Institutions (6,940 certificates), Government Agencies (170 certificates), Information Technology (46 certificates), and Travel (Airline) (2,750 certificates). All are scheduled to be revoked within the next 20 days with limited exceptions as required.

Comment 18

[Wayne]

Please provide a per-subscriber breakdown as required, that you're proposing 'limited exceptions' to revocation after 20 days when we are 58 days in says it all. When will each certificate expire naturally, and when will revocation, at the absolute latest, occur per-certificate.

Entrust shouldn't need to be reminded daily to re-read Mozilla's Root Program 'Responding To An Incident' page as if this is the first time they've been told.

When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.
Any decision to not comply with the timeline specified in the Baseline Requirements must also be accompanied by a clear timeline describing if and when the problematic certificates will be revoked or expire naturally, and supported by the rationale to delay revocation.

We are walking into wilful non-compliance in this aspect too.

Comment 19

[Wayne]

I've had multiple of Entrust's subscribers privately contact me on their own accord so I'll be very specific in my questioning additional to the above.

Can Entrust give a step by step process of how precisely these extensions you are providing are approved? To be clearer:

• During the cPSuri incident what was the contents of the email sent to providers asking for revocation to begin with?
• The Incident Report mentions contact on 03-19 requesting revocation. What mechanisms do Entrust use to check that a newly issued certificate is now in active use and that the older one can be safely revoked?
• What is the followup process is the subscriber has generated a new certificate and has received it, but has not explicitly said it is safe to revoke?
• Following whichever timeframe of additional time has been approved by the above means, what is the process for following up with subscribers and who provides suggestions on possible timeframes?
• Are there specific risk categories provided to subscribers to choose from, or do you get a freeform text reply to make a calculated risk analysis?
• Likewise are there specific choices of timeframes to postpone revocation? Do this depend on precise categories of subscribers?
• Given either answer above they must be generalized for internal analysis, so can Entrust provide a breakdown as it should be in their documentation on-hand and require no effort to produce?
• Were a subscriber to miss the 5-day window, get an additional extension (14d, 28d, whichever) and then miss that... what is the communication sent from Entrust?
• On the process side then, is it a more detailed process or is it the same process outlined in a previous answer here? If so, which answer?
• Has this been the same process from the start of the incident until now? If not, what lessons were learned and applied?

I've kept my questions brief and simple. I'm sure all CAs to agree this is a good learning ground on what should be said to subscribers, and how best to communicate in this scenario. If we have examples of what precisely Entrust is saying to their subscribers we can try and flesh it out into a generalized incident response format to help all CAs going forward.

Comment 20

[Tim Callan]

(In reply to Bruce Morton from comment #17)

Bruce,

After the recent publication of revoked versus expired numbers in bug 1887705 comment #24, I had thought it meant Entrust intended to report its delayed revocation events this way moving forward. I was surprised subsequently to see this post, without that breakdown.

Does Entrust intend to distinguish revoked from naturally expired certificates in its delayed revocation and no-revocation incidents? When should we expect to see these numbers for this bug, bug 1890685, and bug 1890898? Please note also my comment in bug 1887705 comment 27 clarifying that the important distinction is between certs that expired versus those revoked, as opposed to when they would have expired had they not been revoked.

Comment 21

[ngook.kong]

Here is our progress:

• 18,088 of 26,668 (67.8%) certificates have been revoked or expired.
• 17,894 revoked.
• 194 expired.
• 1,449 certificates have been re-issued with revocation pending.
• 910 out of 944 customer accounts (96.4%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Attached is the information per customer, we are working with each subscriber to accelerate the time to revocation, of the remaining 8580 certificates.

Comment 22

[ngook.kong]

Attachment: EV Delayed Revocation 2024.05.21.xlsx

Comment 23

[JR Moir]

I think the deliberate hideing of who the customers are is not good.
The 30 biggest customers from list:

JPMorgan Chase and Co.
Delta Air Lines, Inc.
Bank of America Corporation
Fidelity Investments (FMR LLC)
Westpac Banking Corporation
American Airlines Inc
DBS Bank Ltd
The Toronto-Dominion Bank
IDFC First Bank Limited
Experian Information Solutions, Inc.
Experian South Africa (Pty) Ltd
St.George Bank (Westpac Banking Corporation)
Insurance Australia Group Limited
Bank of Montreal
BT Financial Group Pty Limited
National Australia Bank Limited
U.S. Securities and Exchange Commission
Bank Of Melbourne (Westpac Banking Corporation)
Experian Limited
Pole Emploi
BankSA (Westpac Banking Corporation)
FMR LLC
Reserve Bank of Australia
Experian Colombia S.A
DFAT (Department of Foreign Affairs and Trade)
Global Payments Inc.
Ministry of Manpower
Asgard Wealth Solutions Ltd
Advanced Info Service Public Company Limited
The Westpac Group (Westpac Banking Corporation)

Comment 24

[Wayne]

Awaiting a response to my previous questions. In the meantime let's break down the above. The original 'reason's are badly formatted and have redundant options so here's a legible version:

Reason	23/05/24	24/05/24	30/05/24	02/06/24	Total
Harm, Critical	-	14	30	-	44
Harm, Critical, Manual, Teams	-	-	3146	2798	5944
Harm, Critical, Teams	43	8	2541	-	2592
Total	43	22	5717	2798	8580

Then breaking it down by subscriber via organizationName:

organizationName	No. of Certs
Advanced Info Service Public Company Limited	14
American Airlines Inc	370
Bank of America Corporation	1446
Bank of Montreal	43
Bank of the West	4
Board of Investments	1
Commonwealth Bank of Australia	3
DBS Bank Ltd	267
Delta Air Lines, Inc.	1684
DFAT (Department of Foreign Affairs and Trade)	16
Experian Information Solutions, Inc.	203
Experian PLC	4
Fidelity Investments (FMR LLC)	509
First Abu Dhabi Bank PJSC	7
Global Payments Inc.	15
IDFC First Bank Limited	100
Inland Revenue Authority of Singapore	8
Insurance Australia Group Limited	40
JPMorgan Chase and Co.	2798
M&G FA Limited	4
M&T Bank Corporation	5
MasterCard International Incorporated	2
Ministry of Manpower	12
National Australia Bank Limited	42
Philippine National Bank	1
Pole Emploi	30
Reserve Bank of Australia	19
Sampath Bank PLC	2
The Saudi National Bank	6
The Toronto-Dominion Bank	243
The Westpac Group (Westpac Banking Corporation)	10
U.S. Securities and Exchange Commission	31
Westpac Banking Corporation	641

Are your subscribers specifying what they mean by critical infrastructure? If it is truly critical then are you helping them move to a private root and away from WebPKI? It is deeply concerning that 'critical' infrastructure subscribers are taking this long, in addition to my previous questions can you please give me an idea of what Entrust have been doing to educate this category of subscriber?

I'll note 1.4.2 from your own CPS:

1.4.2 Prohibited Certificate Uses
The use of all Certificates issued by the CA shall be for lawful purposes and consistent with applicable laws, including without limitation, applicable export or import laws.

Certificates and the services provided by Entrust in respect to Certificates are not designed, manufactured, or intended for use in or in conjunction with any application in which failure could lead to death, personal injury or severe physical or property damage, including the monitoring, operation or control of nuclear facilities, mass transit systems, aircraft navigation or communications systems, air traffic control, weapon systems, medical devices or direct life support machines, and all such uses are prohibited.

Can you confirm none of the entities above are in that definition of 'critical infrastructure'?

Comment 25

[ngook.kong]

(In reply to Wayne from comment #18)

Thank you for your comments, Wayne. A breakdown with these details by affected certificate is in the attachment 9403115 [details].

Comment 26

[ngook.kong]

(In reply to Tim Callan from comment #20)

(In reply to Bruce Morton from comment #17)

Bruce,

After the recent publication of revoked versus expired numbers in bug 1887705 comment #24, I had thought it meant Entrust intended to report its delayed revocation events this way moving forward. I was surprised subsequently to see this post, without that breakdown.

We agree. These details have now been provided for bug 1887705 and will be provided in future updates for this and other bugs.

Does Entrust intend to distinguish revoked from naturally expired certificates in its delayed revocation and no-revocation incidents? When should we expect to see these numbers for this bug, bug 1890685, and bug 1890898? Please note also my comment in bug 1887705 comment 27 clarifying that the important distinction is between certs that expired versus those revoked, as opposed to when they would have expired had they not been revoked.

We will take this suggestion into consideration for no-revocation events.

Comment 27

[Mike Shaver (:shaver emeritus)]

(In reply to ngook.kong from comment #26)

We will take this suggestion into consideration for no-revocation events.

Hmm. My understanding is that what Tim describes is a requirement of the incident reporting process, and not a suggestion or optional activity. Does Entrust disagree?

[e: fixed name]

Comment 28

[ngook.kong]

(In reply to Wayne from comment #19)

I've had multiple of Entrust's subscribers privately contact me on their own accord, so I'll be very specific in my questioning additional to the above.

Can Entrust give a step by step process of how precisely these extensions you are providing are approved? To be clearer:

• During the cPSuri incident what was the contents of the email sent to providers asking for revocation to begin with?

We communicated with subscribers via email, video calls, phone calls, and through support ticket dialogue to ask them to justify their need for delayed revocation.  Specifically, we asked them to detail their technical challenges and/or impacts to the web ecosystem if forced to revoke within 5 days as well as how much time they needed to replace and revoke their certificates.
Because of the volume, we did not always receive or record this information effectively for all customers. We have identified this as an area of process improvement. Going forward, we are looking at the use of a standardized way to collect and record this data to ensure that if there are future mis-issuances, the collection of this information is more structured and complete.

• The Incident Report mentions contact on 03-19 requesting revocation. What mechanisms do Entrust use to check that a newly issued certificate is now in active use and that the older one can be safely revoked?

Unfortunately, there is no reliable mechanism to verify that old certificates can safely be revoked aside from confirming with the subscriber or having the subscriber do this themselves. However, to support our subscribers, we performed TLS checks for each DNS name included in the certificate.
This did not necessarily detect all certificates installed.  For example:

• on multiple servers through DNS load balancing (we do not check all IP addresses returned by the DNS server)
• on multiple servers through anycast
• on multiple servers via TCP load balancing
• behind a gateway or proxy, such as a Content Delivery Network (CDN)

We would appreciate this community’s thought leadership and assistance on ideas on this to improve this in the future.

• What is the followup process is the subscriber has generated a new certificate and has received it, but has not explicitly said it is safe to revoke?

In this type of case, we have been following up with the subscribers by email and phone to gain confirmation prior to revoking.  If we cannot get a response from the subscriber, we have made the decision to automatically revoke after a certain date.

• Following whichever timeframe of additional time has been approved by the above means, what is the process for following up with subscribers and who provides suggestions on possible timeframes?

We have been following up with subscribers by email and by phone leveraging our Support, Sales, Customer Success Managers and their teams. Customer timeframes are discussed between our compliance team and senior leadership team.

• Are there specific risk categories provided to subscribers to choose from, or do you get a freeform text reply to make a calculated risk analysis?

We have not given subscribers specific risk categories to choose from, but we have been considering whether providing specific risk categories would make sense in future incidents. For our incidents, we have only accepted freeform text replies and responses from subscribers given to our Support teams by email or by phone. We thought that would be best, but we have discovered that this method is not scalable. We would be interested in your view—and others in the community—on the use of a standard form with risk categories versus freeform text replies. We would only use such a form with subscribers if they understood that there is a presumption of denial. We would also use this form to discover customer certificate automation capability and commitment.

• Likewise are there specific choices of timeframes to postpone revocation? Do this depend on precise categories of subscribers?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation). We have not given subscribers specific revocation timeframes to choose from; however, in line with the comment above, we would appreciate the community’s view regarding the use of a fillable form for subscribers requesting delayed revocation that could be used to analyze and either approve or reject their requests. We would only use such a form with subscribers if they understood there is a presumption of denial.

When we reach the end of our delayed revocation, we plan to share this experience with this community. We are seeing the difference between what the customers are asking for and what the customers can do. To further break this down, we are also getting a sense of what a customer can do in an emergency and what they can do with operational efficiency.

• Given either answer above they must be generalized for internal analysis, so can Entrust provide a breakdown as it should be in their documentation on-hand and require no effort to produce?

Agreed. We provided the rationale for the delay and the expected timeframe to revoke and re-issue alongside each affected certificate. As stated earlier, the free form text in hindsight was not a very effective for categorizing this type of data. This is one of the reasons we provided the information in the formatted reasons.

• Were a subscriber to miss the 5-day window, get an additional extension (14d, 28d, whichever) and then miss that... what is the communication sent from Entrust?

We have been following up with subscribers on an individual basis in this situation regularly to gauge progress and to assess the reason for the continued delay, and then making determinations based on the specific circumstances whether to automatically revoke or provide further extension. We are looking to improve the process, but a customer might not acknowledge receipt of a communication, as that happens, we expand our list of contacts and methods of communications.

• On the process side then, is it a more detailed process or is it the same process outlined in a previous answer here? If so, which answer?

The process above is the process we have been consistently applying.  We are, however, open to community feedback on changing to use of a standardized form in the event of future mis-issuances and customer requests for approvals to delay revocation.

• Has this been the same process from the start of the incident until now? If not, what lessons were learned and applied?

Yes, it is the same process.  With the volume of certificates that had to be revoked and re-issued, this approach has put a strain on our Support teams. We are trying to prevent delayed revocation, but if this process was needed if the future, the use of a standard form for incidents could be useful for purposes of review and management. However, we will not change our approach without discussion and agreement from the community.

I've kept my questions brief and simple. I'm sure all CAs to agree this is a good learning ground on what should be said to subscribers, and how best to communicate in this scenario. If we have examples of what precisely Entrust is saying to their subscribers we can try and flesh it out into a generalized incident response format to help all CAs going forward.

We can share details of what we have communicated to our subscribers. As you can see, we submitted a categorized version of what we have received from our subscribers for the community to see.It would be a good idea if we could propose a form to the community for discussion. Please let us know if you would like us to supply a draft to start this discussion. If so, should we start a separate discussion or continue it here?

Comment 29

[Mike Shaver (:shaver emeritus)]

(In reply to ngook.kong from comment #28)

We communicated with subscribers via email, video calls, phone calls, and through support ticket dialogue to ask them to justify their need for delayed revocation.  Specifically, we asked them to detail their technical challenges and/or impacts to the web ecosystem if forced to revoke within 5 days as well as how much time they needed to replace and revoke their certificates.
Because of the volume, we did not always receive or record this information effectively for all customers. We have identified this as an area of process improvement. Going forward, we are looking at the use of a standardized way to collect and record this data to ensure that if there are future mis-issuances, the collection of this information is more structured and complete.

What criteria did Entrust use to determine that whether an extension was appropriate? Who made the decisions? How did they do so without any record being captured of the Subscriber’s reasoning?

Please also give examples of Subscribers that requested an extension and were not granted one.

Not “receiving” that information effectively sounds totally unacceptable in terms of Entrust’s responsibility to uphold the BRs. This is not the first time that Entrust has had to file a delayed revocation incident, and Entrust has been reminded many times (over multiple years) of the requirement to clearly specify the reasons for each certificate’s delayed revocation. It is hard to imagine a promise related to this responsibility that Entrust has not made and broken during their time as a root CA.

In this type of case, we have been following up with the subscribers by email and phone to gain confirmation prior to revoking.  If we cannot get a response from the subscriber, we have made the decision to automatically revoke after a certain date.

How is this date chosen? Please give an example of a case in which you revoked without a response from the subscriber, and how you decided that it was now acceptable to harm the critical services or similar under which justification the delay was permitted.

We have not given subscribers specific risk categories to choose from, but we have been considering whether providing specific risk categories would make sense in future incidents. For our incidents, we have only accepted freeform text replies and responses from subscribers given to our Support teams by email or by phone.

Please share examples of the text replies from Subscribers that Entrust determined were, and were not, appropriate for delayed revocation.

 We would only use such a form with subscribers if they understood that there is a presumption of denial. We would also use this form to discover customer certificate automation capability and commitment.

All subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Please describe in detail how you analyze this balance.

 We would only use such a form with subscribers if they understood there is a presumption of denial.

Was there a presumption of denial when evaluating Subscriber requests for delayed revocation in this incident? How was that presumption communicated to Subscribers?

Yes, it is the same process.  With the volume of certificates that had to be revoked and re-issued, this approach has put a strain on our Support teams.

Do you feel that Entrust is capable of effectively responding to a wide-scale revocation requirement, such as due to exposure of root key material?

If so, please explain how that response would differ from the one employed in this and other recent delayed revocation incidents.

We are trying to prevent delayed revocation,

I want to remind you that you can absolutely prevent delayed revocation, simply by revoking under the timelines specified in the BRs that Entrust and its subscribers have agreed to, with legal force. If Entrust issues certificates to Subscribers who are not able or willing to meet their legal commitment to accept immediate revocation, then the ensuing risk must live with Entrust and its Subscribers, and not with the users who depend on the integrity of WebPKI. A CA’s pattern of misissuance can disrupt their Subscribers’ operations, certainly, but that is for the Subscribers (and Entrust) to deal with, and extending the use of misissued certificates is not an acceptable way to avoid that disruption.

We can share details of what we have communicated to our subscribers. As you can see, we submitted a categorized version of what we have received from our subscribers for the community to see. It would be a good idea if we could propose a form to the community for discussion. Please let us know if you would like us to supply a draft to start this discussion. If so, should we start a separate discussion or continue it here?

Yes, this would be a welcome contribution, thank you. I think you should start a discussion about this topic on mdsp. It would also be valuable for Entrust to give its opinion on the discussion that Wayne recently started about communication of revocation-requiring incidents to Subscribers.

Comment 30

[Wayne]

My question was:

1. During the cPSuri incident what was the contents of the email sent to providers asking for revocation to begin with?

I will repeat that as it was not answered. What was the contents of the email? I'll provide a tip my mdsp post is based on it, so you can be transparent here.

In this type of case, we have been following up with the subscribers by email and phone to gain confirmation prior to revoking.  If we cannot get a response from the subscriber, we have made the decision to automatically revoke after a certain date.

What is a 'certain date'? If you email me on 04-01, I don't respond by your deadline of say 04-06 then do you start a 5-day clock ending on 04-11? Of the subscribers in this incident how many have not replied, and what timeframes were provided? How many ended up revoked due to this?

We have been following up with subscribers by email and by phone leveraging our Support, Sales, Customer Success Managers and their teams. Customer timeframes are discussed between our compliance team and senior leadership team.

Okay that answers the who on the timeframe question: Entrust provide the timeframe.

We have not given subscribers specific risk categories to choose from, but we have been considering whether providing specific risk categories would make sense in future incidents. For our incidents, we have only accepted freeform text replies and responses from subscribers given to our Support teams by email or by phone. We thought that would be best, but we have discovered that this method is not scalable. We would be interested in your view—and others in the community—on the use of a standard form with risk categories versus freeform text replies. We would only use such a form with subscribers if they understood that there is a presumption of denial. We would also use this form to discover customer certificate automation capability and commitment.

Okay that clears that part up, thank you.

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation). We have not given subscribers specific revocation timeframes to choose from; however, in line with the comment above, we would appreciate the community’s view regarding the use of a fillable form for subscribers requesting delayed revocation that could be used to analyze and either approve or reject their requests. We would only use such a form with subscribers if they understood there is a presumption of denial.

When we reach the end of our delayed revocation, we plan to share this experience with this community. We are seeing the difference between what the customers are asking for and what the customers can do. To further break this down, we are also getting a sense of what a customer can do in an emergency and what they can do with operational efficiency.

I look forward to seeing this breakdown. Entrust are aware they are not the entity that should balance risks? It is their duty to enforce their revocation policies and it is the subscriber who should be performing any risk calculus per the subscriber agreement, correct?

Agreed. We provided the rationale for the delay and the expected timeframe to revoke and re-issue alongside each affected certificate. As stated earlier, the free form text in hindsight was not a very effective for categorizing this type of data. This is one of the reasons we provided the information in the formatted reasons.

Ah, there is a misunderstanding I am asking you to provide that documentation. I hope it makes sense as to why.

Yes, it is the same process.  With the volume of certificates that had to be revoked and re-issued, this approach has put a strain on our Support teams. We are trying to prevent delayed revocation, but if this process was needed if the future, the use of a standard form for incidents could be useful for purposes of review and management. However, we will not change our approach without discussion and agreement from the community.

Haven't Entrust already provided action items to change their incident response without discussion and agreement from the community? How is this answer compatible with this incident?

We can share details of what we have communicated to our subscribers. As you can see, we submitted a categorized version of what we have received from our subscribers for the community to see.It would be a good idea if we could propose a form to the community for discussion. Please let us know if you would like us to supply a draft to start this discussion. If so, should we start a separate discussion or continue it here?

Agreed, I'd recommend the mdsp thread on incident correspondence as a starting point: https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/OKzHYta8AxM

Comment 31

[ngook.kong]

(In reply to Wayne from comment #24)

Awaiting a response to my previous questions. In the meantime let's break down the above. The original 'reason's are badly formatted and have redundant options so here's a legible version:

Reason 23/05/24 24/05/24 30/05/24 02/06/24 Total
Harm, Critical - 14 30 - 44
Harm, Critical, Manual, Teams - - 3146 2798 5944
Harm, Critical, Teams 43 8 2541 - 2592
Total 43 22 5717 2798 8580

Then breaking it down by subscriber via organizationName:

Are your subscribers specifying what they mean by critical infrastructure? If it is truly critical then are you helping them move to a private root and away from WebPKI? It is deeply concerning that 'critical' infrastructure subscribers are taking this long, in addition to my previous questions can you please give me an idea of what Entrust have been doing to educate this category of subscriber?

Wayne, our subscribers are generally describing critical infrastructure as impactful to the economy or society. As we have started talking to these customers about other options including private root PKIs that can potentially de-risk their environment, but we are also talking about more traditional strategies as well including better automation tools (many are free from Entrust), processes, architecture and practicing out of period renewal because tools will be needed even in a privately rooted PKI.

I'll note 1.4.2 from your own CPS:

1.4.2 Prohibited Certificate Uses
The use of all Certificates issued by the CA shall be for lawful purposes and consistent with applicable laws, including without limitation, applicable export or import laws.
Certificates and the services provided by Entrust in respect to Certificates are not designed, manufactured, or intended for use in or in conjunction with any application in which failure could lead to death, personal injury or severe physical or property damage, including the monitoring, operation or control of nuclear facilities, mass transit systems, aircraft navigation or communications systems, air traffic control, weapon systems, medical devices or direct life support machines, and all such uses are prohibited.

Can you confirm none of the entities above are in that definition of 'critical infrastructure'?

To the best of our knowledge, we are not aware of our customers using any of these prohibited use cases.

Comment 32

[ngook.kong]

Attachment: EV Bugzilla Post 5.28.2024.csv

Comment 33

[ngook.kong]

Here is our progress:
· 19,924 of 26,668 (74.7%) certificates have been revoked or expired.
· 19,701 revoked.
· 195 expired.
· 1,153 certificates have been re-issued with revocation pending.
· 915 out of 944 customer accounts (96.9%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Attached is the information per customer, we are working with each subscriber to accelerate the time to revocation, of the remaining 6,744 certificates. Attachment 9404486 [details]

Comment 34

[Wayne]

(In reply to ngook.kong from comment #31)

Wayne, our subscribers are generally describing critical infrastructure as impactful to the economy or society. As we have started talking to these customers about other options including private root PKIs that can potentially de-risk their environment, but we are also talking about more traditional strategies as well including better automation tools (many are free from Entrust), processes, architecture and practicing out of period renewal because tools will be needed even in a privately rooted PKI.

I'll note 1.4.2 from your own CPS:

1.4.2 Prohibited Certificate Uses
The use of all Certificates issued by the CA shall be for lawful purposes and consistent with applicable laws, including without limitation, applicable export or import laws.
Certificates and the services provided by Entrust in respect to Certificates are not designed, manufactured, or intended for use in or in conjunction with any application in which failure could lead to death, personal injury or severe physical or property damage, including the monitoring, operation or control of nuclear facilities, mass transit systems, aircraft navigation or communications systems, air traffic control, weapon systems, medical devices or direct life support machines, and all such uses are prohibited.

Can you confirm none of the entities above are in that definition of 'critical infrastructure'?

To the best of our knowledge, we are not aware of our customers using any of these prohibited use cases.

You cannot have it both ways. Either a subscriber is critical infrastructure and thus forbidden to even have a certificate issued by Entrust, or they are not and should have been revoked already. Please provide a per-subscriber breakdown of what exactly you mean by critical infrastructure and the subscriber's risk.

To clarify Entrust are stating no entity is using their certificates in any application:

• designed, manufactured, or intended for us in,
• or in conjunction with any application that

could lead to death, personal injury or severe physical or property damage,

including the monitoring, operation, or control of:

• nuclear facilities
• mass transit systems
• aircraft navigation or communications systems
• air traffic control
• weapon systems
• medical devices or direct life support machines

and all such uses

As your comments in another incident state otherwise:

It might appear that customers of a subscriber can easily move from one service to another. The disruption is more significant as most of these subscribers are dealing with thousands or millions of consumers who have financial information located with a specific provider or have tickets with a certain airline, or have other financial transaction that are held with a subscriber. This potential disruption is further compounded when these subscribers have a large number of certificates.

Could Entrust explain how a ticket to an airline is not used in conjunction with an application for a mass transit system? I will note that the last item of 'and all such uses' encompasses 'any application that could lead to death, personal injury or severe physical or property damage'. That is what people general call 'critical infrastructure'. Can Entrust explain what definition they are operating under? What measures are even taken to ensure no prohibited certificates are issued to a misleading subscriber? What is the revocation plan in this event, given there can't be a re-issuance?

(In reply to ngook.kong from comment #33)

Here is our progress:
· 19,924 of 26,668 (74.7%) certificates have been revoked or expired.
· 19,701 revoked.
· 195 expired.
· 1,153 certificates have been re-issued with revocation pending.
· 915 out of 944 customer accounts (96.9%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Attached is the information per customer, we are working with each subscriber to accelerate the time to revocation, of the remaining 6,744 certificates. Attachment 9404486 [details]

5 Subscribers have been dealt with in the past week. Are Entrust still confident about dealing with this by May 30th? We are still lacking in any guarantee that a delayed revocation like this will never happen again. I also can't help but notice that the deadlines have been removed.

Last week's entry noted that the Bank of Montreal would revoke this certificate by 2024-05-23, and it is still active. Same situation for Netrust Pte Ltd - Philippines for May 24th. It has been 5 days since the self-imposed deadline for those and they're still delayed...

Of the remaining 6,744 certificates only 1,153 have been re-issued, so what is the situation with the other 5,591?

Comment 35

[ngook.kong]

(In reply to Mike from comment #27)

(In reply to ngook.kong from comment #26)

We will take this suggestion into consideration for no-revocation events.

Hmm. My understanding is that what Tim describes is a requirement of the incident reporting process, and not a suggestion or optional activity. Does Entrust disagree?

Our interpretation is any delayed revocation will comply with the Mozilla revocation policy at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

As we indicated in our response on Bug 1887705 https://bugzilla.mozilla.org/show_bug.cgi?id=1887705#c30, the data we generated for current Bugs “didn’t clarify if these certificates were revoked before they expired or not, we agree this is valuable information and will ensure we include this distinction in our future updates.” That’s our intention for the future.

Comment 36

[ngook.kong]

(In reply to Mike from comment #29)

(In reply to ngook.kong from comment #28)

We communicated with Subscribers via email, video calls, phone calls, and through support ticket dialogue to ask them to justify their need for delayed revocation.  Specifically, we asked them to detail their technical challenges and/or impacts to the web ecosystem if forced to revoke within 5 days as well as how much time they needed to replace and revoke their certificates.
Because of the volume, we did not always receive or record this information effectively for all customers. We have identified this as an area of process improvement. Going forward, we are looking at the use of a standardized way to collect and record this data to ensure that if there are future mis-issuances, the collection of this information is more structured and complete.

What criteria did Entrust use to determine that whether an extension was appropriate? Who made the decisions? How did they do so without any record being captured of the Subscriber’s reasoning?

Criteria: Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem, we did not grant their delayed revocation requests.
Who made the decisions: Our product compliance leader initially reviewed and accepted the requests for delayed revocation. As the request volume escalated, a cross functional senior leadership team, advised by our product compliance team was activated for such decisions.
How: Our Support and Customer Success teams were empowered to communicate with Subscribers and were instructed not to approve requests. Some of these conversations occurred via phone or video conference and the articulated rationale was not captured in the very early days of the response.

Ultimately, a formal process was stood up to capture all requests in more detail. This is a key reason we are exploring the use of a centralized request form and/or formal tracking mechanism to ensure all this information is captured in a central location and easily reportable. We value input from the community on this.

Please also give examples of Subscribers that requested an extension and were not granted one.

Following are 2 examples of supplied reasons where the request was denied.

If subscribers provided little to no information using terms like “unknown” or “confidential” when asked to provide a rational for delayed revocation.
If subscribers provided reason that they had other critical business priorities and could not spare the resources

Not “receiving” that information effectively sounds totally unacceptable in terms of Entrust’s responsibility to uphold the BRs. This is not the first time that Entrust has had to file a delayed revocation incident, and Entrust has been reminded many times (over multiple years) of the requirement to clearly specify the reasons for each certificate’s delayed revocation. It is hard to imagine a promise related to this responsibility that Entrust has not made and broken during their time as a root CA.

In this type of case, we have been following up with the Subscribers by email and phone to gain confirmation prior to revoking.  If we cannot get a response from the subscriber, we have made the decision to automatically revoke after a certain date.

How is this date chosen? Please give an example of a case in which you revoked without a response from the subscriber, and how you decided that it was now acceptable to harm the critical services or similar under which justification the delay was permitted.

We have not given Subscribers specific risk categories to choose from, but we have been considering whether providing specific risk categories would make sense in future incidents. For our incidents, we have only accepted freeform text replies and responses from Subscribers given to our Support teams by email or by phone.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Please share examples of the text replies from Subscribers that Entrust determined were, and were not, appropriate for delayed revocation.

We would only use such a form with Subscribers if they understood that there is a presumption of denial. We would also use this form to discover customer certificate automation capability and commitment.

Here are some specific examples of rationales we received from Subscribers:

· “These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

· “Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

· “We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

· “Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

Please describe in detail how you analyze this balance.

We would only use such a form with Subscribers if they understood there is a presumption of denial.

We start with a presumption of denial, but there are circumstances that justify some period of delay.
We have used as precedence the approach and other CAs have also allowed for delayed revocation on the same basis.
We understand and agree that delayed revocation requests are the exception and not the rule and that is how we approached this with our Subscribers. We are committed to continuous improvement of this process.

Was there a presumption of denial when evaluating Subscriber requests for delayed revocation in this incident? How was that presumption communicated to Subscribers?

Yes, it is the same process.  With the volume of certificates that had to be revoked and re-issued, this approach has put a strain on our Support teams.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Do you feel that Entrust is capable of effectively responding to a wide-scale revocation requirement, such as due to exposure of root key material?
If so, please explain how that response would differ from the one employed in this and other recent delayed revocation incidents.

We are trying to prevent delayed revocation,

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

I want to remind you that you can absolutely prevent delayed revocation, simply by revoking under the timelines specified in the BRs that Entrust and its Subscribers have agreed to, with legal force. If Entrust issues certificates to Subscribers who are not able or willing to meet their legal commitment to accept immediate revocation, then the ensuing risk must live with Entrust and its Subscribers, and not with the users who depend on the integrity of WebPKI. A CA’s pattern of misissuance can disrupt their Subscribers’ operations, certainly, but that is for the Subscribers (and Entrust) to deal with, and extending the use of misissued certificates is not an acceptable way to avoid that disruption.

We can share details of what we have communicated to our Subscribers. As you can see, we submitted a categorized version of what we have received from our Subscribers for the community to see. It would be a good idea if we could propose a form to the community for discussion. Please let us know if you would like us to supply a draft to start this discussion. If so, should we start a separate discussion or continue it here?
Yes, this would be a welcome contribution, thank you. I think you should start a discussion about this topic on mdsp. It would also be valuable for Entrust to give its opinion on the discussion that Wayne recently started about communication of revocation-requiring incidents to Subscribers.

We welcome any opportunity to engage with the community on these issues. We will comment on mdsp where we believe we can add value to the discussion.

Comment 37

[walter.j.marks]

Setting aside the initial failure to revoke and the long tail of this process, I'd like to dig a bit further into the Subscriber arguments against revoking, and some of your statements included there. I'm going to make a few assumptions here, including that Entrust agrees that there is in fact a violation of the BRs requiring revocation of affected certs, otherwise I assume that you wouldn't have revoked in the first place.

“These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

“Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

“We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Q1. Were each of these Subscribers reminded of their agreement to the Terms of Use of Entrust Certificate and Signing Services? Specifically, Exhibit B, Part 1.12.3?

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Q2. Why was a presumption of a Subscriber being unable to revoke without positive confirmation assumed as stated above? Again following from Exhibit B, Part 1.12.3, Entrust is entitled to revoke a certificate immediately without notifying the Subscriber.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Q3. How does the above statement square with the the subscriber justification given here?

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

“Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Again, you seem to misunderstand the question posed by the following quote:

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

Of those three exemplar Subscribers I've referenced above, none seem to have a lack of automation as a blocking fact that they were unable to revoke (which again, should be an action taken by Subscribers by the powers given to you by Exhibit B, Part 1.12.3) and replace in time. A commitment to automation does not mean anything if the subscribers are simply refusing to accept a revocation within the required and expected time. A more accurate commitment that I would expect to answer the question would be something along the lines of "Entrust will refuse to accept any requests to delay revocation past 5 days....." along with a measurable and actionable key index for that commitment.
Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

Q5.1. Of these three exemplar Subscribers, were any reminded that if this was a key compromise event that had fundamental security risks, that this argument of "needing to notify downstream contracted parties that could take up to 3 months" (paraphrased) would potentially result in their TLS certificates being unsafe for the web ecosystem and their business operations for the entire 90 day period?
Q5.2. If so, what was the justification provided by the Subscriber that they would be able to handle a situation like that within the 24h/120h period, and why that process was unable to be performed in this case, given that from a pure compliance perspective, in either case the affected certificate is invalid and should not be used by the Subscriber?

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Q6. The above statement seems to indicate that Entrust believes that this incident is still not at the level of a revocation requirement. If that is the case, please explain how the processes followed for this mis-issuance are different from the planned handling of a wide-scale revocation requirement.

Comment 38

[ngook.kong]

(In reply to Wayne from comment #30)

My question was:
1. During the cPSuri incident what was the contents of the email sent to providers asking for revocation to begin with?
I will repeat that as it was not answered. What was the contents of the email? I'll provide a tip my mdsp post is based on it, so you can be transparent here.

Email content

Subject:URGENT ACTION REQUIRED: EV Revocation May be Required
Message Group: System Interruptions
Message Expiry Date: 3/30/24

We are writing to inform you about a recent issue that affected some of the EV digital certificates issued by Entrust. We apologize for any inconvenience this may have caused you and we are committed to ensuring the security and integrity of your online transactions.

Summary:

Entrust discovered that some Extended Validation (EV) TLS certificates (EV Multi-Domain SSL, QWAC eIDAS, QWAC PSD2) were missing a specific component required by the EV Guidelines. This component links the certificate to the Certificate Policy (CP) and the Certification Practice Statement (CPS) of the issuer.

Entrust has taken steps to address the issue and prevent it from happening again.
Any of the specified certificate types issued after 21:40 UTC today, Mar 18, 2024 are not affected.
Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC

ACTION REQUIRED:

Certificates issued after September 11th, 17:40 UTC, 2023, to March 18th, 21:40 UTC, 2024, will need to be replaced by issuing a new certificate and revoking the old certificate(s) shortly thereafter. We will assist you with this process and ensure a smooth transition.

Steps: [REDACTED]

Thank you,
Entrust Certificates Services

In this type of case, we have been following up with the Subscribers by email and phone to gain confirmation prior to revoking. If we cannot get a response from the subscriber, we have made the decision to automatically revoke after a certain date.

What is a 'certain date'? If you email me on 04-01, I don't respond by your deadline of say 04-06 then do you start a 5-day clock ending on 04-11? Of the Subscribers in this incident how many have not replied, and what timeframes were provided? How many ended up revoked due to this?

We have been following up with Subscribers by email and by phone leveraging our Support, Sales, Customer Success Managers and their teams. Customer timeframes are discussed between our compliance team and senior leadership team.

As noted in response to Comment 29, we were ultimately able to communicate with all Subscribers. All Subscribers were reached before the date we set internally to hard revoke all remaining affected certificates.

Okay that answers the who on the timeframe question: Entrust provide the timeframe.

We have not given Subscribers specific risk categories to choose from, but we have been considering whether providing specific risk categories would make sense in future incidents. For our incidents, we have only accepted freeform text replies and responses from Subscribers given to our Support teams by email or by phone. We thought that would be best, but we have discovered that this method is not scalable. We would be interested in your view—and others in the community—on the use of a standard form with risk categories versus freeform text replies. We would only use such a form with Subscribers if they understood that there is a presumption of denial. We would also use this form to discover customer certificate automation capability and commitment.

Okay that clears that part up, thank you.

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation). We have not given Subscribers specific revocation timeframes to choose from; however, in line with the comment above, we would appreciate the community’s view regarding the use of a fillable form for Subscribers requesting delayed revocation that could be used to analyze and either approve or reject their requests. We would only use such a form with Subscribers if they understood there is a presumption of denial.

When we reach the end of our delayed revocation, we plan to share this experience with this community. We are seeing the difference between what the customers are asking for and what the customers can do. To further break this down, we are also getting a sense of what a customer can do in an emergency and what they can do with operational efficiency.

I look forward to seeing this breakdown. Entrust are aware they are not the entity that should balance risks? It is their duty to enforce their revocation policies and it is the subscriber who should be performing any risk calculus per the subscriber agreement, correct?

Agreed. We provided the rationale for the delay and the expected timeframe to revoke and re-issue alongside each affected certificate. As stated earlier, the free form text in hindsight was not a very effective for categorizing this type of data. This is one of the reasons we provided the information in the formatted reasons.

Ah, there is a misunderstanding I am asking you to provide that documentation. I hope it makes sense as to why.

At the time that we collected the information, there was no reasonable expectation either by Entrust or the Subscribers that every detail of the communication would be public information. Due to standard NDA provisions between Entrust and its Subscribers, the Subscribers provided some information that we do not feel at liberty to share, nor do we feel that those details are necessary to addressing the issues identified in this incident. We believe the formatted reasons we provided are sufficient and the discussion around a standard form is a more productive path to making improvements going forward.

Yes, it is the same process. With the volume of certificates that had to be revoked and re-issued, this approach has put a strain on our Support teams. We are trying to prevent delayed revocation, but if this process was needed if the future, the use of a standard form for incidents could be useful for purposes of review and management. However, we will not change our approach without discussion and agreement from the community.

Haven't Entrust already provided action items to change their incident response without discussion and agreement from the community? How is this answer compatible with this incident?

Yes, we have made commitments to improve our incident response process. Some improvements we can make unilaterally, and other improvements would benefit from discussion and agreement from the community. In our response above, we indicated that the use of a standard form for incidents could be useful, and we think discussion and agreement from the community would be particularly applicable here.

We can share details of what we have communicated to our Subscribers. As you can see, we submitted a categorized version of what we have received from our Subscribers for the community to see.It would be a good idea if we could propose a form to the community for discussion. Please let us know if you would like us to supply a draft to start this discussion. If so, should we start a separate discussion or continue it here?

Agreed, I'd recommend the mdsp thread on incident correspondence as a starting point: https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/OKzHYta8AxM

Comment 39

[walter.j.marks]

(In reply to walter.j.marks from comment #37)

[...]

Minor typo corrections to ensure we're all on the same page:

which again, should be an action taken by Subscribers Entrust by the powers given to you by Exhibit B, Part 1.12.3

Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

Comment 40

[Wayne]

(In reply to ngook.kong from comment #36)

Criteria: Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem, we did not grant their delayed revocation requests.
Who made the decisions: Our product compliance leader initially reviewed and accepted the requests for delayed revocation. As the request volume escalated, a cross functional senior leadership team, advised by our product compliance team was activated for such decisions.
How: Our Support and Customer Success teams were empowered to communicate with Subscribers and were instructed not to approve requests. Some of these conversations occurred via phone or video conference and the articulated rationale was not captured in the very early days of the response.

Ultimately, a formal process was stood up to capture all requests in more detail. This is a key reason we are exploring the use of a centralized request form and/or formal tracking mechanism to ensure all this information is captured in a central location and easily reportable. We value input from the community on this.

Can we take a step back from this rationale and just talk numbers:

• How many requests came in (approx), how many were initially denied? I understand a longer conversation then occurred, but give us an idea of the initial first week of Entrust taking action.
• What was the minimum and maximum length of time given to a subscriber to delay originally?
• With those subscribers as an anonymous basis, when were they ultimately revoked?
• Given deadlines would have been presented were hit, how many hard revocations occurred?
• How many subscribers manually revoked, and how many did so via their deadline expiring but after a new certificate was issued?

Now I'll be frank - I do not expect perfection on this front, this is where being transparent is helpful. There will be cases of subscribers going past the deadline and the case not being immediately handled, what I am looking for is a sense of scale.

Please also give examples of Subscribers that requested an extension and were not granted one.

Following are 2 examples of supplied reasons where the request was denied.

If subscribers provided little to no information using terms like “unknown” or “confidential” when asked to provide a rational for delayed revocation.
If subscribers provided reason that they had other critical business priorities and could not spare the resources

This answer is not clear. Were these subscribers then subsequently hard revoked after their initial request was denied? You are not identifying any specific subscribers, and that is okay and should free you for answering this question.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Do you mean that there was always a response from the subscriber prior to revocation?

Here are some specific examples of rationales we received from Subscribers:

· “These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

· “Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

· “We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

We can agree that none of these are discussing critical infrastructure correct? Inconvenient, busy, change freeze. None of these reflect any actual harm and instead reflect an indifference to the situation as explained by Entrust to their subscribers. That there are subscribers who have stated they have a 3-month minimum turnaround and yet it hasn't been 3 months shows that you need to actually enforce your duty to revoke. Ultimately they are stating they are incapable of handling revocation in the event of a 24 hour event and the continued issuance to them is introducing harm to the ecosystem.

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

· “Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Given you are not identifying this subscriber can you clarify what was given by 'some time to delay'? As an anonymous subscriber, could a breakdown be provided of roughly the days of each communication and subsequent shift in the revocation window?

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

Automation will not save any of the above subscribers, this is not the time for a sales pitch to them but in showing how you handle compliance. The subscribers are already legally committed to accept immediate revocation, it is on Entrust to actually enforce. The failure to do so reflects on how seriously Entrust have taken this situation.

Please describe in detail how you analyze this balance.

We would only use such a form with Subscribers if they understood there is a presumption of denial.

We start with a presumption of denial, but there are circumstances that justify some period of delay.
We have used as precedence the approach and other CAs have also allowed for delayed revocation on the same basis.
We understand and agree that delayed revocation requests are the exception and not the rule and that is how we approached this with our Subscribers. We are committed to continuous improvement of this process.

What we need to hear is "this will never happen again", not "continuous improvement".

Yes and it was communicated exactly as such. No delayed revocation options were offered.

To clarify, no delayed revocation options were offered - therefore any specific dates were agreed on a subscriber-by-subscriber basis? Surely there were some benchmark numbers for each support team member to work with? This should have been treated as an immediate emergency by all parties and any date provided should have been by end of business day.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Do you feel that if Entrust were to be required to perform a wide-scale revocation it would be done within 24 hours? Even just this limited sample of 24k certificates? The reliance on subscribers manually taking action and making a per-subscriber deal on when it would be convenient for them says otherwise.

If this incident were to happen again with the same subscribers tomorrow, would we be complete within 5 days?

(In reply to ngook.kong from comment #38)

Steps: [REDACTED]

Any particular reason Entrust have decided to redact this? I have not been subtle on this already being known, so I'll give an example of what was being sent to subscribers in this incident:

Perform a REISSUE, and select "Revoke within 30 days" so your production certificate maintains validity, providing sufficient time to perform the replacement

This is from an example in the public domain that I'll identify privately to a Root Program (my email is always open). That you're still not being transparent on this front is not improving the level of trust. Given the above process and it already being a bit more than 30 days, what actually happened?

What is a 'certain date'? If you email me on 04-01, I don't respond by your deadline of say 04-06 then do you start a 5-day clock ending on 04-11? Of the Subscribers in this incident how many have not replied, and what timeframes were provided? How many ended up revoked due to this?

We have been following up with Subscribers by email and by phone leveraging our Support, Sales, Customer Success Managers and their teams. Customer timeframes are discussed between our compliance team and senior leadership team.

As noted in response to Comment 29, we were ultimately able to communicate with all Subscribers. All Subscribers were reached before the date we set internally to hard revoke all remaining affected certificates.

As the above should make clear I was very specific in my question for a reason. The email stated revocation would occur Saturday, March 23, 2024 at 21:00 UTC. Given that, did every subscriber respond by that deadline? The statements so far are only that communication was possible with every subscriber across an ill-defined timeframe, the specificity matters.

Currently we are being informed that a hard deadline was stated by Entrust, but that no certificates were revoked due to this. Is that the case?

At the time that we collected the information, there was no reasonable expectation either by Entrust or the Subscribers that every detail of the communication would be public information. Due to standard NDA provisions between Entrust and its Subscribers, the Subscribers provided some information that we do not feel at liberty to share, nor do we feel that those details are necessary to addressing the issues identified in this incident. We believe the formatted reasons we provided are sufficient and the discussion around a standard form is a more productive path to making improvements going forward.

In Comment 36 there is more specificity included than my question asks for. I am not asking for every subscriber's full reason, my point is that during this a generalized report would have been generated that gives an overall picture of their justifications. As part of the response to Mozilla I would recommend including it to fully explain what has occurred.

Any statements about a standard form for delayed revocation is truly misunderstanding the situation. This is not supposed to be standard at all, that there is more than one is already a serious situation. All that statement is saying to me, is that Entrust is comfortable that this will happen again and wish to have a letter than can be used as an excuse for later improvements. My MDSP post is about the initial revocation correspondence for a reason - that is all that should occur. A fixed deadline that is enforced, no exceptions.

Yes, we have made commitments to improve our incident response process. Some improvements we can make unilaterally, and other improvements would benefit from discussion and agreement from the community. In our response above, we indicated that the use of a standard form for incidents could be useful, and we think discussion and agreement from the community would be particularly applicable here.

In keeping with the above it is essential that a distinction is made between what is made standard. Incidents will occur, the important part is communication and enforcement of policy. To that end I'll repeat one of my questions that got lost:

Entrust are aware they are not the entity that should balance risks? It is their duty to enforce their revocation policies and it is the subscriber who should be performing any risk calculus per the subscriber agreement, correct?

Comment 41

[Wayne]

(In reply to ngook.kong from comment #33)

Here is our progress:
· 19,924 of 26,668 (74.7%) certificates have been revoked or expired.
· 19,701 revoked.
· 195 expired.
· 1,153 certificates have been re-issued with revocation pending.
· 915 out of 944 customer accounts (96.9%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Attached is the information per customer, we are working with each subscriber to accelerate the time to revocation, of the remaining 6,744 certificates. Attachment 9404486 [details]

As with #1887705 here is an ongoing update on the revocation status based off of Attachment 9404486:

Date	Revoked
2024-05-28	257
2024-05-29	415
2024-05-30	3017
2024-05-31	280
2024-06-01	2766

This leaves 9 certificates that are due to be revoked, checking Attachment 9403115 we can see when these were due to be revoked:

crt.sh SHA256 Link	Due Date
FA1A87488B3FB3537D014EEBA171011EDD7F08D2A3E43F1C227C0CF470E4403E	2024-05-30
46960B91CB3422343F7F6407255D150A0520CC053529614675DE4634921A91B5	2024-05-30
FABDCA5071477AE28B3789630A5E0E0AF0E9D56EC4DCB1525CE55E919E1A7D68	2024-05-30
4DFDE8E7E4D4510F8C3E64273669C6BE6D356DE5916FF72911BFF3E4A8BCA8EE	2024-05-30
42CE9531C187834E74B07495552F553129DD69C105D526232B6BC43CEF4733BA	2024-05-30
FCB7F4F440236BA3CA7DF8EECC2E82673E97919B884B384621C50CBD466E80B5	2024-06-02
121FFC5063A1AE96B66884388E6894FEF17F2F5964A00BD5C1BAFEE0D84CD3D4	2024-06-02
77D93210505C73CAC1BFD4F1014533F69891E717BC30FA0F9B7CF360FFF2534D	2024-06-02
E2E3E23C22558131E9BD7ACFCF15FFA1FBBC2F529AA3FAB21AD220EF93436AD7	2024-06-02

Comment 42

[ngook.kong]

(In reply to Wayne from comment #34)

(In reply to ngook.kong from comment #31)

Wayne, our subscribers are generally describing critical infrastructure as impactful to the economy or society. As we have started talking to these customers about other options including private root PKIs that can potentially de-risk their environment, but we are also talking about more traditional strategies as well including better automation tools (many are free from Entrust), processes, architecture and practicing out of period renewal because tools will be needed even in a privately rooted PKI.

I'll note 1.4.2 from your own CPS:

1.4.2 Prohibited Certificate Uses
The use of all Certificates issued by the CA shall be for lawful purposes and consistent with applicable laws, including without limitation, applicable export or import laws. Certificates and the services provided by Entrust in respect to Certificates are not designed, manufactured, or intended for use in or in conjunction with any application in which failure could lead to death, personal injury or severe physical or property damage, including the monitoring, operation or control of nuclear facilities, mass transit systems, aircraft navigation or communications systems, air traffic control, weapon systems, medical devices or direct life support machines, and all such uses are prohibited.

Can you confirm none of the entities above are in that definition of 'critical infrastructure'?

To the best of our knowledge, we are not aware of our customers using any of these prohibited use cases.

You cannot have it both ways. Either a subscriber is critical infrastructure and thus forbidden to even have a certificate issued by Entrust, or they are not and should have been revoked already. Please provide a per-subscriber breakdown of what exactly you mean by critical infrastructure and the subscriber's risk.

To clarify Entrust are stating no entity is using their certificates in any application:

• designed, manufactured, or intended for us in,
• or in conjunction with any application that

could lead to death, personal injury or severe physical or property damage,

including the monitoring, operation, or control of:

• nuclear facilities
• mass transit systems
• aircraft navigation or communications systems
• air traffic control
• weapon systems
• medical devices or direct life support machines

and all such uses

As your comments in another incident state otherwise:

It might appear that customers of a subscriber can easily move from one service to another. The disruption is more significant as most of these subscribers are dealing with thousands or millions of consumers who have financial information located with a specific provider or have tickets with a certain airline, or have other financial transaction that are held with a subscriber. This potential disruption is further compounded when these subscribers have a large number of certificates.

Could Entrust explain how a ticket to an airline is not used in conjunction with an application for a mass transit system? I will note that the last item of 'and all such uses' encompasses 'any application that could lead to death, personal injury or severe physical or property damage'. That is what people general call 'critical infrastructure'. Can Entrust explain what definition they are operating under? What measures are even taken to ensure no prohibited certificates are issued to a misleading subscriber? What is the revocation plan in this event, given there can't be a re-issuance?

The concept of "critical infrastructure" can be complex, but there are some definitions such as in CISA and the EU CER and NIS-2, which provide helpful guidance. There's a spectrum of risk within critical infrastructure. Activities directly involved in life-or-death scenarios (e.g., controlling air traffic) fall under our prohibited use cases. However, there is a range of critical infrastructure where the use of certificates is not prohibited.

For instance, airline ticketing and check-in systems wouldn't be classified as a prohibited use case. While an outage caused by certificate revocation could lead to serious negative impacts for air travelers, it wouldn't directly risk human life. Irresponsible revocation without considering potential consequences could lead to widespread disruption, impacting millions of travelers and causing significant economic harm. This highlights the importance of our collaborative approach with subscribers to ensure a smooth re-issuance process and minimize disruption.

We have learned that subscribers often confuse "critical infrastructure" with “mission critical infrastructure”, this is why we are committed to improving our processes and subscriber education around revocation and are trying to further bridge the gap between perceived and real impact. Through collaborative risk assessments based on detailed questionnaires, we could more objectively assess true disruption in real critical infrastructure.

(In reply to ngook.kong from comment #33)

Here is our progress:
· 19,924 of 26,668 (74.7%) certificates have been revoked or expired.
· 19,701 revoked.
· 195 expired.
· 1,153 certificates have been re-issued with revocation pending.
· 915 out of 944 customer accounts (96.9%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Attached is the information per customer, we are working with each subscriber to accelerate the time to revocation, of the remaining 6,744 certificates. Attachment 9404486 [details]

5 Subscribers have been dealt with in the past week. Are Entrust still confident about dealing with this by May 30th?

As stated in Comment 17, most of the remaining certificates were revoked on or before May 30, with some limited exceptions expected beyond that date. Another batch was revoked on the first of June and currently only 9 certificates belonging to 4 subscribers remain in terms of pending revocation.

Of the remaining 6,744 certificates only 1,153 have been re-issued, so what is the situation with the other 5,591?

As stated above, 9 certificates are remaining.

Comment 43

[ngook.kong]

Here is our progress:

• 26,659 of 26,668 (99.9%) certificates have been revoked or expired.
  • 26,464 revoked before expiration.
  • 195 expired before revocation.
• 1 certificates have been re-issued with revocation pending.
• 942 out of 944 customer accounts (99.8%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Below is the information for the remaining subscribers, we are working with each subscriber to accelerate the time to revocation for the remaining 9 certificates.

crt.sh link	Due Date
https://crt.sh/?sha256=FA1A87488B3FB3537D014EEBA171011EDD7F08D2A3E43F1C227C0CF470E4403E	2024-06-06
https://crt.sh/?sha256=46960B91CB3422343F7F6407255D150A0520CC053529614675DE4634921A91B5	2024-06-06
https://crt.sh/?sha256=FABDCA5071477AE28B3789630A5E0E0AF0E9D56EC4DCB1525CE55E919E1A7D68	2024-06-06
https://crt.sh/?sha256=4DFDE8E7E4D4510F8C3E64273669C6BE6D356DE5916FF72911BFF3E4A8BCA8EE	2024-06-06
https://crt.sh/?sha256=FCB7F4F440236BA3CA7DF8EECC2E82673E97919B884B384621C50CBD466E80B5	2024-06-16
https://crt.sh/?sha256=121FFC5063A1AE96B66884388E6894FEF17F2F5964A00BD5C1BAFEE0D84CD3D4	2024-06-16
https://crt.sh/?sha256=77D93210505C73CAC1BFD4F1014533F69891E717BC30FA0F9B7CF360FFF2534D	2024-06-16
https://crt.sh/?sha256=E2E3E23C22558131E9BD7ACFCF15FFA1FBBC2F529AA3FAB21AD220EF93436AD7	2024-06-16
https://crt.sh/?sha256=42CE9531C187834E74B07495552F553129DD69C105D526232B6BC43CEF4733BA	2024-06-06

Comment 44

[walter.j.marks]

(In reply to ngook.kong from comment #43)

Here is our progress:

• 26,659 of 26,668 (99.9%) certificates have been revoked or expired.
  • 26,464 revoked before expiration.
  • 195 expired before revocation.
• 1 certificates have been re-issued with revocation pending.
• 942 out of 944 customer accounts (99.8%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Below is the information for the remaining subscribers, we are working with each subscriber to accelerate the time to revocation for the remaining 9 certificates.

crt.sh link	Due Date
https://crt.sh/?sha256=FA1A87488B3FB3537D014EEBA171011EDD7F08D2A3E43F1C227C0CF470E4403E	2024-06-06
https://crt.sh/?sha256=46960B91CB3422343F7F6407255D150A0520CC053529614675DE4634921A91B5	2024-06-06
https://crt.sh/?sha256=FABDCA5071477AE28B3789630A5E0E0AF0E9D56EC4DCB1525CE55E919E1A7D68	2024-06-06
https://crt.sh/?sha256=4DFDE8E7E4D4510F8C3E64273669C6BE6D356DE5916FF72911BFF3E4A8BCA8EE	2024-06-06
https://crt.sh/?sha256=FCB7F4F440236BA3CA7DF8EECC2E82673E97919B884B384621C50CBD466E80B5	2024-06-16
https://crt.sh/?sha256=121FFC5063A1AE96B66884388E6894FEF17F2F5964A00BD5C1BAFEE0D84CD3D4	2024-06-16
https://crt.sh/?sha256=77D93210505C73CAC1BFD4F1014533F69891E717BC30FA0F9B7CF360FFF2534D	2024-06-16
https://crt.sh/?sha256=E2E3E23C22558131E9BD7ACFCF15FFA1FBBC2F529AA3FAB21AD220EF93436AD7	2024-06-16
https://crt.sh/?sha256=42CE9531C187834E74B07495552F553129DD69C105D526232B6BC43CEF4733BA	2024-06-06

Just to dig into this a bit further, the (already against the BR required timeline) of the original due date you listed for these, varying between May 30 and June 2 was agreed upon with the relevant subscriber, correct?

If that's so, then why are some subscribers now in need of an additional 2 weeks? Even if we're extremely generous and say that the 24h/120h revocation period starts from the moment you posted the update, we're still looking at 4 certs that will be an entire week late, even with the most generous interpretation of starting the revocation deadline at right now. If we actually use when these certificates were known to be misissued, we're looking at nearly 90 days by my math.

Additionally, my questions in comment #37 have not been answered yet, and I have an additional correction:

Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not that? In other words, you already have a binding legal agreement with your customer that Entrust is entitled to revoke a certificate with zero notice (and is required to by the BRs and various root programs). I believe what was being asked there is "what additional commitments are needed from a Subscriber beyond what they have already agreed to upon purchase of a certificate?".

Comment 45

[Wayne]

The concept of "critical infrastructure" can be complex, but there are some definitions such as in CISA and the EU CER and NIS-2, which provide helpful guidance. There's a spectrum of risk within critical infrastructure. Activities directly involved in life-or-death scenarios (e.g., controlling air traffic) fall under our prohibited use cases. However, there is a range of critical infrastructure where the use of certificates is not prohibited.

I explicitly asked which definition Entrust operate under, the lack of commitment to any shows it is none.

For instance, airline ticketing and check-in systems wouldn't be classified as a prohibited use case. While an outage caused by certificate revocation could lead to serious negative impacts for air travelers, it wouldn't directly risk human life. Irresponsible revocation without considering potential consequences could lead to widespread disruption, impacting millions of travelers and causing significant economic harm. This highlights the importance of our collaborative approach with subscribers to ensure a smooth re-issuance process and minimize disruption.

You cannot have a "collaborative approach with subscribers" whose use-case includes revocation that could "lead to widespread disruption, impacting millions of travelers and causing significant economic harm".

How is this at all in-keeping with the Prohibited Certificate Usage against any usage "in conjunction with any application that could lead to death, personal injury or severe physical or property damage"?

All Entrust have proven is that they are knowingly mis-issuing certificates to subscribers.

We have learned that subscribers often confuse "critical infrastructure" with “mission critical infrastructure”, this is why we are committed to improving our processes and subscriber education around revocation and are trying to further bridge the gap between perceived and real impact. Through collaborative risk assessments based on detailed questionnaires, we could more objectively assess true disruption in real critical infrastructure.

Entrust misunderstand their role in this situation, it is not to play nice with their clients and handhold them through delayed revocation reasons. It is to ensure the certificate is revoked as soon as possible and to make sure this will never happen again. This is a compliance matter, we need concrete answers as soon as possible not a list of action items that talk around the incident.

Repeated claims about subscriber confusion is deflecting the blame onto subscribers. It is Entrust, the CA, who is at fault for having issued certificates to them in the first place as it is against their own Prohibited Certificate Usage clauses.

If Entrust wish to address the gap between perceived and real impact they should do so with these alleged 'fire drills' they keep mentioning and perform them against subscribers who are noteworthy in "their" delayed responses to actual incidents. I will acknowledge this is once again Entrust trying to deflect responsibility onto the subscriber, further refusing to hold any responsibility to uphold revocation deadlines as required.

As stated in Comment 17, most of the remaining certificates were revoked on or before May 30, with some limited exceptions expected beyond that date. Another batch was revoked on the first of June and currently only 9 certificates belonging to 4 subscribers remain in terms of pending revocation.

Why are Entrust still providing "limited exceptions" 2 months into a delayed revocation failure? That deadlines were placed for May 30th, sailed past, and only noted in this incident after I checked is concerning. The drop of any concrete "deadlines" against the remaining subscribers shows an unwillingness to enforce the baseline requirements and revocation policies that Entrust alleges it is commited to upholding. It is concerning the lack of transparency at this point for any further delays and reflects upon Entrust's willingness to hold itself accountable.

As stated above, 9 certificates are remaining.

And as stated in Comment 18:

We are walking into wilful non-compliance in this aspect too.

(In reply to ngook.kong from comment #43)

Here is our progress:

• 26,659 of 26,668 (99.9%) certificates have been revoked or expired.
  • 26,464 revoked before expiration.
  • 195 expired before revocation.
• 1 certificates have been re-issued with revocation pending.
• 942 out of 944 customer accounts (99.8%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Below is the information for the remaining subscribers, we are working with each subscriber to accelerate the time to revocation for the remaining 9 certificates.

crt.sh link	Due Date
https://crt.sh/?sha256=FA1A87488B3FB3537D014EEBA171011EDD7F08D2A3E43F1C227C0CF470E4403E	2024-06-06
https://crt.sh/?sha256=46960B91CB3422343F7F6407255D150A0520CC053529614675DE4634921A91B5	2024-06-06
https://crt.sh/?sha256=FABDCA5071477AE28B3789630A5E0E0AF0E9D56EC4DCB1525CE55E919E1A7D68	2024-06-06
https://crt.sh/?sha256=4DFDE8E7E4D4510F8C3E64273669C6BE6D356DE5916FF72911BFF3E4A8BCA8EE	2024-06-06
https://crt.sh/?sha256=FCB7F4F440236BA3CA7DF8EECC2E82673E97919B884B384621C50CBD466E80B5	2024-06-16
https://crt.sh/?sha256=121FFC5063A1AE96B66884388E6894FEF17F2F5964A00BD5C1BAFEE0D84CD3D4	2024-06-16
https://crt.sh/?sha256=77D93210505C73CAC1BFD4F1014533F69891E717BC30FA0F9B7CF360FFF2534D	2024-06-16
https://crt.sh/?sha256=E2E3E23C22558131E9BD7ACFCF15FFA1FBBC2F529AA3FAB21AD220EF93436AD7	2024-06-16
https://crt.sh/?sha256=42CE9531C187834E74B07495552F553129DD69C105D526232B6BC43CEF4733BA	2024-06-06

Are Entrust capable of giving any reason for these delays? Should we prepare ourselves for a further delay once these are missed?

I put forward that these certificates should be included in a revocation list outside of Entrust's control by the proposed due dates at the latest, e.g. OneCRL.

Comment 46

[Wayne]

Confirming that all outstanding certificates have now been revoked. This incident took 81 days to resolve.

Cert	Revocation Date
FABDCA5071477AE28B3789630A5E0E0AF0E9D56EC4DCB1525CE55E919E1A7D68	2024-06-05 04:50:17 UTC
4DFDE8E7E4D4510F8C3E64273669C6BE6D356DE5916FF72911BFF3E4A8BCA8EE	2024-06-05 04:50:18 UTC
42CE9531C187834E74B07495552F553129DD69C105D526232B6BC43CEF4733BA	2024-06-05 04:50:20 UTC
FA1A87488B3FB3537D014EEBA171011EDD7F08D2A3E43F1C227C0CF470E4403E	2024-06-06 07:16:23 UTC
46960B91CB3422343F7F6407255D150A0520CC053529614675DE4634921A91B5	2024-06-06 07:16:23 UTC
77D93210505C73CAC1BFD4F1014533F69891E717BC30FA0F9B7CF360FFF2534D	2024-06-06 14:04:33 UTC
E2E3E23C22558131E9BD7ACFCF15FFA1FBBC2F529AA3FAB21AD220EF93436AD7	2024-06-06 14:55:24 UTC
121FFC5063A1AE96B66884388E6894FEF17F2F5964A00BD5C1BAFEE0D84CD3D4	2024-06-06 15:48:34 UTC
FCB7F4F440236BA3CA7DF8EECC2E82673E97919B884B384621C50CBD466E80B5	2024-06-06 20:04:06 UTC

Comment 47

[Wayne]

Apologies, mis-issuance was confirmed 03-04 as per 1883843 therefore this incident took 95 days to resolve.

Comment 48

[Bruce Morton]

(In reply to walter.j.marks from comment #37)

Setting aside the initial failure to revoke and the long tail of this process, I'd like to dig a bit further into the Subscriber arguments against revoking, and some of your statements included there. I'm going to make a few assumptions here, including that Entrust agrees that there is in fact a violation of the BRs requiring revocation of affected certs, otherwise I assume that you wouldn't have revoked in the first place.

“These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

“Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

“We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Q1. Were each of these Subscribers reminded of their agreement to the Terms of Use of Entrust Certificate and Signing Services? Specifically, Exhibit B, Part 1.12.3?

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

We have not reminded subscribers about this section of our Terms of Use, nor have subscribers raised questions about our contractual rights in this matter.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Q2. Why was a presumption of a Subscriber being unable to revoke without positive confirmation assumed as stated above? Again following from Exhibit B, Part 1.12.3, Entrust is entitled to revoke a certificate immediately without notifying the Subscriber.

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Q3. How does the above statement square with the the subscriber justification given here?

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

“Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Again, you seem to misunderstand the question posed by the following quote:

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

We understand the question. As stated in your quote, we tried to balance the risk of the relying parties in the broader internet and the subscriber relying parties.

Of those three exemplar Subscribers I've referenced above, none seem to have a lack of automation as a blocking fact that they were unable to revoke (which again, should be an action taken by Subscribers by the powers given to you by Exhibit B, Part 1.12.3) and replace in time. A commitment to automation does not mean anything if the subscribers are simply refusing to accept a revocation within the required and expected time. A more accurate commitment that I would expect to answer the question would be something along the lines of "Entrust will refuse to accept any requests to delay revocation past 5 days....." along with a measurable and actionable key index for that commitment.
Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

We agree that a key outcome for this incident is for us to define a more rigorous revocation plan that includes a formal plan to define our policies and process for handling subscriber requests for delayed revocation. We expect to have these guidelines in place by the end of June.

Q5.1. Of these three exemplar Subscribers, were any reminded that if this was a key compromise event that had fundamental security risks, that this argument of "needing to notify downstream contracted parties that could take up to 3 months" (paraphrased) would potentially result in their TLS certificates being unsafe for the web ecosystem and their business operations for the entire 90 day period?

Yes, revocation requirements were communicated to customers.

Q5.2. If so, what was the justification provided by the Subscriber that they would be able to handle a situation like that within the 24h/120h period, and why that process was unable to be performed in this case, given that from a pure compliance perspective, in either case the affected certificate is invalid and should not be used by the Subscriber?

Subscriber feedback indicates that many subscribers non security events differently from security related events. In the event of a key compromise, we believe that our customers take security very seriously across their organizations, and subscribers understand the potential consequences for themselves and the web ecosystem.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Q6. The above statement seems to indicate that Entrust believes that this incident is still not at the level of a revocation requirement. If that is the case, please explain how the processes followed for this mis-issuance are different from the planned handling of a wide-scale revocation requirement.

We have agreed that this was a mis-issuance event and have reiterated our intent to follow the TLS Baseline Requirements and root program policies. We have acknowledged that these certificates were mis-issued, and initiated revocation. To date all certificates in this incident have been revoked.

Comment 49

[walter.j.marks]

(In reply to Bruce Morton from comment #48)

(In reply to walter.j.marks from comment #37)

Setting aside the initial failure to revoke and the long tail of this process, I'd like to dig a bit further into the Subscriber arguments against revoking, and some of your statements included there. I'm going to make a few assumptions here, including that Entrust agrees that there is in fact a violation of the BRs requiring revocation of affected certs, otherwise I assume that you wouldn't have revoked in the first place.

“These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

“Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

“We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Q1. Were each of these Subscribers reminded of their agreement to the Terms of Use of Entrust Certificate and Signing Services? Specifically, Exhibit B, Part 1.12.3?

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

We have not reminded subscribers about this section of our Terms of Use, nor have subscribers raised questions about our contractual rights in this matter.

Since it wasn't mentioned in the June 7 Report (along with other omissions but I'll leave those opinions for MDSP to avoid mixing up action items here), what was the internal decision making that led to not reminding subscribers of this section of the ToU?

A lot of this aftermath and the ensuing incident failures that were due to "Entrust handling numerous incidents at the same time" (paraphrased) could have been mitigated or reduced by "We are required to revoke certificates within 24h/120h. We are here to assist you in any way we can during the next few days, but as part of our agreement to PKI as a whole as well as the BRs and various root programs, it is mandatory that these revocations occur."

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Q2. Why was a presumption of a Subscriber being unable to revoke without positive confirmation assumed as stated above? Again following from Exhibit B, Part 1.12.3, Entrust is entitled to revoke a certificate immediately without notifying the Subscriber.

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Falls under my thoughts from the above un-quoted statement.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Q3. How does the above statement square with the the subscriber justification given here?

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

“Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Again, you seem to misunderstand the question posed by the following quote:

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

We understand the question. As stated in your quote, we tried to balance the risk of the relying parties in the broader internet and the subscriber relying parties.

Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

Of those three exemplar Subscribers I've referenced above, none seem to have a lack of automation as a blocking fact that they were unable to revoke (which again, should be an action taken by Subscribers by the powers given to you by Exhibit B, Part 1.12.3) and replace in time. A commitment to automation does not mean anything if the subscribers are simply refusing to accept a revocation within the required and expected time. A more accurate commitment that I would expect to answer the question would be something along the lines of "Entrust will refuse to accept any requests to delay revocation past 5 days....." along with a measurable and actionable key index for that commitment.
Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

We agree that a key outcome for this incident is for us to define a more rigorous revocation plan that includes a formal plan to define our policies and process for handling subscriber requests for delayed revocation. We expect to have these guidelines in place by the end of June.

There was a commitment to never delay revocation again after a delrev incident 4 years ago. What happened to those guidelines and internal policies then, and why were they not followed now? If they did exist prior to the June 7 Report, why do these policies need additional time to be created and implemented.

Q5.1. Of these three exemplar Subscribers, were any reminded that if this was a key compromise event that had fundamental security risks, that this argument of "needing to notify downstream contracted parties that could take up to 3 months" (paraphrased) would potentially result in their TLS certificates being unsafe for the web ecosystem and their business operations for the entire 90 day period?

Yes, revocation requirements were communicated to customers.

Q5.2. If so, what was the justification provided by the Subscriber that they would be able to handle a situation like that within the 24h/120h period, and why that process was unable to be performed in this case, given that from a pure compliance perspective, in either case the affected certificate is invalid and should not be used by the Subscriber?

Subscriber feedback indicates that many subscribers non security events differently from security related events. In the event of a key compromise, we believe that our customers take security very seriously across their organizations, and subscribers understand the potential consequences for themselves and the web ecosystem.

The response here would be most appropriate in MDSP.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Q6. The above statement seems to indicate that Entrust believes that this incident is still not at the level of a revocation requirement. If that is the case, please explain how the processes followed for this mis-issuance are different from the planned handling of a wide-scale revocation requirement.

We have agreed that this was a mis-issuance event and have reiterated our intent to follow the TLS Baseline Requirements and root program policies. We have acknowledged that these certificates were mis-issued, and initiated revocation. To date all certificates in this incident have been revoked.

If you have reiterated your intent to follow the BR and root program policies, why are we in a situation where we can expect Entrust to delay revocation of certificates for 90 days as evidenced by this bug? I don't want to be flippant, but there's really no other way to square that circle. You cannot in good faith say "Entrust will follow the policies" in the exact same bug number as where there's proof that a known mis-issued certificate remained unrevoked for 95 days, especially since 4 years ago this statement was given by Bruce Morton:

• We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

and yet somehow we ended up in the exact same situation. In other words, I see more evidence that I should not be taking this commitment seriously than evidence that I should take this commitment seriously.

Comment 50

[ngook.kong]

Attachment: Original EV Notification 2024.03.18.pdf

Comment 51

[ngook.kong]

(In reply to Wayne from comment #40)

(In reply to ngook.kong from comment #36)

Criteria: Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem, we did not grant their delayed revocation requests.
Who made the decisions: Our product compliance leader initially reviewed and accepted the requests for delayed revocation. As the request volume escalated, a cross functional senior leadership team, advised by our product compliance team was activated for such decisions.
How: Our Support and Customer Success teams were empowered to communicate with Subscribers and were instructed not to approve requests. Some of these conversations occurred via phone or video conference and the articulated rationale was not captured in the very early days of the response.

Ultimately, a formal process was stood up to capture all requests in more detail. This is a key reason we are exploring the use of a centralized request form and/or formal tracking mechanism to ensure all this information is captured in a central location and easily reportable. We value input from the community on this.

Can we take a step back from this rationale and just talk numbers:

• How many requests came in (approx), how many were initially denied? I understand a longer conversation then occurred, but give us an idea of the initial first week of Entrust taking action.
• What was the minimum and maximum length of time given to a subscriber to delay originally?
• With those subscribers as an anonymous basis, when were they ultimately revoked?
• Given deadlines would have been presented were hit, how many hard revocations occurred?
• How many subscribers manually revoked, and how many did so via their deadline expiring but after a new certificate was issued?

Now I'll be frank - I do not expect perfection on this front, this is where being transparent is helpful. There will be cases of subscribers going past the deadline and the case not being immediately handled, what I am looking for is a sense of scale.

Appreciate the question; as we have noted, because of the volume, we did not always receive or record delayed revocation requests effectively for all customers, and we have identified this as an area of process improvement. Going forward, if another revocation event occurs, we will focus on reducing the number of delay requests, standardizing the intake and review process for delay requests, and communicating both the date and time of when affected certificates will be revoked.

To give you a sense of scale in this incident, however, we captured roughly 292 extension requests from subscribers coming directly out of the original communication; 277 (~98%) of these requests were for 6 weeks or less. Nearly 2/3 of these (63%) were approved for 30-day extensions – the rest were between 1 and 4 weeks. We approved a shorter extension than requested in 8 cases.

Other requests occurred during phone or email contact with subscribers during and after that first week. Across all subscribers, at just over 30 days (4/19) -- about 62% of subscribers had completed revocation. At just over 6 weeks (5/2), 91.7% of subscribers had completed revocation; and as of 6/7 all affected certificates were either expired or had been revoked.

The largest subscribers with the most complex environments requested delays of 60-90 days or more. These included banks involved in critical parts of the financial infrastructure where their internal processes or dependencies on third parties impacted their ability to revoke the affected certificates. We worked with all of them to accelerate the revocation process and often pushed back against additional extension requests.

While this does not specifically answer all of your questions, we hope this gives you a sense of the scale of the process.

Please also give examples of Subscribers that requested an extension and were not granted one.

Following are 2 examples of supplied reasons where the request was denied.

If subscribers provided little to no information using terms like “unknown” or “confidential” when asked to provide a rational for delayed revocation.
If subscribers provided reason that they had other critical business priorities and could not spare the resources

This answer is not clear. Were these subscribers then subsequently hard revoked after their initial request was denied? You are not identifying any specific subscribers, and that is okay and should free you for answering this question.

We revoked any hard remaining mis-issued certificates once a delayed revocation date had expired.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Do you mean that there was always a response from the subscriber prior to revocation?

Yes, we confirmed that our communication reached all subscribers prior to revoking. We wanted to ensure that no subscribers' certificates were revoked without their having viewed the message, understanding that we have the right to do this without this assurance. It took us longer than we would have liked, and that is an area of improvement in our efforts to minimize disruption in the event of a non-security or security-related issue.

Here are some specific examples of rationales we received from Subscribers:

· “These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

· “Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

· “We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

We can agree that none of these are discussing critical infrastructure correct? Inconvenient, busy, change freeze. None of these reflect any actual harm and instead reflect an indifference to the situation as explained by Entrust to their subscribers. That there are subscribers who have stated they have a 3-month minimum turnaround and yet it hasn't been 3 months shows that you need to actually enforce your duty to revoke. Ultimately they are stating they are incapable of handling revocation in the event of a 24 hour event and the continued issuance to them is introducing harm to the ecosystem.

Actually, the examples above represent organizations where major outages and disruptions would have had an impact on the operation of the web ecosystem.

Many of our subscribers have recognized that their internal readiness to re-issue and revoke on a short turnaround time needs to be greatly enhanced and this incident has brought sufficient attention in their organizations. We are engaging further with all subscribers who required delayed revocations during this incident to work with them on enhancing their process and systems for resilience and faster revocation (in accordance with CA/B guidelines), should rapid revocation be required in the future.

Agreement from subscribers to implement solutions like Certificate Lifecycle Management and/or automation capabilities like those found in ACME Renewal Information, as well as moving to private CAs where appropriate, will be critical to the success of this effort.

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

· “Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Given you are not identifying this subscriber can you clarify what was given by 'some time to delay'? As an anonymous subscriber, could a breakdown be provided of roughly the days of each communication and subsequent shift in the revocation window?

This subscriber requested an extension of June 30, 2024; however, based on the justification provided, the subscriber was only granted an extension of May 16, 2024.

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

Automation will not save any of the above subscribers, this is not the time for a sales pitch to them but in showing how you handle compliance. The subscribers are already legally committed to accept immediate revocation, it is on Entrust to actually enforce. The failure to do so reflects on how seriously Entrust have taken this situation.

We agree that it is critical to ensure that our subscribers are aware of our responsibilities – and theirs – under the TLS Baseline Requirements and our terms and conditions. We are taking this incident as an opportunity to remind and educate our subscribers about their responsibilities and the CA/B Forum requirements.

Please describe in detail how you analyze this balance.

We would only use such a form with Subscribers if they understood there is a presumption of denial.

We start with a presumption of denial, but there are circumstances that justify some period of delay.
We have used as precedence the approach and other CAs have also allowed for delayed revocation on the same basis.
We understand and agree that delayed revocation requests are the exception and not the rule and that is how we approached this with our Subscribers. We are committed to continuous improvement of this process.

What we need to hear is "this will never happen again", not "continuous improvement".

Yes and it was communicated exactly as such. No delayed revocation options were offered.

To clarify, no delayed revocation options were offered - therefore any specific dates were agreed on a subscriber-by-subscriber basis? Surely there were some benchmark numbers for each support team member to work with? This should have been treated as an immediate emergency by all parties and any date provided should have been by end of business day.

See our answer to your first question in this comment. Most subscribers who requested delayed revocation were granted extensions of 30 days or less; requests from larger, more complex organizations were handled on a subscriber-by-subscriber basis.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Do you feel that if Entrust were to be required to perform a wide-scale revocation it would be done within 24 hours? Even just this limited sample of 24k certificates? The reliance on subscribers manually taking action and making a per-subscriber deal on when it would be convenient for them says otherwise.

Yes, we are equipped to perform a wide scale revocation if needed and necessary.

If this incident were to happen again with the same subscribers tomorrow, would we be complete within 5 days?

We have the technical capability to revoke within the required timelines. Our learning from this incident is that we need to meaningfully harden our internal processes to rapidly gather pertinent justification from our subscribers on certificates that meet the policy for delayed revocation. This process hardening will be actioned as part of improvements we are committing to and would greatly benefit from the community’s feedback.

(In reply to ngook.kong from comment #38)

Steps: [REDACTED]

Any particular reason Entrust have decided to redact this? I have not been subtle on this already being known, so I'll give an example of what was being sent to subscribers in this incident:

Perform a REISSUE, and select "Revoke within 30 days" so your production certificate maintains validity, providing sufficient time to perform the replacement

This is from an example in the public domain that I'll identify privately to a Root Program (my email is always open). That you're still not being transparent on this front is not improving the level of trust. Given the above process and it already being a bit more than 30 days, what actually happened?

The letter we shared is an example of what was sent from us directly to a subscriber and was not posted in the public domain. We were being transparent by sharing the message. The redacted section provides specific instructions to our subscribers on how to revoke and reissue certificates. A full PDF of the message is attached for reference (attachment #9406229 [details]).
“Revoke within 30 days” was an option to ensure subscribers had time to replace their certificates before they were fully revoked. Certificates placed in this status were revoked within 30 days of when they were placed in this status; we revoked them sooner if their extension time was reached, or if the subscriber confirmed they had reissued.

What is a 'certain date'? If you email me on 04-01, I don't respond by your deadline of say 04-06 then do you start a 5-day clock ending on 04-11? Of the Subscribers in this incident how many have not replied, and what timeframes were provided? How many ended up revoked due to this?

We have been following up with Subscribers by email and by phone leveraging our Support, Sales, Customer Success Managers and their teams. Customer timeframes are discussed between our compliance team and senior leadership team.

As noted in response to Comment 29, we were ultimately able to communicate with all Subscribers. All Subscribers were reached before the date we set internally to hard revoke all remaining affected certificates.

As the above should make clear I was very specific in my question for a reason. The email stated revocation would occur Saturday, March 23, 2024 at 21:00 UTC. Given that, did every subscriber respond by that deadline? The statements so far are only that communication was possible with every subscriber across an ill-defined timeframe, the specificity matters.

1. Every subscriber was contacted with the letter
2. Not all subscribers responded by the date
3. We decided not to revoke if we did not see revocation progress, and/or no confirmed contact (either an email reply or talking to someone directly). The intent was to avoid unintended harmful consequences. As expressed before, improving this process is a key learning and area of improvement we are committed to and welcome your feedback accordingly.

Currently we are being informed that a hard deadline was stated by Entrust, but that no certificates were revoked due to this. Is that the case?

If we had contact with the subscriber, and their extension expired, we revoked their certificates. We sent final reminders with the date and time of revocation in UTC and then started revocation during that time. In some individual cases, subscribers realized that certain certificates would require more time and requested short additional extensions for only those certificates.

At the time that we collected the information, there was no reasonable expectation either by Entrust or the Subscribers that every detail of the communication would be public information. Due to standard NDA provisions between Entrust and its Subscribers, the Subscribers provided some information that we do not feel at liberty to share, nor do we feel that those details are necessary to addressing the issues identified in this incident. We believe the formatted reasons we provided are sufficient and the discussion around a standard form is a more productive path to making improvements going forward.

In Comment 36 there is more specificity included than my question asks for. I am not asking for every subscriber's full reason, my point is that during this a generalized report would have been generated that gives an overall picture of their justifications. As part of the response to Mozilla I would recommend including it to fully explain what has occurred.

Any statements about a standard form for delayed revocation is truly misunderstanding the situation. This is not supposed to be standard at all, that there is more than one is already a serious situation. All that statement is saying to me, is that Entrust is comfortable that this will happen again and wish to have a letter than can be used as an excuse for later improvements. My MDSP post is about the initial revocation correspondence for a reason - that is all that should occur. A fixed deadline that is enforced, no exceptions.

Yes, we have made commitments to improve our incident response process. Some improvements we can make unilaterally, and other improvements would benefit from discussion and agreement from the community. In our response above, we indicated that the use of a standard form for incidents could be useful, and we think discussion and agreement from the community would be particularly applicable here.

In keeping with the above it is essential that a distinction is made between what is made standard. Incidents will occur, the important part is communication and enforcement of policy. To that end I'll repeat one of my questions that got lost:

Entrust are aware they are not the entity that should balance risks? It is their duty to enforce their revocation policies and it is the subscriber who should be performing any risk calculus per the subscriber agreement, correct?

You are correct, subscribers should be aware of their responsibilities and risks. They should perform risk assessments and understand the potential consequences of design and implementation decisions on their operations. And when they deploy public certificates with a CA, they should understand what is required of the CA in the event of a mis-issuance – and their role and impact in remediation. As we have previously stated, we will also be working with our subscribers on their use cases and explore options for private PKI offerings where appropriate.

Many delayed revocation incidents indicate that some subscribers are not prepared at the level we would like them to be. We are balancing the risks and believe it is important to continue engaging proactively with subscribers so that following the BRs and minimizing disruption when there is a revocation are not in conflict.

Comment 52

[walter.j.marks]

(In reply to ngook.kong from comment #51)

(In reply to Wayne from comment #40)

1. Every subscriber was contacted with the letter
2. Not all subscribers responded by the date
3. We decided not to revoke if we did not see revocation progress, and/or no confirmed contact (either an email reply or talking to someone directly). The intent was to avoid unintended harmful consequences. As expressed before, improving this process is a key learning and area of improvement we are committed to and welcome your feedback accordingly.

Just to specifically dig into this, as I touched on it above and this is seemingly a tacit admission of what I was poking at above:

Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

If you do not believe that Subscribers were prepared for the revocation (and the "harmful consequences"), you logically most also not be holding them to Exhibit B, Part 1.12.3 of your Terms of Use, quoted again for your convenience:

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

Question: If so, what other portions of the Terms of Use are flexible in Entrust's enforcement of the Terms of Use?

We decided not to revoke if we did not see revocation progress,

Question: Without naming any specific Entrust employees, is there a specific Directly Responsible Individual (or Individuals) who made the decision to not revoke (the "we decided") without affirmative consent from the Subscriber, and is that noted in your internal history of this incident?

Question: If so, did this DRI make the decision wholesale, or was the decision made per subscriber?

Comment 53

[Martijn Katerbarg]

Hi Ngook,

Could you please help me understand something?

In comment #0 Entrust wrote:

We expect further revocation delays, details with customer feedback to follow, initial feedback includes:
Uncapable to deal with high volumes of manual re-issuance
Lockdown because of end of the fiscal year/quarter
Lockdown due to easter holidays

In comment #1 Entrust wrote (emphasis mine):

We have experienced significant pushback from customers on the feasibility of revocation timelines. The revocation for customers is in most cases manual and complex, involving multiple internal parties to ensure that the change does not create an adverse impact on web services and back-end processing.
As a result, we have moved these customers into delayed revocation, particularly where their operations are critical to the web ecosystem (e.g., banks, payment networks, airlines, and government agencies).

In comment #28 Entrust wrote:

We communicated with subscribers via email, video calls, phone calls, and through support ticket dialogue to ask them to justify their need for delayed revocation. Specifically, we asked them to detail their technical challenges and/or impacts to the web ecosystem if forced to revoke within 5 days as well as how much time they needed to replace and revoke their certificates.
Because of the volume, we did not always receive or record this information effectively for all customers. We have identified this as an area of process improvement. Going forward, we are looking at the use of a standardized way to collect and record this data to ensure that if there are future mis-issuances, the collection of this information is more structured and complete.

In comment #36 Entrust wrote (emphasis mine):

Criteria: Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem, we did not grant their delayed revocation requests.

In comment #48 Entrust wrote (emphasis mine):

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Additionally, in several posts Entrust also wrote an update of their revocation progress. Nowhere there did it state, or show progress, of how many customers were contacted, and thus moved into the revocation phase.

All of the above statements are related to notifications that were sent out, and that only customers that specifically requested a delayed revocation due to “business/industry disruption, technical challenges and/or impacts to the web ecosystem” were granted a delay.

Yet, now in comment #51, Entrust states:

1. Every subscriber was contacted with the letter
2. Not all subscribers responded by the date
3. We decided not to revoke if we did not see revocation progress, and/or no confirmed contact (either an email reply or talking to someone directly). The intent was to avoid unintended harmful consequences. As expressed before, improving this process is a key learning and area of improvement we are committed to and welcome your feedback accordingly.

Can you please explain:

• The discrepancies between these statements, where previously you have stated that you did not grant any delayed revocation “Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem”, yet now you state you automatically granted a delay if you were unable to reach the subscriber?
• If you used the same approach as described in the quoted section from comment #51 in other delayed revocation bugs?
• Why it took Entrust up to now to come out with this information?

If it was not implied, there is a big difference between granting (very limited) delayed revocation in exceptional circumstances and blindly just not revoking certificates if you were unable to reach your customers or confirm that they are taking action.

Comment 54

[ngook.kong]

(In reply to walter.j.marks from comment #44)

(In reply to ngook.kong from comment #43)

Here is our progress:

• 26,659 of 26,668 (99.9%) certificates have been revoked or expired.
  • 26,464 revoked before expiration.
  • 195 expired before revocation.
• 1 certificates have been re-issued with revocation pending.
• 942 out of 944 customer accounts (99.8%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Below is the information for the remaining subscribers, we are working with each subscriber to accelerate the time to revocation for the remaining 9 certificates.

crt.sh link	Due Date
https://crt.sh/?sha256=FA1A87488B3FB3537D014EEBA171011EDD7F08D2A3E43F1C227C0CF470E4403E	2024-06-06
https://crt.sh/?sha256=46960B91CB3422343F7F6407255D150A0520CC053529614675DE4634921A91B5	2024-06-06
https://crt.sh/?sha256=FABDCA5071477AE28B3789630A5E0E0AF0E9D56EC4DCB1525CE55E919E1A7D68	2024-06-06
https://crt.sh/?sha256=4DFDE8E7E4D4510F8C3E64273669C6BE6D356DE5916FF72911BFF3E4A8BCA8EE	2024-06-06
https://crt.sh/?sha256=FCB7F4F440236BA3CA7DF8EECC2E82673E97919B884B384621C50CBD466E80B5	2024-06-16
https://crt.sh/?sha256=121FFC5063A1AE96B66884388E6894FEF17F2F5964A00BD5C1BAFEE0D84CD3D4	2024-06-16
https://crt.sh/?sha256=77D93210505C73CAC1BFD4F1014533F69891E717BC30FA0F9B7CF360FFF2534D	2024-06-16
https://crt.sh/?sha256=E2E3E23C22558131E9BD7ACFCF15FFA1FBBC2F529AA3FAB21AD220EF93436AD7	2024-06-16
https://crt.sh/?sha256=42CE9531C187834E74B07495552F553129DD69C105D526232B6BC43CEF4733BA	2024-06-06

Just to dig into this a bit further, the (already against the BR required timeline) of the original due date you listed for these, varying between May 30 and June 2 was agreed upon with the relevant subscriber, correct?

If that's so, then why are some subscribers now in need of an additional 2 weeks? Even if we're extremely generous and say that the 24h/120h revocation period starts from the moment you posted the update, we're still looking at 4 certs that will be an entire week late, even with the most generous interpretation of starting the revocation deadline at right now. If we actually use when these certificates were known to be misissued, we're looking at nearly 90 days by my math.

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

Additionally, my questions in comment #37 have not been answered yet, and I have an additional correction:

Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not that? In other words, you already have a binding legal agreement with your customer that Entrust is entitled to revoke a certificate with zero notice (and is required to by the BRs and various root programs). I believe what was being asked there is "what additional commitments are needed from a Subscriber beyond what they have already agreed to upon purchase of a certificate?".

As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.

That is why we evaluated their delayed revocation requests and communicated with them to verify that impacted certificates could safely be revoked.

As noted above, we will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

Comment 55

[ngook.kong]

(In reply to Wayne from comment #45)

The concept of "critical infrastructure" can be complex, but there are some definitions such as in CISA and the EU CER and NIS-2, which provide helpful guidance. There's a spectrum of risk within critical infrastructure. Activities directly involved in life-or-death scenarios (e.g., controlling air traffic) fall under our prohibited use cases. However, there is a range of critical infrastructure where the use of certificates is not prohibited.

I explicitly asked which definition Entrust operate under, the lack of commitment to any shows it is none.

There is no single definition. Entrust looks at the definitions of critical infrastructure from US CISA, EU CER and NIS-2 in formulating its definition.

For instance, airline ticketing and check-in systems wouldn't be classified as a prohibited use case. While an outage caused by certificate revocation could lead to serious negative impacts for air travelers, it wouldn't directly risk human life. Irresponsible revocation without considering potential consequences could lead to widespread disruption, impacting millions of travelers and causing significant economic harm. This highlights the importance of our collaborative approach with subscribers to ensure a smooth re-issuance process and minimize disruption.

You cannot have a "collaborative approach with subscribers" whose use-case includes revocation that could "lead to widespread disruption, impacting millions of travelers and causing significant economic harm".

How is this at all in-keeping with the Prohibited Certificate Usage against any usage "in conjunction with any application that could lead to death, personal injury or severe physical or property damage"?

All Entrust have proven is that they are knowingly mis-issuing certificates to subscribers.

We do not issue certificates to subscribers for the uses prohibited in our CPS. We do, however, still have subscribers who operate in critical infrastructure sectors as defined by CISA, e.g., transportation and financial services.

We have learned that subscribers often confuse "critical infrastructure" with “mission critical infrastructure”, this is why we are committed to improving our processes and subscriber education around revocation and are trying to further bridge the gap between perceived and real impact. Through collaborative risk assessments based on detailed questionnaires, we could more objectively assess true disruption in real critical infrastructure.

Entrust misunderstand their role in this situation, it is not to play nice with their clients and handhold them through delayed revocation reasons. It is to ensure the certificate is revoked as soon as possible and to make sure this will never happen again. This is a compliance matter, we need concrete answers as soon as possible not a list of action items that talk around the incident.

Repeated claims about subscriber confusion is deflecting the blame onto subscribers. It is Entrust, the CA, who is at fault for having issued certificates to them in the first place as it is against their own Prohibited Certificate Usage clauses.

If Entrust wish to address the gap between perceived and real impact they should do so with these alleged 'fire drills' they keep mentioning and perform them against subscribers who are noteworthy in "their" delayed responses to actual incidents. I will acknowledge this is once again Entrust trying to deflect responsibility onto the subscriber, further refusing to hold any responsibility to uphold revocation deadlines as required.

As stated in Comment 17, most of the remaining certificates were revoked on or before May 30, with some limited exceptions expected beyond that date. Another batch was revoked on the first of June and currently only 9 certificates belonging to 4 subscribers remain in terms of pending revocation.

Why are Entrust still providing "limited exceptions" 2 months into a delayed revocation failure? That deadlines were placed for May 30th, sailed past, and only noted in this incident after I checked is concerning. The drop of any concrete "deadlines" against the remaining subscribers shows an unwillingness to enforce the baseline requirements and revocation policies that Entrust alleges it is commited to upholding. It is concerning the lack of transparency at this point for any further delays and reflects upon Entrust's willingness to hold itself accountable.

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

As stated above, 9 certificates are remaining.

And as stated in Comment 18:

We are walking into wilful non-compliance in this aspect too.

(In reply to ngook.kong from comment #43)

Here is our progress:

• 26,659 of 26,668 (99.9%) certificates have been revoked or expired.
  • 26,464 revoked before expiration.
  • 195 expired before revocation.
• 1 certificates have been re-issued with revocation pending.
• 942 out of 944 customer accounts (99.8%) have fully remediated the issue (certificates re-issued and old certificates revoked or expired).

Below is the information for the remaining subscribers, we are working with each subscriber to accelerate the time to revocation for the remaining 9 certificates.

crt.sh link	Due Date
https://crt.sh/?sha256=FA1A87488B3FB3537D014EEBA171011EDD7F08D2A3E43F1C227C0CF470E4403E	2024-06-06
https://crt.sh/?sha256=46960B91CB3422343F7F6407255D150A0520CC053529614675DE4634921A91B5	2024-06-06
https://crt.sh/?sha256=FABDCA5071477AE28B3789630A5E0E0AF0E9D56EC4DCB1525CE55E919E1A7D68	2024-06-06
https://crt.sh/?sha256=4DFDE8E7E4D4510F8C3E64273669C6BE6D356DE5916FF72911BFF3E4A8BCA8EE	2024-06-06
https://crt.sh/?sha256=FCB7F4F440236BA3CA7DF8EECC2E82673E97919B884B384621C50CBD466E80B5	2024-06-16
https://crt.sh/?sha256=121FFC5063A1AE96B66884388E6894FEF17F2F5964A00BD5C1BAFEE0D84CD3D4	2024-06-16
https://crt.sh/?sha256=77D93210505C73CAC1BFD4F1014533F69891E717BC30FA0F9B7CF360FFF2534D	2024-06-16
https://crt.sh/?sha256=E2E3E23C22558131E9BD7ACFCF15FFA1FBBC2F529AA3FAB21AD220EF93436AD7	2024-06-16
https://crt.sh/?sha256=42CE9531C187834E74B07495552F553129DD69C105D526232B6BC43CEF4733BA	2024-06-06

Are Entrust capable of giving any reason for these delays? Should we prepare ourselves for a further delay once these are missed?

All affected certificates were revoked by June 6, 2024.

I put forward that these certificates should be included in a revocation list outside of Entrust's control by the proposed due dates at the latest, e.g. OneCRL.

Comment 56

[boet]

(In reply to ngook.kong from comment #55)

The concept of "critical infrastructure" can be complex, but there are some definitions such as in CISA and the EU CER and NIS-2, which provide helpful guidance. There's a spectrum of risk within critical infrastructure. Activities directly involved in life-or-death scenarios (e.g., controlling air traffic) fall under our prohibited use cases. However, there is a range of critical infrastructure where the use of certificates is not prohibited.

I explicitly asked which definition Entrust operate under, the lack of commitment to any shows it is none.

There is no single definition. Entrust looks at the definitions of critical infrastructure from US CISA, EU CER and NIS-2 in formulating its definition.

Does Entrust explicitly formulate a definition ahead of time based on these sources, or is a working definition created ad-hoc when circumstances call for one?

Comment 57

[Wayne]

Given the lack of answers to questions all questions posed in this comment have been bolded for clarity.

(In reply to ngook.kong from comment 54)

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

To clarify Entrust's understanding the questions posed were:

Just to dig into this a bit further, the (already against the BR required timeline) of the original due date you listed for these, varying between May 30 and June 2 was agreed upon with the relevant subscriber, correct?

If that's so, then why are some subscribers now in need of an additional 2 weeks?

An answer to them would be preferable. The same goes for:

As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.

That is why we evaluated their delayed revocation requests and communicated with them to verify that impacted certificates could safely be revoked.

As noted above, we will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

The question posed was:

I believe what was being asked there is "what additional commitments are needed from a Subscriber beyond what they have already agreed to upon purchase of a certificate?"

I will then add on further questions:

• Can Entrust explain why it is irresponsible to revoke a certificate over 90 certs into an incident?
• What 'context' is being missed, and can that be provided per-subscriber?
• Yes or No: Will Entrust commit to not causing delayed revocation to go past, say, 30 days going forward?
• Provide a simple figure for the number of days that Entrust feel they are organizationally capable of handling revocation given their subscriber base?

Per their own Prohibited Certificate Usage terms, no such certificate should be used in a situation that can cause harm or serious economic impact. All authority has been bestowed onto Entrust in order to fulfill their revocation obligations, there is just an odd refusal to do so. If this could be explained at all I'd appreciate it.

(In reply to ngook.kong from comment 55)

There is no single definition. Entrust looks at the definitions of critical infrastructure from US CISA, EU CER and NIS-2 in formulating its definition.

If there is no single definition, then what did Entrust 'formulate' in its own definition, which you are claiming does not exist?

We do not issue certificates to subscribers for the uses prohibited in our CPS. We do, however, still have subscribers who operate in critical infrastructure sectors as defined by CISA, e.g., transportation and financial services.

Entrust are explicitly confirming that no subscribers were using their certificates "in conjunction with any application that could lead to death, personal injury or severe physical or property damage"?

Following that question, and given these are self-reported reasons provided by subscribers:

• What rationale was there in delaying revocation due to 'harm' and 'critical infrastructure'?
• Can Entrust document a single instance of them pushing back against these self-reported claims?
• Subscribers state they are in the above categories, does Entrust agree?
• We have a list of subscribers and these statements so please provide some examples

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

My question was not answered. I will reiterate it as Entrust seem to have a problem with answering questions lately:

• Why are Entrust still providing "limited exceptions" 2 months into a delayed revocation failure?

I am not asking what you are planning to put into policy going forward, but what happened historically in regards to this incident. I was hoping this would also be in your formal report, but I can see nothing like that.

All affected certificates were revoked by June 6, 2024.

Again my question has not been answered:

• Are Entrust capable of giving any reason for these delays?

I will note that the certificates were miraculously revoked by June 6th, and an explanation of what happened to get us into this situation is surely required.

Comment 58

[walter.j.marks]

As a reminder, my questions asked in Comment #49 remain unanswered at the 7 day mark.

Here they are bolded for your convenience:

• Since it wasn't mentioned in the June 7 Report (along with other omissions but I'll leave those opinions for MDSP to avoid mixing up action items here), what was the internal decision making that led to not reminding subscribers of this section of the ToU?
• Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?
• There was a commitment to never delay revocation again after a delrev incident 4 years ago. What happened to those guidelines and internal policies then, and why were they not followed now? If they did exist prior to the June 7 Report, why do these policies need additional time to be created and implemented.
• If you have reiterated your intent to follow the BR and root program policies, why are we in a situation where we can expect Entrust to delay revocation of certificates for 90 days as evidenced by this bug?

Comment 59

[Wayne]

The time for a response goes until 168 hours, however I agree it is not in-keeping with timely responses. This is an ongoing issue where responses are kept until the last minute as part of what can only be malicious compliance. Notably when Entrust overstepped the 168h mark recently the response window briefly changed to just within 6 days, so there is basis for this reasoning.

Comment 60

[ngook.kong]

(In reply to walter.j.marks from comment #49)

(In reply to Bruce Morton from comment #48)

(In reply to walter.j.marks from comment #37)

Setting aside the initial failure to revoke and the long tail of this process, I'd like to dig a bit further into the Subscriber arguments against revoking, and some of your statements included there. I'm going to make a few assumptions here, including that Entrust agrees that there is in fact a violation of the BRs requiring revocation of affected certs, otherwise I assume that you wouldn't have revoked in the first place.

“These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

“Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

“We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Q1. Were each of these Subscribers reminded of their agreement to the Terms of Use of Entrust Certificate and Signing Services? Specifically, Exhibit B, Part 1.12.3?

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

We have not reminded subscribers about this section of our Terms of Use, nor have subscribers raised questions about our contractual rights in this matter.

Since it wasn't mentioned in the June 7 Report (along with other omissions but I'll leave those opinions for MDSP to avoid mixing up action items here), what was the internal decision making that led to not reminding subscribers of this section of the ToU?

A lot of this aftermath and the ensuing incident failures that were due to "Entrust handling numerous incidents at the same time" (paraphrased) could have been mitigated or reduced by "We are required to revoke certificates within 24h/120h. We are here to assist you in any way we can during the next few days, but as part of our agreement to PKI as a whole as well as the BRs and various root programs, it is mandatory that these revocations occur."

Thanks for your suggestion. The internal process is for our support team to use the standard message to subscribers, included as attachment 9406229 [details], with update date. “Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC”. We did not point subscribers to the terms of use, and although subscribers have not questioned our right to revoke.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Q2. Why was a presumption of a Subscriber being unable to revoke without positive confirmation assumed as stated above? Again following from Exhibit B, Part 1.12.3, Entrust is entitled to revoke a certificate immediately without notifying the Subscriber.

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Falls under my thoughts from the above un-quoted statement.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Q3. How does the above statement square with the the subscriber justification given here?

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

“Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Again, you seem to misunderstand the question posed by the following quote:

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

We understand the question. As stated in your quote, we tried to balance the risk of the relying parties in the broader internet and the subscriber relying parties.

Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

While our terms of service might allow us to revoke certificates without contacting subscribers, we prioritized responsible action. We understand that certificate revocation can be disruptive, even for subscribers familiar with our terms of service.

Regarding your concerns about trust moving forward, following these recent incidents we have done a thorough review and root cause analysis of the commitments we made in 2020 and these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests.

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident

Of those three exemplar Subscribers I've referenced above, none seem to have a lack of automation as a blocking fact that they were unable to revoke (which again, should be an action taken by Subscribers by the powers given to you by Exhibit B, Part 1.12.3) and replace in time. A commitment to automation does not mean anything if the subscribers are simply refusing to accept a revocation within the required and expected time. A more accurate commitment that I would expect to answer the question would be something along the lines of "Entrust will refuse to accept any requests to delay revocation past 5 days....." along with a measurable and actionable key index for that commitment.
Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

We agree that a key outcome for this incident is for us to define a more rigorous revocation plan that includes a formal plan to define our policies and process for handling subscriber requests for delayed revocation. We expect to have these guidelines in place by the end of June.

There was a commitment to never delay revocation again after a delrev incident 4 years ago. What happened to those guidelines and internal policies then, and why were they not followed now? If they did exist prior to the June 7 Report, why do these policies need additional time to be created and implemented.

Please refer to the response above regarding our acknowledgement of lapses and actions to address. In our review of the root causes, we want to be sure that we solved for these issues today. We did not handle the recent incidents in a way that lives up to our standards. We are taking the time required to make lasting change organizationally, in policies and procedures, and in technology where appropriate.

Q5.1. Of these three exemplar Subscribers, were any reminded that if this was a key compromise event that had fundamental security risks, that this argument of "needing to notify downstream contracted parties that could take up to 3 months" (paraphrased) would potentially result in their TLS certificates being unsafe for the web ecosystem and their business operations for the entire 90 day period?

Yes, revocation requirements were communicated to customers.

Q5.2. If so, what was the justification provided by the Subscriber that they would be able to handle a situation like that within the 24h/120h period, and why that process was unable to be performed in this case, given that from a pure compliance perspective, in either case the affected certificate is invalid and should not be used by the Subscriber?

Subscriber feedback indicates that many subscribers non security events differently from security related events. In the event of a key compromise, we believe that our customers take security very seriously across their organizations, and subscribers understand the potential consequences for themselves and the web ecosystem.

The response here would be most appropriate in MDSP.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Q6. The above statement seems to indicate that Entrust believes that this incident is still not at the level of a revocation requirement. If that is the case, please explain how the processes followed for this mis-issuance are different from the planned handling of a wide-scale revocation requirement.

We have agreed that this was a mis-issuance event and have reiterated our intent to follow the TLS Baseline Requirements and root program policies. We have acknowledged that these certificates were mis-issued, and initiated revocation. To date all certificates in this incident have been revoked.

If you have reiterated your intent to follow the BR and root program policies, why are we in a situation where we can expect Entrust to delay revocation of certificates for 90 days as evidenced by this bug? I don't want to be flippant, but there's really no other way to square that circle. You cannot in good faith say "Entrust will follow the policies" in the exact same bug number as where there's proof that a known mis-issued certificate remained unrevoked for 95 days, especially since 4 years ago this statement was given by Bruce Morton:

• We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

and yet somehow we ended up in the exact same situation. In other words, I see more evidence that I should not be taking this commitment seriously than evidence that I should take this commitment seriously.

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

As noted above, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have and will continue to clarify across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident. We also will be adding certificate and CA/B compliance to our corporate governance committees to ensure sufficient visibility and prioritization of investment for this work.

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.

Comment 61

[Wayne]

(In reply to ngook.kong from comment #60)

Thanks for your suggestion. The internal process is for our support team to use the standard message to subscribers, included as attachment 9406229 [details], with update date. “Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC”. We did not point subscribers to the terms of use, and although subscribers have not questioned our right to revoke.

I take this as confirmation that Entrust handled other incidents similarly. Has anything changed since and been publicly communicated? I can see no mention of this in the report and would appreciate details.

While our terms of service might allow us to revoke certificates without contacting subscribers, we prioritized responsible action. We understand that certificate revocation can be disruptive, even for subscribers familiar with our terms of service.

Given that Entrust's position is that their actions were "responsible actions", then I presume that the same actions would be taken if this incident were to happen again tomorrow?

Regarding your concerns about trust moving forward, following these recent incidents we have done a thorough review and root cause analysis of the commitments we made in 2020 and these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests.

Can we see this 'thorough review and root cause analysis'? I presume you are not talking about the document submitted to MDSP, and given that it should be shared.

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident

Please provide details on all of this to the MDSP thread, it is not in your report.

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

Okay so exceptional circumstances will be the norm at Entrust despite previously stating:

What didn't go well

• The initial decision not to revoke was not made on the basis of a thorough review of the relevant documents and rules, and instead relied on an overused “exceptional circumstances” justification.

Can you confirm this interpretation?

As noted above, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have and will continue to clarify across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident. We also will be adding certificate and CA/B compliance to our corporate governance committees to ensure sufficient visibility and prioritization of investment for this work.

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.

The first half is reiterating commitments made 4 years ago that weren't kept - what trust is there that they will now? The second half is stating that Entrust will push to remove the 5-day requirement as it is not something they are capable of performing. I do not need to hear that Entrust are 'technical capable' of revocation, when they are organizationally incapable of handling incident response to get to that level.

What I do need to hear is that Entrust will ensure that no subscriber involved in this incident will be part of a delayed revocation event going forward. Can Entrust commit to this bare minimum guarantee that they are able to hold themselves to the baseline requirements?

Additionally please respond to statements issued on MDSP, the report is threadbare and we need to see more substance.

Comment 62

[walter.j.marks]

(In reply to ngook.kong from comment #60)

(In reply to walter.j.marks from comment #49)

(In reply to Bruce Morton from comment #48)

(In reply to walter.j.marks from comment #37)

Setting aside the initial failure to revoke and the long tail of this process, I'd like to dig a bit further into the Subscriber arguments against revoking, and some of your statements included there. I'm going to make a few assumptions here, including that Entrust agrees that there is in fact a violation of the BRs requiring revocation of affected certs, otherwise I assume that you wouldn't have revoked in the first place.

“These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

“Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

“We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Q1. Were each of these Subscribers reminded of their agreement to the Terms of Use of Entrust Certificate and Signing Services? Specifically, Exhibit B, Part 1.12.3?

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

We have not reminded subscribers about this section of our Terms of Use, nor have subscribers raised questions about our contractual rights in this matter.

Since it wasn't mentioned in the June 7 Report (along with other omissions but I'll leave those opinions for MDSP to avoid mixing up action items here), what was the internal decision making that led to not reminding subscribers of this section of the ToU?

A lot of this aftermath and the ensuing incident failures that were due to "Entrust handling numerous incidents at the same time" (paraphrased) could have been mitigated or reduced by "We are required to revoke certificates within 24h/120h. We are here to assist you in any way we can during the next few days, but as part of our agreement to PKI as a whole as well as the BRs and various root programs, it is mandatory that these revocations occur."

Thanks for your suggestion. The internal process is for our support team to use the standard message to subscribers, included as attachment 9406229 [details], with update date. “Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC”. We did not point subscribers to the terms of use, and although subscribers have not questioned our right to revoke.

Does not answer my question, yet again. In clear terms: what was the internal decision process that led Entrust to begin from a place of not revoking certificates, and only after being asked by multiple community members did the revocation process begin? This was also missing from the report that was provided on June 7.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Q2. Why was a presumption of a Subscriber being unable to revoke without positive confirmation assumed as stated above? Again following from Exhibit B, Part 1.12.3, Entrust is entitled to revoke a certificate immediately without notifying the Subscriber.

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Falls under my thoughts from the above un-quoted statement.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Q3. How does the above statement square with the the subscriber justification given here?

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

“Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Again, you seem to misunderstand the question posed by the following quote:

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

We understand the question. As stated in your quote, we tried to balance the risk of the relying parties in the broader internet and the subscriber relying parties.

Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

While our terms of service might allow us to revoke certificates without contacting subscribers, we prioritized responsible action. We understand that certificate revocation can be disruptive, even for subscribers familiar with our terms of service.

Regarding your concerns about trust moving forward, following these recent incidents we have done a thorough review and root cause analysis of the commitments we made in 2020 and these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests.

Does not answer the question. The question as asked is "If you are implying that customers did not read that part of the Terms of Use, and thus shouldn't be held to it, what other parts of the Terms of Use are not strictly enforced by Entrust?". Again, this was not included in the report.

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident

I see no detailed explanation of the changes made to Entrust's operational procedures in the June 7 Report.

Of those three exemplar Subscribers I've referenced above, none seem to have a lack of automation as a blocking fact that they were unable to revoke (which again, should be an action taken by Subscribers by the powers given to you by Exhibit B, Part 1.12.3) and replace in time. A commitment to automation does not mean anything if the subscribers are simply refusing to accept a revocation within the required and expected time. A more accurate commitment that I would expect to answer the question would be something along the lines of "Entrust will refuse to accept any requests to delay revocation past 5 days....." along with a measurable and actionable key index for that commitment.
Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

We agree that a key outcome for this incident is for us to define a more rigorous revocation plan that includes a formal plan to define our policies and process for handling subscriber requests for delayed revocation. We expect to have these guidelines in place by the end of June.

There was a commitment to never delay revocation again after a delrev incident 4 years ago. What happened to those guidelines and internal policies then, and why were they not followed now? If they did exist prior to the June 7 Report, why do these policies need additional time to be created and implemented.

Please refer to the response above regarding our acknowledgement of lapses and actions to address. In our review of the root causes, we want to be sure that we solved for these issues today. We did not handle the recent incidents in a way that lives up to our standards. We are taking the time required to make lasting change organizationally, in policies and procedures, and in technology where appropriate.

Does not answer the question. Even if I take this statement to refer to the above response, it is still a poor answer to the question. Again: what happened to the internal policies that were allegedly created in 2020 and why were they not used as a response to handle this incident, or were they not created as committed to then? Additionally, if I take this answer in concert with the answer above, it is even more questionable. This answer seems to imply that the policies were not created as committed to, however the above answer seems to imply that the policies were created. Which answer is the truth?

Q5.1. Of these three exemplar Subscribers, were any reminded that if this was a key compromise event that had fundamental security risks, that this argument of "needing to notify downstream contracted parties that could take up to 3 months" (paraphrased) would potentially result in their TLS certificates being unsafe for the web ecosystem and their business operations for the entire 90 day period?

Yes, revocation requirements were communicated to customers.

Q5.2. If so, what was the justification provided by the Subscriber that they would be able to handle a situation like that within the 24h/120h period, and why that process was unable to be performed in this case, given that from a pure compliance perspective, in either case the affected certificate is invalid and should not be used by the Subscriber?

Subscriber feedback indicates that many subscribers non security events differently from security related events. In the event of a key compromise, we believe that our customers take security very seriously across their organizations, and subscribers understand the potential consequences for themselves and the web ecosystem.

The response here would be most appropriate in MDSP.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Q6. The above statement seems to indicate that Entrust believes that this incident is still not at the level of a revocation requirement. If that is the case, please explain how the processes followed for this mis-issuance are different from the planned handling of a wide-scale revocation requirement.

We have agreed that this was a mis-issuance event and have reiterated our intent to follow the TLS Baseline Requirements and root program policies. We have acknowledged that these certificates were mis-issued, and initiated revocation. To date all certificates in this incident have been revoked.

If you have reiterated your intent to follow the BR and root program policies, why are we in a situation where we can expect Entrust to delay revocation of certificates for 90 days as evidenced by this bug? I don't want to be flippant, but there's really no other way to square that circle. You cannot in good faith say "Entrust will follow the policies" in the exact same bug number as where there's proof that a known mis-issued certificate remained unrevoked for 95 days, especially since 4 years ago this statement was given by Bruce Morton:

• We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

and yet somehow we ended up in the exact same situation. In other words, I see more evidence that I should not be taking this commitment seriously than evidence that I should take this commitment seriously.

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

As noted above, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have and will continue to clarify across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident. We also will be adding certificate and CA/B compliance to our corporate governance committees to ensure sufficient visibility and prioritization of investment for this work.

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.
Arguing that 24h/120h is too hard to accomplish while you're currently under review for failure to revoke within 24h/120h for multiple mis-issuances is a hard sell, and I'd be very hesitant to lean on this argument.

Comment 63

[walter.j.marks]

Re Comment #62: It seems that the quote swallowed my final statement.

The last paragraph should read:

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.

Arguing that 24h/120h is too hard to accomplish while you're currently under review for failure to revoke within 24h/120h for multiple mis-issuances is a hard sell, and I'd be very hesitant to lean on this argument.

Comment 64

[Mike Shaver (:shaver emeritus)]

(In reply to ngook.kong from comment #60)

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident

As noted above, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have and will continue to clarify across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident. We also will be adding certificate and CA/B compliance to our corporate governance committees to ensure sufficient visibility and prioritization of investment for this work.

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.

You are describing activity, but not outcomes that the community can observe or use to evaluate Entrust's success at improving its operations. If you want to regain trust from the community, you should be:

• making it clear what you expect future conduct by Entrust to be, and how you expect the community to evaluate you on it
• demonstrating that you fully understand the root causes (not surface causes, like "not enough staffing") of the failures to conduct operations appropriately over the past four years
• describing in detail the changes being made in response
• providing a convincing argument that the changes will result in the promised operational outcomes

Frankly, that's what should have been in the report that Bruce posted to mdsp.

Right now we have ample evidence of Entrust not operating in accordance with the expectations that the Mozilla root program and BRs have for a CA. We also have evidence that explicit commitments made previously—regarding the exact same issue of improperly delayed revocation—were not honoured, and that Entrust did not notice for more than four years that they were not being honoured. The latter means that there is a very high bar for Entrust to clear in demonstrating that it both understands what is expected of it as a public web CA, and is capable and motivated enough to make the changes required to meet that expectation.

A leadership change might be good or bad or meaningless; there is not enough information provided to really warrant confidence. How did the old leadership lead to the problems we have observed? What will the new leadership do differently?

You've reorganized: what was wrong with the old organization, and what caused it to be that way? What forces were acting on the old organization that contributed to the failures, and how is the new organization designed to avoid them?

You're reducing the length of time approved for delayed revocation. What was the previous length of time, and how was it determined? The previous statements in this incident bug about the process for determining if Entrust would violate the BRs on behalf of a customer describe something tremendously sloppy, to be polite, for managing certificate lifecycles for such critical industries. What is the new length of time? How was it determined?

Will Entrust continue to issue public web PKI certificates to subscribers who they know to be unable to tolerate an appropriately-prompt revocation without catastrophic consequences to society?

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

Nobody is saying that there will never be exceptional circumstances that warrant or explain a good faith failure to revoke on time. The issue is a serious misalignment between Entrust's behaviour and the community's expectations in terms of how those delayed revocations are determined. There is ample evidence that Entrust is knowingly creating situations that they will later use to justify delrev incidents, by issuing web PKI certificates to subscribers who are known to be unable to replace a certificate promptly. This pattern of issuance since the 2020 commitments has not changed in any materially visible way, and there has been no commitment from Entrust to avoid recreating the situations with future issuances. You write of "pushing customers", but again not what exactly that entails or how the community should observe Entrust's progress on it without waiting for another misissuance event with three month revocation timelines and knowingly and deliberately insufficient description of the reasons for the delays. Is Entrust asking nicely? Requiring subscribers to also approve and prepare an alternative certificate from another CA that they can quickly switch to if their Entrust certificate needs to be revoked?

I would caution against using other CA behaviour as evidence of a problem, when Entrust is the one most contributing to a pattern of not meeting the standards expected.

I want to emphasize this part again:

In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident.

It is not enough to be "guided". They are requirements and need to be treated as such. As far as I can tell Entrust has never fully met the requirements outlined in Mozilla incident-response expectations for delayed revocation. Entrust needs to demonstrate that it fully understands what is expected of it, and that it has the means to comply.

And this part:

pushing customers to take measures that will enable more rapid revocation

Customers do not "enable" revocation. CAs revoke. Only CAs can do so, and it is to CAs that the responsibility falls to revoke as a remedy for cases where the CA failed to issue certificates correctly. Customers take responsibility for their own operational processes, and deploy web PKI certificates with the full knowledge, legally assured, that they might be revoked on short notice. Entrust might decide to take some responsibility for customer operational processes through their own products or services, but they absolutely cannot force the root programs and the web PKI at large to bear the consequences of subscribers' inability or unwillingness to build systems that are resilient to the realities of web PKI use.

Comment 65

[Wayne]

Please note that comment 52 has not been answered within 7 days.

Comment 66

[ngook.kong]

(In reply to walter.j.marks from comment #52)

(In reply to ngook.kong from comment #51)

(In reply to Wayne from comment #40)

1. Every subscriber was contacted with the letter
2. Not all subscribers responded by the date
3. We decided not to revoke if we did not see revocation progress, and/or no confirmed contact (either an email reply or talking to someone directly). The intent was to avoid unintended harmful consequences. As expressed before, improving this process is a key learning and area of improvement we are committed to and welcome your feedback accordingly.

Just to specifically dig into this, as I touched on it above and this is seemingly a tacit admission of what I was poking at above:

Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

If you do not believe that Subscribers were prepared for the revocation (and the "harmful consequences"), you logically most also not be holding them to Exhibit B, Part 1.12.3 of your Terms of Use, quoted again for your convenience:

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

Question: If so, what other portions of the Terms of Use are flexible in Entrust's enforcement of the Terms of Use?

We decided not to revoke if we did not see revocation progress,

Question: Without naming any specific Entrust employees, is there a specific Directly Responsible Individual (or Individuals) who made the decision to not revoke (the "we decided") without affirmative consent from the Subscriber, and is that noted in your internal history of this incident?

Yes, there is a directly responsible individual, and yes the internal discussions capture this.

Question: If so, did this DRI make the decision wholesale, or was the decision made per subscriber?

See previous answer.

Comment 67

[ngook.kong]

(In reply to Martijn Katerbarg from comment #53)

Hi Ngook,

Could you please help me understand something?

In comment #0 Entrust wrote:

We expect further revocation delays, details with customer feedback to follow, initial feedback includes:
Uncapable to deal with high volumes of manual re-issuance
Lockdown because of end of the fiscal year/quarter
Lockdown due to easter holidays

In comment #1 Entrust wrote (emphasis mine):

We have experienced significant pushback from customers on the feasibility of revocation timelines. The revocation for customers is in most cases manual and complex, involving multiple internal parties to ensure that the change does not create an adverse impact on web services and back-end processing.
As a result, we have moved these customers into delayed revocation, particularly where their operations are critical to the web ecosystem (e.g., banks, payment networks, airlines, and government agencies).

In comment #28 Entrust wrote:

We communicated with subscribers via email, video calls, phone calls, and through support ticket dialogue to ask them to justify their need for delayed revocation. Specifically, we asked them to detail their technical challenges and/or impacts to the web ecosystem if forced to revoke within 5 days as well as how much time they needed to replace and revoke their certificates.
Because of the volume, we did not always receive or record this information effectively for all customers. We have identified this as an area of process improvement. Going forward, we are looking at the use of a standardized way to collect and record this data to ensure that if there are future mis-issuances, the collection of this information is more structured and complete.

In comment #36 Entrust wrote (emphasis mine):

Criteria: Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem, we did not grant their delayed revocation requests.

In comment #48 Entrust wrote (emphasis mine):

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Additionally, in several posts Entrust also wrote an update of their revocation progress. Nowhere there did it state, or show progress, of how many customers were contacted, and thus moved into the revocation phase.

All of the above statements are related to notifications that were sent out, and that only customers that specifically requested a delayed revocation due to “business/industry disruption, technical challenges and/or impacts to the web ecosystem” were granted a delay.

Yet, now in comment #51, Entrust states:

1. Every subscriber was contacted with the letter
2. Not all subscribers responded by the date
3. We decided not to revoke if we did not see revocation progress, and/or no confirmed contact (either an email reply or talking to someone directly). The intent was to avoid unintended harmful consequences. As expressed before, improving this process is a key learning and area of improvement we are committed to and welcome your feedback accordingly.

Can you please explain:

• The discrepancies between these statements, where previously you have stated that you did not grant any delayed revocation “Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem”, yet now you state you automatically granted a delay if you were unable to reach the subscriber?
• If you used the same approach as described in the quoted section from comment #51 in other delayed revocation bugs?
• Why it took Entrust up to now to come out with this information?

If it was not implied, there is a big difference between granting (very limited) delayed revocation in exceptional circumstances and blindly just not revoking certificates if you were unable to reach your customers or confirm that they are taking action.

We have described the incident in this bug in great detail. As we have said, it’s clear from these notes above and the week-to-week revocation timeline that we did not handle this delayed revocation in a way that lives up to our standards. We recognize that it is the CA’s responsibility to revoke, not the subscriber’s.

At the start of this incident, we received significant pushback from customers on revocation timelines, and these customers often represented organizations that provide critical infrastructure services. As a result, as stated in comment 51 and elsewhere, in this case, with subscribers who had not responded, we decided to prioritize avoiding unintended harmful consequences.

As we have noted elsewhere, we have done a thorough review and root cause analysis of these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests. As a result, we made leadership changes and reorganized our certificate services business to leverage our global compliance resourcing, expertise, and governance more fully.

And we have made it clear across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident.

Comment 68

[ngook.kong]

(In reply to boet from comment #56)

(In reply to ngook.kong from comment #55)

The concept of "critical infrastructure" can be complex, but there are some definitions such as in CISA and the EU CER and NIS-2, which provide helpful guidance. There's a spectrum of risk within critical infrastructure. Activities directly involved in life-or-death scenarios (e.g., controlling air traffic) fall under our prohibited use cases. However, there is a range of critical infrastructure where the use of certificates is not prohibited.

I explicitly asked which definition Entrust operate under, the lack of commitment to any shows it is none.

There is no single definition. Entrust looks at the definitions of critical infrastructure from US CISA, EU CER and NIS-2 in formulating its definition.

Does Entrust explicitly formulate a definition ahead of time based on these sources, or is a working definition created ad-hoc when circumstances call for one?

We have described the working definition that we used to guide our decisions.

Comment 69

[ngook.kong]

(In reply to Wayne from comment #57)

Given the lack of answers to questions all questions posed in this comment have been bolded for clarity.

(In reply to ngook.kong from comment 54)

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

To clarify Entrust's understanding the questions posed were:

Just to dig into this a bit further, the (already against the BR required timeline) of the original due date you listed for these, varying between May 30 and June 2 was agreed upon with the relevant subscriber, correct?

If that's so, then why are some subscribers now in need of an additional 2 weeks?

This is basically correct, yes. The subscriber requested additional time to safely revoke the certificates. They also were able to complete revocation more quickly than they originally estimated.

We provided limited exceptions for a small number of certificates through June 16 because the subscribers were involved in a global financial change event aimed at increasing the security of global financial transactions and requested additional time. They were able to complete revocation of these certificates on June 6.

An answer to them would be preferable. The same goes for:

As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.

That is why we evaluated their delayed revocation requests and communicated with them to verify that impacted certificates could safely be revoked.

As noted above, we will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

The question posed was:

I believe what was being asked there is "what additional commitments are needed from a Subscriber beyond what they have already agreed to upon purchase of a certificate?"

I will then add on further questions:

• Can Entrust explain why it is irresponsible to revoke a certificate over 90 certs into an incident?
• What 'context' is being missed, and can that be provided per-subscriber?
• Yes or No: Will Entrust commit to not causing delayed revocation to go past, say, 30 days going forward?
• Provide a simple figure for the number of days that Entrust feel they are organizationally capable of handling revocation given their subscriber base?

We acknowledge the question and would note that it has been answered above: “As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.” In terms of context, we provided reasons for extensions in comment 25 and several examples of correspondence with individual customers including details on reasons for extensions.

Respectfully, we’re not going to make blanket statements about arbitrary delayed revocation timelines. We have stated our intention to comply with the requirements set by the CA/Browser Forum and the root programs, guided by the TLS Baseline Requirements.

Per their own Prohibited Certificate Usage terms, no such certificate should be used in a situation that can cause harm or serious economic impact. All authority has been bestowed onto Entrust in order to fulfill their revocation obligations, there is just an odd refusal to do so. If this could be explained at all I'd appreciate it.

The issue of Prohibited Certificate Usage Terms was covered in comments 24 and 42.

(In reply to ngook.kong from comment 55)

There is no single definition. Entrust looks at the definitions of critical infrastructure from US CISA, EU CER and NIS-2 in formulating its definition.

If there is no single definition, then what did Entrust 'formulate' in its own definition, which you are claiming does not exist?

We do not issue certificates to subscribers for the uses prohibited in our CPS. We do, however, still have subscribers who operate in critical infrastructure sectors as defined by CISA, e.g., transportation and financial services.

Entrust are explicitly confirming that no subscribers were using their certificates "in conjunction with any application that could lead to death, personal injury or severe physical or property damage"?

Following that question, and given these are self-reported reasons provided by subscribers:

• What rationale was there in delaying revocation due to 'harm' and 'critical infrastructure'?
• Can Entrust document a single instance of them pushing back against these self-reported claims?
• Subscribers state they are in the above categories, does Entrust agree?
• We have a list of subscribers and these statements so please provide some examples

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

My question was not answered. I will reiterate it as Entrust seem to have a problem with answering questions lately:

• Why are Entrust still providing "limited exceptions" 2 months into a delayed revocation failure?

I am not asking what you are planning to put into policy going forward, but what happened historically in regards to this incident. I was hoping this would also be in your formal report, but I can see nothing like that.

All affected certificates were revoked by June 6, 2024.

Again my question has not been answered:

• Are Entrust capable of giving any reason for these delays?

See above.

I will note that the certificates were miraculously revoked by June 6th, and an explanation of what happened to get us into this situation is surely required.

Comment 70

[walter.j.marks]

(In reply to ngook.kong from comment #69)

(In reply to Wayne from comment #57)

Given the lack of answers to questions all questions posed in this comment have been bolded for clarity.

(In reply to ngook.kong from comment 54)

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

To clarify Entrust's understanding the questions posed were:

Just to dig into this a bit further, the (already against the BR required timeline) of the original due date you listed for these, varying between May 30 and June 2 was agreed upon with the relevant subscriber, correct?

If that's so, then why are some subscribers now in need of an additional 2 weeks?

This is basically correct, yes. The subscriber requested additional time to safely revoke the certificates. They also were able to complete revocation more quickly than they originally estimated.

The point I am trying to elaborate on is "why did they need an extension?" Setting aside the decision to delay revocation for a second, if a conversation was being had with a large financial institution that involved various teams involved both from Entrust and the large financial institution, why was the initial self-imposed revocation deadline missed? I realize that this is straying into conversations that I understand are confidential information between you and your Subscribers, so I'm not asking for specifics or even generalities, but what I am getting at here is the following:

• Entrust made a decision in collaboration with the Subscriber to delay revocation X number of days.
• As far as I can tell, Entrust offered no upper bound on the amount of time they would reasonably accept on this extension.
  • If there is an upper bound that Entrust would have accepted, I see no evidence of this. This is something that should have been included in the June 7 report, but I repeat myself (along with other community members)
• In other words, the timeline that was agreed upon should have been an achievable goal by both Entrust and the Subscribers as there was seemingly no upper limit. "We'll need 90 days to replace it given x,y,z" "Alright, we'll grant that" Given your previous responses, it seems that there was indeed involved tier 2/3 support which would have resulted in those conversations around a revocation timeline with these large enterprise customers, which again, I understand given the scope and scale of these customers.
• The Subscriber was hopefully made aware of the urgency of the revocation requirements.
• With all of this in mind, we had certificates make it to 72 hours past the agreed upon revocation date as mentioned in Comment #41.
• This failure to make the self-imposed deadline was not acknowledged by Entrust until 2024-06-04, with new deadlines varying between 2024-06-06 and 2024-06-16. Even if those certificates were mis-issued on 2024-06-04 and a 5 day revocation timer started then, the revocation deadline of 2024-06-16 is still ~7 days late.

I'm speaking somewhat rhetorically here to avoid having to put Entrust in a position where it needs to share specifics or communications from a Subscriber, but what I am concerned about is that we have dates in Attachment #9403115 [details] that were supposedly committed to by both Entrust and the Subscriber after a thoughtful and considered communication between the two parties, which were then promptly missed by 2 to 5 days without any acknowledgement, and then on the upper end were extended by an additional 14 days. I fail to see any sort of urgency or re-iteration of haste required being elaborated by Entrust to their Subscribers.

We provided limited exceptions for a small number of certificates through June 16 because the subscribers were involved in a global financial change event aimed at increasing the security of global financial transactions and requested additional time. They were able to complete revocation of these certificates on June 6.

An answer to them would be preferable. The same goes for:

As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.

That is why we evaluated their delayed revocation requests and communicated with them to verify that impacted certificates could safely be revoked.

As noted above, we will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

The question posed was:

I believe what was being asked there is "what additional commitments are needed from a Subscriber beyond what they have already agreed to upon purchase of a certificate?"

I will then add on further questions:

• Can Entrust explain why it is irresponsible to revoke a certificate over 90 certs into an incident?
• What 'context' is being missed, and can that be provided per-subscriber?
• Yes or No: Will Entrust commit to not causing delayed revocation to go past, say, 30 days going forward?
• Provide a simple figure for the number of days that Entrust feel they are organizationally capable of handling revocation given their subscriber base?

We acknowledge the question and would note that it has been answered above: “As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.” In terms of context, we provided reasons for extensions in comment 25 and several examples of correspondence with individual customers including details on reasons for extensions.

Respectfully, we’re not going to make blanket statements about arbitrary delayed revocation timelines. We have stated our intention to comply with the requirements set by the CA/Browser Forum and the root programs, guided by the TLS Baseline Requirements.

Per their own Prohibited Certificate Usage terms, no such certificate should be used in a situation that can cause harm or serious economic impact. All authority has been bestowed onto Entrust in order to fulfill their revocation obligations, there is just an odd refusal to do so. If this could be explained at all I'd appreciate it.

The issue of Prohibited Certificate Usage Terms was covered in comments 24 and 42.

(In reply to ngook.kong from comment 55)

There is no single definition. Entrust looks at the definitions of critical infrastructure from US CISA, EU CER and NIS-2 in formulating its definition.

If there is no single definition, then what did Entrust 'formulate' in its own definition, which you are claiming does not exist?

We do not issue certificates to subscribers for the uses prohibited in our CPS. We do, however, still have subscribers who operate in critical infrastructure sectors as defined by CISA, e.g., transportation and financial services.

Entrust are explicitly confirming that no subscribers were using their certificates "in conjunction with any application that could lead to death, personal injury or severe physical or property damage"?

Following that question, and given these are self-reported reasons provided by subscribers:

• What rationale was there in delaying revocation due to 'harm' and 'critical infrastructure'?
• Can Entrust document a single instance of them pushing back against these self-reported claims?
• Subscribers state they are in the above categories, does Entrust agree?
• We have a list of subscribers and these statements so please provide some examples

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

My question was not answered. I will reiterate it as Entrust seem to have a problem with answering questions lately:

• Why are Entrust still providing "limited exceptions" 2 months into a delayed revocation failure?

I am not asking what you are planning to put into policy going forward, but what happened historically in regards to this incident. I was hoping this would also be in your formal report, but I can see nothing like that.

All affected certificates were revoked by June 6, 2024.

Again my question has not been answered:

• Are Entrust capable of giving any reason for these delays?

See above.

I will note that the certificates were miraculously revoked by June 6th, and an explanation of what happened to get us into this situation is surely required.

Comment 71

[Wayne]

Given there are some difficulties in reading where questions are I have bolded them for clarity, despite previous commenters already doing so and it not helping...

(In reply to ngook.kong from comment #66)

Question: If so, what other portions of the Terms of Use are flexible in Entrust's enforcement of the Terms of Use?

Entrust please read the questions provided this has not been answered.

Yes, there is a directly responsible individual, and yes the internal discussions capture this.

Question: If so, did this DRI make the decision wholesale, or was the decision made per subscriber?

See previous answer.

Your previous answer does not answer the question, please read the questions before attempting to answer.

(In reply to ngook.kong from comment #67)

Can you please explain:

• The discrepancies between these statements, where previously you have stated that you did not grant any delayed revocation “Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem”, yet now you state you automatically granted a delay if you were unable to reach the subscriber?
• If you used the same approach as described in the quoted section from comment #51 in other delayed revocation bugs?
• Why it took Entrust up to now to come out with this information?

If it was not implied, there is a big difference between granting (very limited) delayed revocation in exceptional circumstances and blindly just not revoking certificates if you were unable to reach your customers or confirm that they are taking action.

We have described the incident in this bug in great detail. As we have said, it’s clear from these notes above and the week-to-week revocation timeline that we did not handle this delayed revocation in a way that lives up to our standards. We recognize that it is the CA’s responsibility to revoke, not the subscriber’s.

At the start of this incident, we received significant pushback from customers on revocation timelines, and these customers often represented organizations that provide critical infrastructure services. As a result, as stated in comment 51 and elsewhere, in this case, with subscribers who had not responded, we decided to prioritize avoiding unintended harmful consequences.

As we have noted elsewhere, we have done a thorough review and root cause analysis of these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests. As a result, we made leadership changes and reorganized our certificate services business to leverage our global compliance resourcing, expertise, and governance more fully.

And we have made it clear across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident.

I will note that, once again, the questions posed are not answered. Further the attempt to conflate "these customers often represented organizations that provide critical infrastructure services" with the certificates being used in such services is concerning. Entrust do know that the criteria is about certificate usage, not that a subscriber allegedly providing critical infrastructure services?

Can Entrust explain how "in this case, with subscribers who had not responded, we decided to prioritize avoiding unintended harmful consequences" is compatible with previous statements that Entrust do not make unilateral decisions on revocation?

Furthermore, none of what Entrust have stated is clear otherwise there would not be so many questions. Please attempt to answer in a straightforward, direct, and honest answer. The current list of possible speaking points are not even being correctly put against the relevant questions.

(In reply to ngook.kong from comment #68)

Does Entrust explicitly formulate a definition ahead of time based on these sources, or is a working definition created ad-hoc when circumstances call for one?

We have described the working definition that we used to guide our decisions.

If you did then there would not be so many questions on this topic. Can Entrust please provide their definition in their own words in a clear and articulate way?

(In reply to ngook.kong from comment #69)

Just to dig into this a bit further, the (already against the BR required timeline) of the original due date you listed for these, varying between May 30 and June 2 was agreed upon with the relevant subscriber, correct?

If that's so, then why are some subscribers now in need of an additional 2 weeks?

This is basically correct, yes. The subscriber requested additional time to safely revoke the certificates. They also were able to complete revocation more quickly than they originally estimated.

We provided limited exceptions for a small number of certificates through June 16 because the subscribers were involved in a global financial change event aimed at increasing the security of global financial transactions and requested additional time. They were able to complete revocation of these certificates on June 6.

Even into June we are still in the position of the subscriber telling Entrust when it will be convenient for them to rotate certificates. I will note that in your response no actual critical services or harm is mentioned or implied - at most the customer was too busy. I do not feel that Entrust are articulating the importance of a timely revocation and are still not handling this in a manner that is befitting of a CA.

If Entrust feels this is not the case then please articulate why?

I will then add on further questions:

• Can Entrust explain why it is irresponsible to revoke a certificate over 90 certs into an incident?
• What 'context' is being missed, and can that be provided per-subscriber?
• Yes or No: Will Entrust commit to not causing delayed revocation to go past, say, 30 days going forward?
• Provide a simple figure for the number of days that Entrust feel they are organizationally capable of handling revocation given their subscriber base?

We acknowledge the question and would note that it has been answered above: “As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.” In terms of context, we provided reasons for extensions in comment 25 and several examples of correspondence with individual customers including details on reasons for extensions.

Respectfully, we’re not going to make blanket statements about arbitrary delayed revocation timelines. We have stated our intention to comply with the requirements set by the CA/Browser Forum and the root programs, guided by the TLS Baseline Requirements.

Entrust are entitled to refuse to answer questions. Refusing will only reflect on their appearance of transparency, honesty, and their intentions are to comply with the requirements set by the CA/Browser Forum and the root programs. Entrust are more than entitled to return at a later date to attempt to answer those questions in a satisfactory manner. I will note that the figures quoted are far outwith timeframes that Entrust are mandated to perform, so no attempt at even agreeing is a tacit statement that things will not change.

Per their own Prohibited Certificate Usage terms, no such certificate should be used in a situation that can cause harm or serious economic impact. All authority has been bestowed onto Entrust in order to fulfill their revocation obligations, there is just an odd refusal to do so. If this could be explained at all I'd appreciate it.

The issue of Prohibited Certificate Usage Terms was covered in comments 24 and 42.

If the question were sufficiently answered I would not be posing it. Even in response to this comment I have been told that "The subscriber requested additional time to safely revoke the certificates." and were part of a financial entity. Entrust can't have it both ways and needs to address their conflicting statements to date. If Entrust does not feel these are conflicting please articulate why without referencing other answers?

If there is no single definition, then what did Entrust 'formulate' in its own definition, which you are claiming does not exist?

Entrust are explicitly confirming that no subscribers were using their certificates "in conjunction with any application that could lead to death, personal injury or severe physical or property damage"?

Following that question, and given these are self-reported reasons provided by subscribers:

• What rationale was there in delaying revocation due to 'harm' and 'critical infrastructure'?
• Can Entrust document a single instance of them pushing back against these self-reported claims?
• Subscribers state they are in the above categories, does Entrust agree?
• We have a list of subscribers and these statements so please provide some examples

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

My question was not answered. I will reiterate it as Entrust seem to have a problem with answering questions lately:

• Why are Entrust still providing "limited exceptions" 2 months into a delayed revocation failure?

I am not asking what you are planning to put into policy going forward, but what happened historically in regards to this incident. I was hoping this would also be in your formal report, but I can see nothing like that.

All affected certificates were revoked by June 6, 2024.

Again my question has not been answered:

• Are Entrust capable of giving any reason for these delays?

See above.

Entrust please actually read the questions posed. If this is a refusal to ask the questions then please say so against each question so it is clear what your intentions are.

Comment 72

[Tim Callan]

(In reply to ngook.kong from comment #60)

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation.
. . .
What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays …

So, to put it plainly, you absolutely expect to continue to delay mandatory revocation in the future and are telling us so right now.

Question 1: Yes or no, is the above statement correct? Please start your response with either the word yes or the word no. You may add other words after, but please avoid the Entrust habit of giving vague evasive responses that dodge the question.

Question 2: If no, state in terms as plain as I did in the paragraph above the commitment you are making, whatever that may be, to the community WRT delayed revocation.

Comment 73

[Bruce Morton]

Action Item	Kind	Due Date
Add support for ACME Renewal Information (ARI) in ACME	Mitigate	2024-09-30
Work with customers to replace and revoked impacted certificates	Mitigate	Done
Improve escalation procedures and documentation	Mitigate	2024-05-31

Comment 74

[Wayne]

(In reply to Bruce Morton from comment #73)

Action Item	Kind	Due Date
Add support for ACME Renewal Information (ARI) in ACME	Mitigate	2024-09-30
Work with customers to replace and revoked impacted certificates	Mitigate	Done
Improve escalation procedures and documentation	Mitigate	2024-05-31

All Entrust have done here is change the due date on revocation to done and notified us that the 'escalation procedures and documentation' action has not been complete. That was due 3 weeks ago.

It also doesn't align with the other action items stated from the report so please clarify if those dates are now changed.

Comment 75

[Bruce Morton]

(In reply to Wayne from comment #74)
Hi Wayne, thanks for the feedback. The action item table has been aligned with the actions listed in bug 1901270.

Action Item	Kind	Due Date
Implement ACME ARI	Mitigate	2024-07-31
Work with customers to replace and revoked impacted certificates	Mitigate	Done
Implement formal incident response process including incident response communication plan to meet mandatory reporting times	Prevent	2024-06-30

Comment 76

[ngook.kong]

(In reply to walter.j.marks from comment #58)

As a reminder, my questions asked in Comment #49 remain unanswered at the 7 day mark.

Here they are bolded for your convenience:

• Since it wasn't mentioned in the June 7 Report (along with other omissions but I'll leave those opinions for MDSP to avoid mixing up action items here), what was the internal decision making that led to not reminding subscribers of this section of the ToU?

As stated in Comment 60:  The internal process is for our support team to use the standard message to subscribers, included as  attachment 9406229 [details] with update date. “Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC”.

This process does not point to the Terms of Use.

• Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

As stated in Comment 60: While our terms of service might allow us to revoke certificates without contacting subscribers, we prioritized responsible action. We understand that certificate revocation can be disruptive, even for subscribers familiar with our terms of service.

Regarding your concerns about trust moving forward, we are investing to scheduling discussions with our subscribers to ensure they understand the ToUs, educate our sales team to ensure ToUs are discussed as part of sales and CRM initiatives. We have repeatedly found that the ToUs are not widely understood by the subscriber base. We consider this continued investment a necessary step to ensure we maintain trust with this community and with our subscribers.

Additionally, following these recent incidents we have done a thorough review and root cause analysis of the commitments we made in 2020 and these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests.

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident.

• There was a commitment to never delay revocation again after a delrev incident 4 years ago. What happened to those guidelines and internal policies then, and why were they not followed now? If they did exist prior to the June 7 Report, why do these policies need additional time to be created and implemented.

As stated in Comment 60. Please refer to the response above regarding our acknowledgement of lapses and actions to address. In our review of the root causes, we want to be sure that we solved for these issues today. We did not handle the recent incidents in a way that lives up to our standards, and we understand that this has created doubt. We are taking the time required to make lasting change organizationally, in policies and procedures, and in technology where appropriate. We have committed to tracking progress on these changes publicly.

• If you have reiterated your intent to follow the BR and root program policies, why are we in a situation where we can expect Entrust to delay revocation of certificates for 90 days as evidenced by this bug?

We replied to this in Comment 60. In addition, we have acknowledged that it is insufficient to have an intention without backing up that intention with the level of organization, investment and organizational commitment described in our June 7 report (and will provide more details based on community feedback). Bug #1901270 is where we are tracking our specific action items related to delayed revocation.

Comment 77

[ngook.kong]

(In reply to Wayne from comment #61)

(In reply to ngook.kong from comment #60)

Thanks for your suggestion. The internal process is for our support team to use the standard message to subscribers, included as attachment 9406229 [details], with update date. “Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC”. We did not point subscribers to the terms of use, and although subscribers have not questioned our right to revoke.

I take this as confirmation that Entrust handled other incidents similarly. Has anything changed since and been publicly communicated? I can see no mention of this in the report and would appreciate details.

If you’re referring to the delayed revocation covered in bug #1887705, yes these incidents were handled similarly.

In our June 7th report, s.2.5.3 (Root Cause Analysis relating to Delayed Revocation Incidents) we stated that over the past several years we had not made the progress necessary (in terms of educating subscribers) to enable five-day revocation without large-scale disruption to their critical operations. We outlined specific items in our improvement plan in s.2.5.4, one of which is to adopt a new revocation policy and ensure that it is communicated to subscribers. We are considering ways to increase visibility of the CA’s right to revoke certificates on short notice beyond our contract language. We also will add a warning to manual order pages and related emails to ensure subscriber understanding of required timelines. These and other action items are being tracked in Bug 1901270. The earliest target completion date for these action items is June 30th. So in response to the question about whether our process has already changed, the answer is no, it has not yet changed, but it will. We understand that only actual change, and nothing we say here, will change anyone’s opinion about us.

While our terms of service might allow us to revoke certificates without contacting subscribers, we prioritized responsible action. We understand that certificate revocation can be disruptive, even for subscribers familiar with our terms of service.

Given that Entrust's position is that their actions were "responsible actions", then I presume that the same actions would be taken if this incident were to happen again tomorrow?

No, we do not believe the same actions would be taken. Although not all of our action items set out in Bug 1901270 are complete, some changes have already occurred, including having more and different people engaged in assessment and decision making around incidents. In the event of a new mis-issuance occurring prior to the new formal processes being in place, we expect we would act differently, especially in the way we communicate with and gather information from subscribers. We will still seek to act responsibly, but by improving our communications to subscribers and by gathering more relevant information from subscribers, we believe that we will be able to responsibly comply with the industry requirements.

Regarding your concerns about trust moving forward, following these recent incidents we have done a thorough review and root cause analysis of the commitments we made in 2020 and these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests.

Can we see this 'thorough review and root cause analysis'? I presume you are not talking about the document submitted to MDSP, and given that it should be shared.

We were talking about the June 7 report submitted to MDSP. We also acknowledge the feedback we have received on the report, including that it lacked sufficient details. We are working to address this feedback and plan to share more information and analysis as soon as possible.

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident

Please provide details on all of this to the MDSP thread, it is not in your report.

Acknowledged, thank you. We will provide more details in an update to the report.

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

Okay so exceptional circumstances will be the norm at Entrust despite previously stating:

What didn't go well

• The initial decision not to revoke was not made on the basis of a thorough review of the relevant documents and rules, and instead relied on an overused “exceptional circumstances” justification.

Can you confirm this interpretation?

No, exceptional circumstances will not be “the norm” at Entrust. We have acknowledged that we have been too lenient and have not adopted, documented or implemented sufficiently strict rules about what might constitute exceptional circumstances. This was a lapse from previous commitments and as mentioned in prior posts, fixes will be tracked publicly to show transparency and progress. That said, as noted, this is a pervasive issue that will take time and effort on multiple fronts to address, and in the meantime, exceptional circumstances warranting delayed revocation may continue to occur more frequently than anyone in the community would like.

As noted above, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have and will continue to clarify across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident. We also will be adding certificate and CA/B compliance to our corporate governance committees to ensure sufficient visibility and prioritization of investment for this work.

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.

The first half is reiterating commitments made 4 years ago that weren't kept - what trust is there that they will now?

The community should trust that the commitments will be kept as we have installed a new leadership team, aligned them with a corporate function that is incentivized only for compliance, compliance with the requirements is part of senior leadership investment reviews, detailed review of compliance status for our certificate business will be standing agenda item at corporate governance committees vs product as in the past. These are some of the specific details that we plan to implement and track publicly. We hope we will regain the confidence of the community by implementing actual changes and showing real improvement.

The second half is stating that Entrust will push to remove the 5-day requirement as it is not something they are capable of performing.

No, that is not what Entrust stated. To be clear, we said we were committed to discussion on these issues, which are important. We said nothing in this statement about our ability to perform a five day revocation.

I do not need to hear that Entrust are 'technical capable' of revocation, when they are organizationally incapable of handling incident response to get to that level.

What I do need to hear is that Entrust will ensure that no subscriber involved in this incident will be part of a delayed revocation event going forward. Can Entrust commit to this bare minimum guarantee that they are able to hold themselves to the baseline requirements?

No, we will not make any such guarantee. Entrust will plan to meet the revocation requirements of the baseline requirements, but may also permit delayed revocation per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation. We understand that the Mozilla policy does not authorize non-compliance with, or grant exceptions to, the BRs. We understand and agree that the “CA is ultimately responsible for deciding if the harm caused by following the requirements of the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement.” If we do allow a delayed revocation, we are committed to doing so in line with Mozilla’s expectations as expressed in the wiki.

Additionally please respond to statements issued on MDSP, the report is threadbare and we need to see more substance.

An updated report will be submitted with additional detail based on community feedback received on MDSP.

Comment 78

[ngook.kong]

(In reply to walter.j.marks from comment #62)

(In reply to ngook.kong from comment #60)

(In reply to walter.j.marks from comment #49)

(In reply to Bruce Morton from comment #48)

(In reply to walter.j.marks from comment #37)

Setting aside the initial failure to revoke and the long tail of this process, I'd like to dig a bit further into the Subscriber arguments against revoking, and some of your statements included there. I'm going to make a few assumptions here, including that Entrust agrees that there is in fact a violation of the BRs requiring revocation of affected certs, otherwise I assume that you wouldn't have revoked in the first place.

“These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

“Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

“We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Q1. Were each of these Subscribers reminded of their agreement to the Terms of Use of Entrust Certificate and Signing Services? Specifically, Exhibit B, Part 1.12.3?

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

We have not reminded subscribers about this section of our Terms of Use, nor have subscribers raised questions about our contractual rights in this matter.

Since it wasn't mentioned in the June 7 Report (along with other omissions but I'll leave those opinions for MDSP to avoid mixing up action items here), what was the internal decision making that led to not reminding subscribers of this section of the ToU?

A lot of this aftermath and the ensuing incident failures that were due to "Entrust handling numerous incidents at the same time" (paraphrased) could have been mitigated or reduced by "We are required to revoke certificates within 24h/120h. We are here to assist you in any way we can during the next few days, but as part of our agreement to PKI as a whole as well as the BRs and various root programs, it is mandatory that these revocations occur."

Thanks for your suggestion. The internal process is for our support team to use the standard message to subscribers, included as attachment 9406229 [details], with update date. “Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC”. We did not point subscribers to the terms of use, and although subscribers have not questioned our right to revoke.

Does not answer my question, yet again. In clear terms: what was the internal decision process that led Entrust to begin from a place of not revoking certificates, and only after being asked by multiple community members did the revocation process begin? This was also missing from the report that was provided on June 7.

We apologize for the confusion. We had responded to the question “what was the internal decision making that led to not reminding subscribers of this section of the ToU?”. You are now asking a different question, which is asking about how we originally arrived at the decision not to revoke certificates. This initial decision to not revoke was based on the belief that the certificates were compliant with the CA/B Forum requirements as they met the updated certificate profile requirements provided by ballot SC-62.

Unfortunately, we missed the fact that ballot SC-62 did not implement the updated certificate profile requirements in the EV Guidelines, but only in the Baseline Requirements. We also believed, based on discussions in the CA/B Forum, that the updated certificate profile was intended to apply to both OV and EV certificates, so we did not believe that the community would consider certificates to be mis-issued by complying with this profile. We began the revocation process after it became clear that the community did not agree. We understand that the community expected us to reverse position earlier.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Q2. Why was a presumption of a Subscriber being unable to revoke without positive confirmation assumed as stated above? Again following from Exhibit B, Part 1.12.3, Entrust is entitled to revoke a certificate immediately without notifying the Subscriber.

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Falls under my thoughts from the above un-quoted statement.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Q3. How does the above statement square with the the subscriber justification given here?

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

“Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Again, you seem to misunderstand the question posed by the following quote:

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

We understand the question. As stated in your quote, we tried to balance the risk of the relying parties in the broader internet and the subscriber relying parties.

Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

While our terms of service might allow us to revoke certificates without contacting subscribers, we prioritized responsible action. We understand that certificate revocation can be disruptive, even for subscribers familiar with our terms of service.

Regarding your concerns about trust moving forward, following these recent incidents we have done a thorough review and root cause analysis of the commitments we made in 2020 and these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests.

Does not answer the question. The question as asked is "If you are implying that customers did not read that part of the Terms of Use, and thus shouldn't be held to it, what other parts of the Terms of Use are not strictly enforced by Entrust?". Again, this was not included in the report.

We are not implying “that customers did not read that part of the Terms of Use, and thus shouldn't be held to it”.

Subscribers are expected to meet all Terms of Use requirements.

At the same time, we expect that all parties to commercial contracts seek to exercise and enforce contractual rights in a reasonable manner, and this includes a consideration of any harms that may be caused by their actions.

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident

I see no detailed explanation of the changes made to Entrust's operational procedures in the June 7 Report.

We will provide additional detail regarding changes in the updated report.

Of those three exemplar Subscribers I've referenced above, none seem to have a lack of automation as a blocking fact that they were unable to revoke (which again, should be an action taken by Subscribers by the powers given to you by Exhibit B, Part 1.12.3) and replace in time. A commitment to automation does not mean anything if the subscribers are simply refusing to accept a revocation within the required and expected time. A more accurate commitment that I would expect to answer the question would be something along the lines of "Entrust will refuse to accept any requests to delay revocation past 5 days....." along with a measurable and actionable key index for that commitment.
Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

We agree that a key outcome for this incident is for us to define a more rigorous revocation plan that includes a formal plan to define our policies and process for handling subscriber requests for delayed revocation. We expect to have these guidelines in place by the end of June.

There was a commitment to never delay revocation again after a delrev incident 4 years ago. What happened to those guidelines and internal policies then, and why were they not followed now? If they did exist prior to the June 7 Report, why do these policies need additional time to be created and implemented.

Please refer to the response above regarding our acknowledgement of lapses and actions to address. In our review of the root causes, we want to be sure that we solved for these issues today. We did not handle the recent incidents in a way that lives up to our standards. We are taking the time required to make lasting change organizationally, in policies and procedures, and in technology where appropriate.

Does not answer the question. Even if I take this statement to refer to the above response, it is still a poor answer to the question. Again: what happened to the internal policies that were allegedly created in 2020 and why were they not used as a response to handle this incident, or were they not created as committed to then? Additionally, if I take this answer in concert with the answer above, it is even more questionable. This answer seems to imply that the policies were not created as committed to, however the above answer seems to imply that the policies were created. Which answer is the truth?

If you are referring to the action items that were given in response to Bug 1651481, none of those action items included a commitment to creating a new internal policy. We had existing internal policies and procedures and in making the commitments in 2020 we assumed that these existing policies and procedures would be sufficient to support our commitments. We know we were mistaken in this, which is why in our June 7th report one of the specific items in our improvement plan is to adopt a new revocation policy. These and other action items are being tracked in Bug 1901270.

Q5.1. Of these three exemplar Subscribers, were any reminded that if this was a key compromise event that had fundamental security risks, that this argument of "needing to notify downstream contracted parties that could take up to 3 months" (paraphrased) would potentially result in their TLS certificates being unsafe for the web ecosystem and their business operations for the entire 90 day period?

Yes, revocation requirements were communicated to customers.

Q5.2. If so, what was the justification provided by the Subscriber that they would be able to handle a situation like that within the 24h/120h period, and why that process was unable to be performed in this case, given that from a pure compliance perspective, in either case the affected certificate is invalid and should not be used by the Subscriber?

Subscriber feedback indicates that many subscribers non security events differently from security related events. In the event of a key compromise, we believe that our customers take security very seriously across their organizations, and subscribers understand the potential consequences for themselves and the web ecosystem.

The response here would be most appropriate in MDSP.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Q6. The above statement seems to indicate that Entrust believes that this incident is still not at the level of a revocation requirement. If that is the case, please explain how the processes followed for this mis-issuance are different from the planned handling of a wide-scale revocation requirement.

We have agreed that this was a mis-issuance event and have reiterated our intent to follow the TLS Baseline Requirements and root program policies. We have acknowledged that these certificates were mis-issued, and initiated revocation. To date all certificates in this incident have been revoked.

If you have reiterated your intent to follow the BR and root program policies, why are we in a situation where we can expect Entrust to delay revocation of certificates for 90 days as evidenced by this bug? I don't want to be flippant, but there's really no other way to square that circle. You cannot in good faith say "Entrust will follow the policies" in the exact same bug number as where there's proof that a known mis-issued certificate remained unrevoked for 95 days, especially since 4 years ago this statement was given by Bruce Morton:

• We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

and yet somehow we ended up in the exact same situation. In other words, I see more evidence that I should not be taking this commitment seriously than evidence that I should take this commitment seriously.

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

As noted above, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have and will continue to clarify across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident. We also will be adding certificate and CA/B compliance to our corporate governance committees to ensure sufficient visibility and prioritization of investment for this work.

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.
Arguing that 24h/120h is too hard to accomplish while you're currently under review for failure to revoke within 24h/120h for multiple mis-issuances is a hard sell, and I'd be very hesitant to lean on this argument.

Comment 79

[walter.j.marks]

(In reply to ngook.kong from comment #78)

(In reply to walter.j.marks from comment #62)

(In reply to ngook.kong from comment #60)

(In reply to walter.j.marks from comment #49)

(In reply to Bruce Morton from comment #48)

(In reply to walter.j.marks from comment #37)

Setting aside the initial failure to revoke and the long tail of this process, I'd like to dig a bit further into the Subscriber arguments against revoking, and some of your statements included there. I'm going to make a few assumptions here, including that Entrust agrees that there is in fact a violation of the BRs requiring revocation of affected certs, otherwise I assume that you wouldn't have revoked in the first place.

“These system [sic] are deeply integrated into our environment and at times require notice and planning to update a single certificate, let alone the [REDACTED NUMBER] that are on this list. Revoking of these certificates would have a major impact across our systems.”

“Yes, these certificates cover a very large amount of SANs and they are managed by multiple teams, teams will need to be engaged and begin their change management review process to schedule the installation of the certificate and the majority are external certificates which also contractually require 90 days notifications to be sent to impacted clients.”

“We have implemented a comprehensive change freeze until April 2, 2024, with the possibility of extension to the first week of April. During this period, no changes will be made to any application or infrastructure of the Bank. As an organization, we are obligated to adhere to these guidelines and cannot make any changes during this change freeze. Additionally, the list of [REDACTED] certificates requiring re-issuance pertains to critical applications. We estimate that a minimum of 3 months will be necessary to re-issue these certificates, deploy them at the application level, and offload them where required.”

Q1. Were each of these Subscribers reminded of their agreement to the Terms of Use of Entrust Certificate and Signing Services? Specifically, Exhibit B, Part 1.12.3?

As a condition of having any Certificate issued to or for Subscriber, each Subscriber makes, on its own behalf and if
applicable on behalf of its principal or agent under a subcontractor or hosting service relationship, the following
representations, commitments, affirmations and warranties for the benefit of Certificate Beneficiaries, Entrust and
any of Entrust’s Affiliates that will issue Certificates to or for Subscriber:
[...]
12. Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if:
12.3. Revocation is required under the CPS or the Industry Standards.

We have not reminded subscribers about this section of our Terms of Use, nor have subscribers raised questions about our contractual rights in this matter.

Since it wasn't mentioned in the June 7 Report (along with other omissions but I'll leave those opinions for MDSP to avoid mixing up action items here), what was the internal decision making that led to not reminding subscribers of this section of the ToU?

A lot of this aftermath and the ensuing incident failures that were due to "Entrust handling numerous incidents at the same time" (paraphrased) could have been mitigated or reduced by "We are required to revoke certificates within 24h/120h. We are here to assist you in any way we can during the next few days, but as part of our agreement to PKI as a whole as well as the BRs and various root programs, it is mandatory that these revocations occur."

Thanks for your suggestion. The internal process is for our support team to use the standard message to subscribers, included as attachment 9406229 [details], with update date. “Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC”. We did not point subscribers to the terms of use, and although subscribers have not questioned our right to revoke.

Does not answer my question, yet again. In clear terms: what was the internal decision process that led Entrust to begin from a place of not revoking certificates, and only after being asked by multiple community members did the revocation process begin? This was also missing from the report that was provided on June 7.

We apologize for the confusion. We had responded to the question “what was the internal decision making that led to not reminding subscribers of this section of the ToU?”. You are now asking a different question, which is asking about how we originally arrived at the decision not to revoke certificates. This initial decision to not revoke was based on the belief that the certificates were compliant with the CA/B Forum requirements as they met the updated certificate profile requirements provided by ballot SC-62.

Unfortunately, we missed the fact that ballot SC-62 did not implement the updated certificate profile requirements in the EV Guidelines, but only in the Baseline Requirements. We also believed, based on discussions in the CA/B Forum, that the updated certificate profile was intended to apply to both OV and EV certificates, so we did not believe that the community would consider certificates to be mis-issued by complying with this profile. We began the revocation process after it became clear that the community did not agree. We understand that the community expected us to reverse position earlier.

We have no examples of a case in which we revoked without a response from the subscriber. Our Support teams worked 24/7 to ensure that every subscriber was contacted and responded, and we received responses from all our Subscribers by the conclusion of that effort.

Q2. Why was a presumption of a Subscriber being unable to revoke without positive confirmation assumed as stated above? Again following from Exhibit B, Part 1.12.3, Entrust is entitled to revoke a certificate immediately without notifying the Subscriber.

While our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), we believe it can be irresponsible to exercise that right without any context.

Falls under my thoughts from the above un-quoted statement.

Yes and it was communicated exactly as such. No delayed revocation options were offered.

Q3. How does the above statement square with the the subscriber justification given here?

Here is an example where we rejected the subscriber’s initial time period request as too long, but allowed some time to delay:

“Despite our best efforts, it has become evident that meeting the current deadline for revocation is simply unfeasible given the complexities of our infrastructure and the scope of necessary updates. The task at hand involves a multifaceted process of updating and retrofitting a considerable number of applications within our ecosystem, coupled with the implementation of SSL pinning at the core level. While we have allocated ample resources and diligently pursued this endeavor, unforeseen technical challenges have emerged, significantly impeding our progress. Furthermore, the sheer volume of applications requiring updates, each with its own unique dependencies and integrations, has proven to be a formidable obstacle. Despite our tireless efforts, it has become evident that achieving full compliance within the allotted timeframe is unattainable without compromising the integrity and security of our systems. Furthermore, we have already started the re-issuance of the Affected certificates and will try to perform the certificate re-issuance as much as possible.”

Again, you seem to misunderstand the question posed by the following quote:

All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?

We try to balance the risk of the relying parties in the broader Internet and the Subscribers relying parties (e.g. financial, transportation).

Commitment to implement automation.

We understand the question. As stated in your quote, we tried to balance the risk of the relying parties in the broader internet and the subscriber relying parties.

Frankly, I disagree with the understanding of the question. As the original question asks: "All Subscribers must already be committed (in a legally binding way) to accepting immediate revocation of certificates that violate the BRs, under the terms of 9.6.3(8). What additional commitment are you seeking to determine?". In other words, what more do you need from a Subscriber other than their acceptance of the Terms of Use that are agreed upon by purchasing the certificate? If you're instead implying that "well the customers didn't read that section of the Terms of Use, so we'll not hold them to it", how can we trust that you'll hold Subscribers to other portions of the Terms of Use?

While our terms of service might allow us to revoke certificates without contacting subscribers, we prioritized responsible action. We understand that certificate revocation can be disruptive, even for subscribers familiar with our terms of service.

Regarding your concerns about trust moving forward, following these recent incidents we have done a thorough review and root cause analysis of the commitments we made in 2020 and these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests.

Does not answer the question. The question as asked is "If you are implying that customers did not read that part of the Terms of Use, and thus shouldn't be held to it, what other parts of the Terms of Use are not strictly enforced by Entrust?". Again, this was not included in the report.

We are not implying “that customers did not read that part of the Terms of Use, and thus shouldn't be held to it”.

Subscribers are expected to meet all Terms of Use requirements.

At the same time, we expect that all parties to commercial contracts seek to exercise and enforce contractual rights in a reasonable manner, and this includes a consideration of any harms that may be caused by their actions.

That may be a preferred reading, which Entrust is entitled to have, but at least as written, the Terms of Use seem fairly clear on the matter. I simply can't read "Subscriber acknowledges and agrees that Entrust is entitled to revoke a Certificate immediately if..." as anything other than an explicit contractual obligation that Entrust is entitled (and required to revoke) to revoke certificates if necessary. If anything, a 5 day period is far more generous. It seems very hard to interpret this lackluster execution of the ToU as anything other than a implied admission of the Terms of Use not being read and fully agreed upon by Subscribers.

If we additionally read the following section (Section 9.4, ToU) in concert with section 1.12.3 of Exhibit B:

Specific Exclusions. In no event will Entrust or its Affiliates be liable for any damages to Subscribers, Relying Parties or any other Person arising out of or related to the use or misuse of, or reliance on any Certificate issued under this Agreement or the CPS that: (i) has expired or been revoked; (ii) has been used for any purpose other than as set forth in this Agreement or the CPS; (iii) has been tampered with; (iv) with respect to which the key pair underlying such Certificate or the cryptography algorithm used to generate such Certificate's key pair, has been compromised by the action of any party other than Entrust or its Affiliates (including without limitation the Subscriber or Relying Party); or (v) is the subject of misrepresentations or other misleading acts or omissions of any other party, including but not limited to Subscribers and Relying Parties.

We clearly seem to have a intent that Entrust is entitled to revoke certificates without notice, and any damages to Subscribers are not Entrust's responsibility. That to me would seem to be that Entrust understands there may be risks to a revocation for Subscribers, but that those risks are the Subscribers' responsibility.

I really don't want to get into rules lawyering any more than we already have here, so at this point I will simply say that Entrust is entitled to their interpretation of their Terms of Use and how they are applied, and I disagree with that.

As a result, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have clarified across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident

I see no detailed explanation of the changes made to Entrust's operational procedures in the June 7 Report.

We will provide additional detail regarding changes in the updated report.

Of those three exemplar Subscribers I've referenced above, none seem to have a lack of automation as a blocking fact that they were unable to revoke (which again, should be an action taken by Subscribers by the powers given to you by Exhibit B, Part 1.12.3) and replace in time. A commitment to automation does not mean anything if the subscribers are simply refusing to accept a revocation within the required and expected time. A more accurate commitment that I would expect to answer the question would be something along the lines of "Entrust will refuse to accept any requests to delay revocation past 5 days....." along with a measurable and actionable key index for that commitment.
Q4. Is there an understanding by Entrust that a proper answer to the question posed above is not answered by that answer, and instead deserves a better answer that assures the community as a whole that Entrust is taking this situation seriously, and not making the same empty promises that were given 4 years ago?

We agree that a key outcome for this incident is for us to define a more rigorous revocation plan that includes a formal plan to define our policies and process for handling subscriber requests for delayed revocation. We expect to have these guidelines in place by the end of June.

There was a commitment to never delay revocation again after a delrev incident 4 years ago. What happened to those guidelines and internal policies then, and why were they not followed now? If they did exist prior to the June 7 Report, why do these policies need additional time to be created and implemented.

Please refer to the response above regarding our acknowledgement of lapses and actions to address. In our review of the root causes, we want to be sure that we solved for these issues today. We did not handle the recent incidents in a way that lives up to our standards. We are taking the time required to make lasting change organizationally, in policies and procedures, and in technology where appropriate.

Does not answer the question. Even if I take this statement to refer to the above response, it is still a poor answer to the question. Again: what happened to the internal policies that were allegedly created in 2020 and why were they not used as a response to handle this incident, or were they not created as committed to then? Additionally, if I take this answer in concert with the answer above, it is even more questionable. This answer seems to imply that the policies were not created as committed to, however the above answer seems to imply that the policies were created. Which answer is the truth?

If you are referring to the action items that were given in response to Bug 1651481, none of those action items included a commitment to creating a new internal policy. We had existing internal policies and procedures and in making the commitments in 2020 we assumed that these existing policies and procedures would be sufficient to support our commitments. We know we were mistaken in this, which is why in our June 7th report one of the specific items in our improvement plan is to adopt a new revocation policy. These and other action items are being tracked in Bug 1901270.

Q5.1. Of these three exemplar Subscribers, were any reminded that if this was a key compromise event that had fundamental security risks, that this argument of "needing to notify downstream contracted parties that could take up to 3 months" (paraphrased) would potentially result in their TLS certificates being unsafe for the web ecosystem and their business operations for the entire 90 day period?

Yes, revocation requirements were communicated to customers.

Q5.2. If so, what was the justification provided by the Subscriber that they would be able to handle a situation like that within the 24h/120h period, and why that process was unable to be performed in this case, given that from a pure compliance perspective, in either case the affected certificate is invalid and should not be used by the Subscriber?

Subscriber feedback indicates that many subscribers non security events differently from security related events. In the event of a key compromise, we believe that our customers take security very seriously across their organizations, and subscribers understand the potential consequences for themselves and the web ecosystem.

The response here would be most appropriate in MDSP.

The remedial actions we are taking to formalize our handling of revocation events will also prepare us for handling of wide-scale revocation requirements.

Q6. The above statement seems to indicate that Entrust believes that this incident is still not at the level of a revocation requirement. If that is the case, please explain how the processes followed for this mis-issuance are different from the planned handling of a wide-scale revocation requirement.

We have agreed that this was a mis-issuance event and have reiterated our intent to follow the TLS Baseline Requirements and root program policies. We have acknowledged that these certificates were mis-issued, and initiated revocation. To date all certificates in this incident have been revoked.

If you have reiterated your intent to follow the BR and root program policies, why are we in a situation where we can expect Entrust to delay revocation of certificates for 90 days as evidenced by this bug? I don't want to be flippant, but there's really no other way to square that circle. You cannot in good faith say "Entrust will follow the policies" in the exact same bug number as where there's proof that a known mis-issued certificate remained unrevoked for 95 days, especially since 4 years ago this statement was given by Bruce Morton:

• We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

and yet somehow we ended up in the exact same situation. In other words, I see more evidence that I should not be taking this commitment seriously than evidence that I should take this commitment seriously.

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

As noted above, we have made leadership changes in the Entrust digital certificate business unit. We've also reorganized to leverage our global compliance resourcing, expertise, and governance more fully within this business unit. And we have and will continue to clarify across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident. We also will be adding certificate and CA/B compliance to our corporate governance committees to ensure sufficient visibility and prioritization of investment for this work.

What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays and pushing customers to take measures that will enable more rapid revocation (e.g., automation, moving off public trust if it is not right for them, shorter certificate lifecycles, etc.). We also commit to further discussions with the community and the governing body around the practicality of the five-day requirement in cases of technical mis-issuances where security is not a factor, especially in context of large globally distributed organizations where automation and education is not enough and a forced revocation may cause an industry wide disruption.
Arguing that 24h/120h is too hard to accomplish while you're currently under review for failure to revoke within 24h/120h for multiple mis-issuances is a hard sell, and I'd be very hesitant to lean on this argument.

Comment 80

[walter.j.marks]

After re-reading your post Ngook, I also had something else I wanted to clarify:

Question: To put it on the record: were any operational changes made in regards to the action items given by Entrust in Bug #1651481? Since it seems the community needs to include it, please answer beginning with a yes or no.

Comment 81

[Bruce Morton]

(In reply to Mike Shaver (:shaver emeritus) from comment #64)

Right now we have ample evidence of Entrust not operating in accordance with the expectations that the Mozilla root program and BRs have for a CA. We also have evidence that explicit commitments made previously—regarding the exact same issue of improperly delayed revocation—were not honoured, and that Entrust did not notice for more than four years that they were not being honoured. The latter means that there is a very high bar for Entrust to clear in demonstrating that it both understands what is expected of it as a public web CA, and is capable and motivated enough to make the changes required to meet that expectation.

A leadership change might be good or bad or meaningless; there is not enough information provided to really warrant confidence. How did the old leadership lead to the problems we have observed? What will the new leadership do differently?

How did the old leadership lead to the problems we have observed? That’s part of the root causes in our June 7 Report. Some of it involves internal communication to the senior leadership levels that (in retrospect) probably were vague and not actionable . Some of it also involved staffing levels to ensure a consistent multi person review and double checks of processes that turned out to be insufficient considering the ever-growing complexity of issuing TLS certificates. We have talked a lot already in our Report and elsewhere of what new leadership plans to do differently, based on what has been learned from these Bugs. At this stage, we can only speak at a high level, because many people in the organization that were not previously involved in the public trust business will now be included. There will be a period of learning and getting up to speed before they can make and implement specific decisions. We plan to report our progress to the community as we implement our changes.

You've reorganized: what was wrong with the old organization, and what caused it to be that way? What forces were acting on the old organization that contributed to the failures, and how is the new organization designed to avoid them?

The old organization was aligned with goals and outcomes of the product line, new organizational model will separate the responsibility of compliance with CA/B and root program policies, and staff will be measured on specifically on compliance outcomes. We believe this comment echos comments that have been made to our June 7th report. We plan to post a revised report with more details to address the feedback and comments from the community.

You're reducing the length of time approved for delayed revocation. What was the previous length of time, and how was it determined? The previous statements in this incident bug about the process for determining if Entrust would violate the BRs on behalf of a customer describe something tremendously sloppy, to be polite, for managing certificate lifecycles for such critical industries. What is the new length of time? How was it determined?

We did not have a specific number of days previously, but rather the length of delay was decided collaboratively with the subscriber based on the circumstances.

Also, we do not believe it would necessarily be appropriate to set an arbitrary number of days as the “new length of time”. We have committed to creating a revocation policy, and that will include stricter criteria for what extraordinary circumstances may justify delayed revocation.

Will Entrust continue to issue public web PKI certificates to subscribers who they know to be unable to tolerate an appropriately-prompt revocation without catastrophic consequences to society?

Yes, we will continue to issue public web PKI certificates to subscribers who meet the requirements to have publicly trusted certificates issued to them under the applicable rules. We do not believe that the CA/B Forum requirements, or any root embedding program requirements, prohibit issuance to an applicant on the grounds you describe. If this policy was to change we commit to adhering to it.

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation. We believe all CAs are similarly situated as evidenced by the number of delayed revocation reports that have been filed and the fact that this issue was raised as a pervasive issue at the most recent CA/B Forum meeting.

Nobody is saying that there will never be exceptional circumstances that warrant or explain a good faith failure to revoke on time. The issue is a serious misalignment between Entrust's behaviour and the community's expectations in terms of how those delayed revocations are determined. There is ample evidence that Entrust is knowingly creating situations that they will later use to justify delrev incidents, by issuing web PKI certificates to subscribers who are known to be unable to replace a certificate promptly. This pattern of issuance since the 2020 commitments has not changed in any materially visible way, and there has been no commitment from Entrust to avoid recreating the situations with future issuances.

Please clarify, are you suggesting that every subscriber who is known to be unable to replace a certificate promptly should be blacklisted by all CAs, or that no CA should be allowed to accept a certificate application from that organization? Would not such a rule need to be clearly articulated in the CA/B Forum requirements and root embedding program requirements so as to apply to all CAs equally before being implemented? For the record, we would have major concerns with such a rule and would not likely be supportive of such a ballot.

You write of "pushing customers", but again not what exactly that entails or how the community should observe Entrust's progress on it without waiting for another misissuance event with three month revocation timelines and knowingly and deliberately insufficient description of the reasons for the delays. Is Entrust asking nicely? Requiring subscribers to also approve and prepare an alternative certificate from another CA that they can quickly switch to if their Entrust certificate needs to be revoked?

No. We agree that our process for collecting the reasons for the delays was insufficient, but it was not knowingly and deliberately insufficient. In our June 7th report, we outlined several specific action items to improve how we handle revocation events. One of these is to adopt a new revocation process and ensure that it is communicated to subscribers. We are considering ways to increase visibility of the CA’s right to revoke certificates on short notice beyond our contract language. We also will add a warning to manual order pages and related emails to ensure subscriber understanding of required timelines. These and other action items are being tracked in Bug 1901270.

We do generally recommend that subscribers have a backup CA, but installing a backup certificate in every application would increase cost, effort and complexity. We do not believe this should be an industry requirement.

I would caution against using other CA behaviour as evidence of a problem, when Entrust is the one most contributing to a pattern of not meeting the standards expected.

I want to emphasize this part again:

In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident.

It is not enough to be "guided". They are requirements and need to be treated as such. As far as I can tell Entrust has never fully met the requirements outlined in Mozilla incident-response expectations for delayed revocation. Entrust needs to demonstrate that it fully understands what is expected of it, and that it has the means to comply.

Understood.

And this part:

pushing customers to take measures that will enable more rapid revocation

Customers do not "enable" revocation. CAs revoke. Only CAs can do so, and it is to CAs that the responsibility falls to revoke as a remedy for cases where the CA failed to issue certificates correctly. Customers take responsibility for their own operational processes, and deploy web PKI certificates with the full knowledge, legally assured, that they might be revoked on short notice. Entrust might decide to take some responsibility for customer operational processes through their own products or services, but they absolutely cannot force the root programs and the web PKI at large to bear the consequences of subscribers' inability or unwillingness to build systems that are resilient to the realities of web PKI use.

Comment 82

[Bruce Morton]

(In reply to walter.j.marks from comment #70)

The point I am trying to elaborate on is "why did they need an extension?" Setting aside the decision to delay revocation for a second, if a conversation was being had with a large financial institution that involved various teams involved both from Entrust and the large financial institution, why was the initial self-imposed revocation deadline missed?

The proposed targets for revocation were established based on a good faith discussions about when the certificates could safely be replaced to avoid harm or minimize harm to the Subscribers and the Relying Parties who depend on the servers.

In some cases the work required to enable safe replacement took longer than expected. We do not have specific granular reasons for why this happened, beyond the reasons why delays were required in the first place. We are confident that our subscribers were making good faith efforts.

In our conversations with Subscribers, we transparently disclosed that there was no security risk to relying parties if the affected certificates were not revoked, and this context understandably influenced the prioritization. We agree that we did not try to create greater urgency with threats or lectures about legal rights and industry requirements.

We sincerely believed that the community would agree that both the circumstances and the objective of avoiding harm to relying parties would justify delayed revocation, in keeping with the expectations set out in the Mozilla policy. Although it has been difficult, we now understand that the community places priority on strict adherence to the rules, and views revocation as a tool to influence subscribers into modifying how they use TLS certificates, and is willing to accept much more harm to subscribers and users of the internet than Entrust believed was acceptable.

In keeping with this, we have agreed to be less lenient with our subscribers, enforce revocation within the timelines more strictly, and accept far fewer circumstances as “exceptional”.

Comment 83

[Wayne]

(In reply to Bruce Morton from comment #81)

A leadership change might be good or bad or meaningless; there is not enough information provided to really warrant confidence. How did the old leadership lead to the problems we have observed? What will the new leadership do differently?

How did the old leadership lead to the problems we have observed? That’s part of the root causes in our June 7 Report. Some of it involves internal communication to the senior leadership levels that (in retrospect) probably were vague and not actionable . Some of it also involved staffing levels to ensure a consistent multi person review and double checks of processes that turned out to be insufficient considering the ever-growing complexity of issuing TLS certificates. We have talked a lot already in our Report and elsewhere of what new leadership plans to do differently, based on what has been learned from these Bugs. At this stage, we can only speak at a high level, because many people in the organization that were not previously involved in the public trust business will now be included. There will be a period of learning and getting up to speed before they can make and implement specific decisions. We plan to report our progress to the community as we implement our changes.

I will note that you seem to be discussing the June 21st report, nothing you describe appears in the June 7th report.

You've reorganized: what was wrong with the old organization, and what caused it to be that way? What forces were acting on the old organization that contributed to the failures, and how is the new organization designed to avoid them?

The old organization was aligned with goals and outcomes of the product line, new organizational model will separate the responsibility of compliance with CA/B and root program policies, and staff will be measured on specifically on compliance outcomes. We believe this comment echos comments that have been made to our June 7th report. We plan to post a revised report with more details to address the feedback and comments from the community.

As already stated on MDSP there is little internal reflection in this new report but it can be further addressed there.

Will Entrust continue to issue public web PKI certificates to subscribers who they know to be unable to tolerate an appropriately-prompt revocation without catastrophic consequences to society?

Yes, we will continue to issue public web PKI certificates to subscribers who meet the requirements to have publicly trusted certificates issued to them under the applicable rules. We do not believe that the CA/B Forum requirements, or any root embedding program requirements, prohibit issuance to an applicant on the grounds you describe. If this policy was to change we commit to adhering to it.

I will note that previously Entrust stated the following:

For instance, airline ticketing and check-in systems wouldn't be classified as a prohibited use case. While an outage caused by certificate revocation could lead to serious negative impacts for air travelers, it wouldn't directly risk human life. Irresponsible revocation without considering potential consequences could lead to widespread disruption, impacting millions of travelers and causing significant economic harm. This highlights the importance of our collaborative approach with subscribers to ensure a smooth re-issuance process and minimize disruption.

Consider that while not a CA/B F requirement that the scenario creating irresponsible revocation is only possible due to irresponsible issuance. If there is truly a separate compliance team now, then is there a single situation where they are able to overrule other departments in this regard? If so, please provide an example that would previously have been issued without the new compliance team active.

Please clarify, are you suggesting that every subscriber who is known to be unable to replace a certificate promptly should be blacklisted by all CAs, or that no CA should be allowed to accept a certificate application from that organization? Would not such a rule need to be clearly articulated in the CA/B Forum requirements and root embedding program requirements so as to apply to all CAs equally before being implemented? For the record, we would have major concerns with such a rule and would not likely be supportive of such a ballot.

The CA/B Forum requirements do not state anything about selling to sanctioned countries. We can agree there are more rules governing issuance than those requirements, correct?

You write of "pushing customers", but again not what exactly that entails or how the community should observe Entrust's progress on it without waiting for another misissuance event with three month revocation timelines and knowingly and deliberately insufficient description of the reasons for the delays. Is Entrust asking nicely? Requiring subscribers to also approve and prepare an alternative certificate from another CA that they can quickly switch to if their Entrust certificate needs to be revoked?

No. We agree that our process for collecting the reasons for the delays was insufficient, but it was not knowingly and deliberately insufficient. In our June 7th report, we outlined several specific action items to improve how we handle revocation events. One of these is to adopt a new revocation process and ensure that it is communicated to subscribers. We are considering ways to increase visibility of the CA’s right to revoke certificates on short notice beyond our contract language. We also will add a warning to manual order pages and related emails to ensure subscriber understanding of required timelines. These and other action items are being tracked in Bug 1901270.

We do generally recommend that subscribers have a backup CA, but installing a backup certificate in every application would increase cost, effort and complexity. We do not believe this should be an industry requirement.

As previously stated please consider what responsible issuance truly means and how Entrust can be considered stewards of public trust if resiliency is considered a problem. If you are truly stewarding subscribers with critical infrastructure then they will require backup plans and alternative contractors in place to be used at a moment's notice. This would already be part of any contract that asserts usage in that field, and is part of why I find these common assertions across CA's with delayed revocation issues so questionable.

Comment 84

[ngook.kong]

(In reply to Wayne from comment #71)

Given there are some difficulties in reading where questions are I have bolded them for clarity, despite previous commenters already doing so and it not helping...

(In reply to ngook.kong from comment #66)

Question: If so, what other portions of the Terms of Use are flexible in Entrust's enforcement of the Terms of Use?

Entrust please read the questions provided this has not been answered.

We disagree with the assertion or presumption made that any portions of Entrust’s Terms of Use are inherently flexible or not enforced as a rule. We do not believe that a discussion about contract enforcement is appropriate as corporations are not generally free to disclose legal strategy in a public forum.

Yes, there is a directly responsible individual, and yes the internal discussions capture this.

Question: If so, did this DRI make the decision wholesale, or was the decision made per subscriber?

See previous answer.

Your previous answer does not answer the question, please read the questions before attempting to answer.

The decision was made per subscriber.

(In reply to ngook.kong from comment #67)

Can you please explain:

• The discrepancies between these statements, where previously you have stated that you did not grant any delayed revocation “Unless a subscriber articulated business/industry disruption, technical challenges and/or impacts to the web ecosystem”, yet now you state you automatically granted a delay if you were unable to reach the subscriber?
• If you used the same approach as described in the quoted section from comment #51 in other delayed revocation bugs?
• Why it took Entrust up to now to come out with this information?

If it was not implied, there is a big difference between granting (very limited) delayed revocation in exceptional circumstances and blindly just not revoking certificates if you were unable to reach your customers or confirm that they are taking action.

We have described the incident in this bug in great detail. As we have said, it’s clear from these notes above and the week-to-week revocation timeline that we did not handle this delayed revocation in a way that lives up to our standards. We recognize that it is the CA’s responsibility to revoke, not the subscriber’s.

At the start of this incident, we received significant pushback from customers on revocation timelines, and these customers often represented organizations that provide critical infrastructure services. As a result, as stated in comment 51 and elsewhere, in this case, with subscribers who had not responded, we decided to prioritize avoiding unintended harmful consequences.

As we have noted elsewhere, we have done a thorough review and root cause analysis of these recent incidents, particularly around decisions to revoke and delayed revocation. In summary: We didn't have sufficient leadership awareness of these commitments, nor a clear enough process for evaluating and closely managing exception requests. As a result, we made leadership changes and reorganized our certificate services business to leverage our global compliance resourcing, expertise, and governance more fully.

And we have made it clear across all levels of the organization how seriously we take the requirements set by the CA/Browser Forum and the root programs and our intent to comply with them. In this we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident.

I will note that, once again, the questions posed are not answered. Further the attempt to conflate "these customers often represented organizations that provide critical infrastructure services" with the certificates being used in such services is concerning. Entrust do know that the criteria is about certificate usage, not that a subscriber allegedly providing critical infrastructure services?

We recognize that you do not like the answers provided, but we feel that we have answered. We have acknowledged that there were deficiencies and gaps in how we granted delayed revocation, which may have led to inconsistencies, and we have provided our commitments for how we propose to improve. We will seek appropriate legal and other expert advice regarding the interpretation of the industry and contractual requirements, and we will act in accordance with their expert advice. We accept responsibility for our actions. We are sorry. We will do better.

Can Entrust explain how "in this case, with subscribers who had not responded, we decided to prioritize avoiding unintended harmful consequences" is compatible with previous statements that Entrust do not make unilateral decisions on revocation?

We can see that the first quote is from Comment 67, but we are not able to locate the other statements you are referencing, can you please provide the Comment number and the direct quote? Many of these responses must be read in the context of the question they are answering. However, if we did indeed state that we did not make any “unilateral” decisions about revocation, then we agree that this seems incorrect. We may have intended to say that our decisions were per-subscriber and contextual (our own assessment and understanding of the subscriber and context), even when we didn’t have a response from that subscriber.

Furthermore, none of what Entrust have stated is clear otherwise there would not be so many questions. Please attempt to answer in a straightforward, direct, and honest answer. The current list of possible speaking points are not even being correctly put against the relevant questions.

(In reply to ngook.kong from comment #68)

Does Entrust explicitly formulate a definition ahead of time based on these sources, or is a working definition created ad-hoc when circumstances call for one?

We have described the working definition that we used to guide our decisions.

If you did then there would not be so many questions on this topic. Can Entrust please provide their definition in their own words in a clear and articulate way?

We do not have our own written definition. We have referenced the definitions in CISA and the EU CER and NIS-2 as the ones that we consulted/used. We have stated that we believe these provide helpful guidance, but we are not claiming that these are authoritative or that every delayed revocation decision included a rigorous legal determination of whether a particular definition was met. As far as we know, the Mozilla Policy does not include a written definition of this term, nor does it provide guidance or restrictions on what definition could be used by CAs, so it is our understanding that strict adherence to any such definition is not a requirement.

(In reply to ngook.kong from comment #69)

Just to dig into this a bit further, the (already against the BR required timeline) of the original due date you listed for these, varying between May 30 and June 2 was agreed upon with the relevant subscriber, correct?

If that's so, then why are some subscribers now in need of an additional 2 weeks?

This is basically correct, yes. The subscriber requested additional time to safely revoke the certificates. They also were able to complete revocation more quickly than they originally estimated.

We provided limited exceptions for a small number of certificates through June 16 because the subscribers were involved in a global financial change event aimed at increasing the security of global financial transactions and requested additional time. They were able to complete revocation of these certificates on June 6.

Even into June we are still in the position of the subscriber telling Entrust when it will be convenient for them to rotate certificates. I will note that in your response no actual critical services or harm is mentioned or implied - at most the customer was too busy. I do not feel that Entrust are articulating the importance of a timely revocation and are still not handling this in a manner that is befitting of a CA.

If Entrust feels this is not the case then please articulate why?

Neither Entrust nor any subscriber has ever stated that “convenience” determines when revocation occurs. We have consistently explained that decisions are made on the basis of an assessment of the balance of harms and logistical challenges. The fact of the matter is, for most subscribers and relying parties, the value of the WebPKI is to facilitate safe and smooth transactions on the internet. The issues with the affected certificates pose no risk to the safety and smoothness of those transactions, but the revocation of those certificates without replacement would be extremely disruptive. This is not a matter of placing blame, it is a matter of dealing with realities. If the strict enforcement of rules begins to take priority over the facilitation of safe and smooth internet transactions, it brings discredit on the entire ecosystem. Entrust has been trying to avoid that, by showing a nuanced understanding of the complex issues faced by subscribers. We acknowledge that this view is in conflict with the desire of this community, hence we are taking actions to correct as detailed in the MDSP report.

I will then add on further questions:

• Can Entrust explain why it is irresponsible to revoke a certificate over 90 certs into an incident?
• What 'context' is being missed, and can that be provided per-subscriber?
• Yes or No: Will Entrust commit to not causing delayed revocation to go past, say, 30 days going forward?
• Provide a simple figure for the number of days that Entrust feel they are organizationally capable of handling revocation given their subscriber base?

We acknowledge the question and would note that it has been answered above: “As noted earlier, while our terms of use allow for immediate revocation without subscriber notification (section 1.12.3 of Exhibit B), it can be irresponsible to exercise that right without any context.” In terms of context, we provided reasons for extensions in comment 25 and several examples of correspondence with individual customers including details on reasons for extensions.

Respectfully, we’re not going to make blanket statements about arbitrary delayed revocation timelines. We have stated our intention to comply with the requirements set by the CA/Browser Forum and the root programs, guided by the TLS Baseline Requirements.

Entrust are entitled to refuse to answer questions. Refusing will only reflect on their appearance of transparency, honesty, and their intentions are to comply with the requirements set by the CA/Browser Forum and the root programs. Entrust are more than entitled to return at a later date to attempt to answer those questions in a satisfactory manner. I will note that the figures quoted are far outwith timeframes that Entrust are mandated to perform, so no attempt at even agreeing is a tacit statement that things will not change.

Please note we have not refused to answer questions. It is our position that these questions have been answered. We regret that you are not satisfied with the answers. We are not making any tacit statement that things will not change, we are making express statements that we will change our processes and our behavior to bring them in line with the rules and expectations expressed in the root embedding programs.

Per their own Prohibited Certificate Usage terms, no such certificate should be used in a situation that can cause harm or serious economic impact. All authority has been bestowed onto Entrust in order to fulfill their revocation obligations, there is just an odd refusal to do so. If this could be explained at all I'd appreciate it.

The issue of Prohibited Certificate Usage Terms was covered in comments 24 and 42.

If the question were sufficiently answered I would not be posing it. Even in response to this comment I have been told that "The subscriber requested additional time to safely revoke the certificates." and were part of a financial entity. Entrust can't have it both ways and needs to address their conflicting statements to date. If Entrust does not feel these are conflicting please articulate why without referencing other answers?

With all due respect, your questions about the Prohibited Certificate Usage terms are asking for an interpretation of contractual terms. This is the purview of lawyers, and we believe is not appropriate for this forum.

If there is no single definition, then what did Entrust 'formulate' in its own definition, which you are claiming does not exist?

Please see our response above regarding the definition.

Entrust are explicitly confirming that no subscribers were using their certificates "in conjunction with any application that could lead to death, personal injury or severe physical or property damage"?

To our knowledge, no subscriber is using a certificate “in or in conjunction with any application in which failure could lead to death, personal injury or severe physical or property damage”. We do not agree with the suggestion that you seem to be making that this language prohibits the use of certificates in critical infrastructure. Further, you seem to be making an argument based on legal interpretation, which we do not believe is appropriate for this forum.

Following that question, and given these are self-reported reasons provided by subscribers:

• What rationale was there in delaying revocation due to 'harm' and 'critical infrastructure'?

The rationales that we have provided in previous responses.

• Can Entrust document a single instance of them pushing back against these self-reported claims?

In Comment 36 we gave examples where requests for extensions were denied. We do not have any examples where we disputed claims made by subscribers. We believe that our subscribers were acting in good faith.

• Subscribers state they are in the above categories, does Entrust agree?

Sorry, what categories? We do not understand this question.

• We have a list of subscribers and these statements so please provide some examples

Please explain what examples you are asking for.

All certificates in this bug are now revoked. We will be formalizing how we respond to any future requests for delayed revocation by subscribers in an updated incident response handling standard.

My question was not answered. I will reiterate it as Entrust seem to have a problem with answering questions lately:

• Why are Entrust still providing "limited exceptions" 2 months into a delayed revocation failure?

We believe this is essentially the same as the question asked in Comment 70, and in response to that question we replied that the initial proposed targets for revocation were established based on a good faith discussion about when the certificates could safely be replaced to avoid harm or minimize harm to the subscribers and the relying parties who depend on the servers.

In some cases the work required to enable safe replacement took longer than expected. We do not have specific granular reasons for why this happened, beyond the reasons why delays were required in the first place.

We have acknowledged that there were deficiencies and gaps in how we granted delayed revocation, which may have led to inconsistencies, and we have provided our commitments for how we propose to improve

I am not asking what you are planning to put into policy going forward, but what happened historically in regards to this incident. I was hoping this would also be in your formal report, but I can see nothing like that.

All affected certificates were revoked by June 6, 2024.

Again my question has not been answered:

• Are Entrust capable of giving any reason for these delays?

See above.

Entrust please actually read the questions posed. If this is a refusal to ask the questions then please say so against each question so it is clear what your intentions are.

We have read every single question posed and have made good faith efforts to answer.

Comment 85

[ngook.kong]

(In reply to Tim Callan from comment #72)

(In reply to ngook.kong from comment #60)

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation.
. . .
What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays …

So, to put it plainly, you absolutely expect to continue to delay mandatory revocation in the future and are telling us so right now.

Question 1: Yes or no, is the above statement correct? Please start your response with either the word yes or the word no. You may add other words after, but please avoid the Entrust habit of giving vague evasive responses that dodge the question.

No, your statement is not a fair reflection of what we have said. Your statement implies that we will continue to allow delayed revocations the same way that we have in the past, but that is contrary to what we have said.

Question 2: If no, state in terms as plain as I did in the paragraph above the commitment you are making, whatever that may be, to the community WRT delayed revocation.

Entrust will plan to meet the revocation requirements of the baseline requirements without delays, but may also permit delayed revocation per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation. If we do allow a delayed revocation, we are committed to doing so in line with Mozilla’s expectations as expressed in the wiki. In order to meet these expectations, understanding that we have not done so in the past, we are committed to the action items in bug 1901270.

Comment 86

[Wayne]

(In reply to ngook.kong from comment #84)

We disagree with the assertion or presumption made that any portions of Entrust’s Terms of Use are inherently flexible or not enforced as a rule. We do not believe that a discussion about contract enforcement is appropriate as corporations are not generally free to disclose legal strategy in a public forum.

Okay, we will move on. Do note I was passing on a question from someone else.

The decision was made per subscriber.

Thank you for answering.

We recognize that you do not like the answers provided, but we feel that we have answered. We have acknowledged that there were deficiencies and gaps in how we granted delayed revocation, which may have led to inconsistencies, and we have provided our commitments for how we propose to improve. We will seek appropriate legal and other expert advice regarding the interpretation of the industry and contractual requirements, and we will act in accordance with their expert advice. We accept responsibility for our actions. We are sorry. We will do better.

Noted.

We can see that the first quote is from Comment 67, but we are not able to locate the other statements you are referencing, can you please provide the Comment number and the direct quote? Many of these responses must be read in the context of the question they are answering. However, if we did indeed state that we did not make any “unilateral” decisions about revocation, then we agree that this seems incorrect. We may have intended to say that our decisions were per-subscriber and contextual (our own assessment and understanding of the subscriber and context), even when we didn’t have a response from that subscriber.

Absolutely, see: #188705 Comment 33

Entrust does not want to make unilateral decisions to choose which subscribers get an extension and which subscribers do not. We are very much asking for assistance to seek a solution that will successfully address your concerns.

We do not have our own written definition. We have referenced the definitions in CISA and the EU CER and NIS-2 as the ones that we consulted/used. We have stated that we believe these provide helpful guidance, but we are not claiming that these are authoritative or that every delayed revocation decision included a rigorous legal determination of whether a particular definition was met. As far as we know, the Mozilla Policy does not include a written definition of this term, nor does it provide guidance or restrictions on what definition could be used by CAs, so it is our understanding that strict adherence to any such definition is not a requirement.

Okay, it was unclear if Entrust were referencing other standards as a broad picture or if they used them. The answer provided clears up the situation.

Neither Entrust nor any subscriber has ever stated that “convenience” determines when revocation occurs. We have consistently explained that decisions are made on the basis of an assessment of the balance of harms and logistical challenges. The fact of the matter is, for most subscribers and relying parties, the value of the WebPKI is to facilitate safe and smooth transactions on the internet. The issues with the affected certificates pose no risk to the safety and smoothness of those transactions, but the revocation of those certificates without replacement would be extremely disruptive. This is not a matter of placing blame, it is a matter of dealing with realities. If the strict enforcement of rules begins to take priority over the facilitation of safe and smooth internet transactions, it brings discredit on the entire ecosystem. Entrust has been trying to avoid that, by showing a nuanced understanding of the complex issues faced by subscribers. We acknowledge that this view is in conflict with the desire of this community, hence we are taking actions to correct as detailed in the MDSP report.

There absolutely is room for nuance, the mention of strict enforcement of rules does rather ignore that subscribers and Entrust put themselves in such a position to begin with. It is not that a one-off delayed revocation occurs that is a problem, it is the basis of this being considered normal operating procedures for any future issues.

Please note we have not refused to answer questions. It is our position that these questions have been answered. We regret that you are not satisfied with the answers. We are not making any tacit statement that things will not change, we are making express statements that we will change our processes and our behavior to bring them in line with the rules and expectations expressed in the root embedding programs.

My point on refusal is more that the questions posed is far more nuanced and detailed than the answer given. However let us move on.

With all due respect, your questions about the Prohibited Certificate Usage terms are asking for an interpretation of contractual terms. This is the purview of lawyers, and we believe is not appropriate for this forum.

Entrust are entitled to do so, however I will note that this is the first time such a concern has been raised.

Please see our response above regarding the definition.

This has been clarified thank you.

To our knowledge, no subscriber is using a certificate “in or in conjunction with any application in which failure could lead to death, personal injury or severe physical or property damage”. We do not agree with the suggestion that you seem to be making that this language prohibits the use of certificates in critical infrastructure. Further, you seem to be making an argument based on legal interpretation, which we do not believe is appropriate for this forum.

I am quite literally using the precise phrasing from your terms, and on a subject you previously stated in that comment you would not answer. I believe I have made the points about issues with certificate usage in critical infrastructure, and Entrust's awareness of such usage clear at this point so I will move on.

The rationales that we have provided in previous responses.

I do not see a specific case outlined about harm. Subscribers gave very broad rationale hinting at it but that would go against your own terms so this contradiction seems to not be reconcilable. The part I find interesting is literally every subscriber in the breakdown list made the claim of harm, which should not be possible.

In Comment 36 we gave examples where requests for extensions were denied. We do not have any examples where we disputed claims made by subscribers. We believe that our subscribers were acting in good faith.

Okay, then we can agree that Entrust took all reasons in good faith and no checks were performed?

Sorry, what categories? We do not understand this question.

Specifically 'Harm' and 'Critical infrastructure', to refresh Entrust's memory please see attachment #9403115: EV Delayed Revocation 2024.05.21.xlsx

Please explain what examples you are asking for.

My point is that of the 33 subscribers listed in that revocation breakdown governing 8580 certificates quite literally all of them claimed 'Harm' and 'Critical Infrastructure'. I am not asking for you to identify a subscriber there, given the scope it should not be impossible to find a clear answer of one of these subscriber's rationale that makes it clear what would go wrong if Entrust revoked.

We believe this is essentially the same as the question asked in Comment 70, and in response to that question we replied that the initial proposed targets for revocation were established based on a good faith discussion about when the certificates could safely be replaced to avoid harm or minimize harm to the subscribers and the relying parties who depend on the servers.

In some cases the work required to enable safe replacement took longer than expected. We do not have specific granular reasons for why this happened, beyond the reasons why delays were required in the first place.

We have acknowledged that there were deficiencies and gaps in how we granted delayed revocation, which may have led to inconsistencies, and we have provided our commitments for how we propose to improve

Okay, we will discuss those more on MDSP.

I am not asking what you are planning to put into policy going forward, but what happened historically in regards to this incident. I was hoping this would also be in your formal report, but I can see nothing like that.

All affected certificates were revoked by June 6, 2024.
Again my question has not been answered:

• Are Entrust capable of giving any reason for these delays?
  See above.
  Entrust please actually read the questions posed. If this is a refusal to ask the questions then please say so against each question so it is clear what your intentions are
  We have read every single question posed and have made good faith efforts to answer.

To repeat the question that somehow keeps getting missed: it took until June 6th for all certificates to be revoked. Can we please just have some explanation as to what caused these additional delays?

(In reply to ngook.kong from comment #85)

So, to put it plainly, you absolutely expect to continue to delay mandatory revocation in the future and are telling us so right now.

Question 1: Yes or no, is the above statement correct? Please start your response with either the word yes or the word no. You may add other words after, but please avoid the Entrust habit of giving vague evasive responses that dodge the question.

No, your statement is not a fair reflection of what we have said. Your statement implies that we will continue to allow delayed revocations the same way that we have in the past, but that is contrary to what we have said.

I do not think the left hand knows what the right hand is doing at this point in time. To clarify that remark let us look at a different comment made today by Entrust:

Will Entrust continue to issue public web PKI certificates to subscribers who they know to be unable to tolerate an appropriately-prompt revocation without catastrophic consequences to society?
Yes, we will continue to issue public web PKI certificates to subscribers who meet the requirements to have publicly trusted certificates issued to them under the applicable rules. We do not believe that the CA/B Forum requirements, or any root embedding program requirements, prohibit issuance to an applicant on the grounds you describe. If this policy was to change we commit to adhering to it.

I do not see a intellectually honest interpretation of the above Yes comment that does not unequivocally conclude that delayed revocations are going to continue for certain subscribers. Far beyond that, it is Entrust's own stated opinion that issuance to subscribers where revocation would result in catastrophic consequences to society will continue until specific requirements appear.

If the compliance team at Entrust does not see the err involved in this rationale then I have some major ethics questions to raise.

Also for clarity is there a general inbox to issue requests for more info towards? I'm never quite sure who to attach to a specific incident. Does the flag help your system for figuring out which comments need answers?

Comment 87

[Zacharias]

I opened the attached .xlsx file in this incident and then navigated to the very first certificate listed. It subject is: dev.dpoi.api.td.com a domain that doesn't even seem to have a public DNS entry. Which isn't that suprising given the dev part of the domain name. The listed reason for this certificate is: "Revoking the affected certificates within the prescribed deadline may cause significant harm; the certificates are used in critical infrastructure and cannot be safely replaced prior to the revocation deadline. The Subscriber needs additional time to coordinate replacement with multiple teams, external companies and with change management process/windows."

The third one listed is for: isaetest3-si.delta.com another domain that doesn't appear to have a public DNS entry and from my searches in crt.sh and censys.io hasn't been issued a new certificate.

The seventh certificate is for: preprod.elb.api2.silver-fuel.com, for me there is no public DNS entry. The reason listed is: "
Revoking the affected certificates within the prescribed deadline may cause significant harm; the certificates are used in critical infrastructure and cannot be safely replaced prior to the revocation deadline. The Subscriber needs additional time to coordinate replacement with multiple teams, external companies and with change management process/windows."
The eigth is appears to be another Delta dev environment: rta.arpt-mgmt-dev.aws.delta.com.

The fifteenth is for: blackduck.aa.com, this is hosting software described as: "Black Duck® software composition analysis (SCA) helps teams manage the security, quality, and license compliance risks that come from the use of open source and third-party code in applications and containers."

After the first twenty certificates I did some random sampling and found: webt-uat-wf1.jpmchase.com, another domain I cannot navigate to.

In the attachment for this incident Subscriber 8 (Delta Airlines) has 1684 certificates by my count, and part of the reason for all of them is: "Subscriber advises that it uses a manual process of installation for replacing the affected certificates, which takes more time. ". Quite frankly, that is insane. I sincerely hope that they are lying because the alternative is terrifying, if Entrust made a mistake when translating their stated reasons to their more general version would also feel safer.

Using Censys to try and get an overview Delta has 4849 certificates issued by Entrust.

1. Are dev environment critical infrastructure?
2. Are test environments critical infrastructure?
3. Are preprod environments critical infrastructure?
4. Are uat environments critical infrastructure?
5. Did Entrust evaluate the veracity or even plausibility of their subscribers claimed reasons for why revocation could not be timely?
6. If the answer to question 5 is Yes, how was this done?
7. If the answer to question 5 is No, why was this not done?
8. When a subscriber stated their reasons for not being able to handle a timely revocation, was that reason per certificate even if the subscriber had hundreds of affected certificates?
9. Even if not required to by any rules or regulations, does a CA have any responsibility to ensure some level of certificate agility if they issue thousands of certificates to a customer?
10. Is a "software composition analysis" critical infrastructure?
11. In the future, will Entrust verify the claimed reasons for why delayed revocation is necessary? If so, how?
12. Can you motivate why the following reasons are good reasons to delay revocation?
    - Change management process
    - Change window or freeze

Comment 88

[Mike Shaver (:shaver emeritus)]

(In reply to ngook.kong from comment #78)

If you are referring to the action items that were given in response to Bug 1651481, none of those action items included a commitment to creating a new internal policy. We had existing internal policies and procedures and in making the commitments in 2020 we assumed that these existing policies and procedures would be sufficient to support our commitments. We know we were mistaken in this, which is why in our June 7th report one of the specific items in our improvement plan is to adopt a new revocation policy. These and other action items are being tracked in Bug 1901270.

I'm sorry, this is a very challenging position to understand. You say that the 2020 action items from bug 1651481 didn't include a commitment to creating new internal policy. In the bug, Bruce makes this commitment:

To remediate certificate revocation issues:
• We will follow the guidance provided by Mozilla, https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation for miss-issued certificates.
• We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

(There is no explicit listing of "action items" that I can see, so I assume that you mean the remediation plan description.)

What is that if not describing new policy? If it was the existing policy, then the certificates in '481 would have been revoked promptly. They were not, because Entrust's policy at the time was to permit universal delayed revocation. Maybe that previous policy (incredibly) wasn't written down, but it's no less a policy commitment because it doesn't use the specific word "policy". I don't want to start quoting dictionaries, but between this and the denial that redaction is concealment, I'm worried that Entrust is not using important words in the commonly expected ways. This will make it much harder to evaluate the commitments Entrust makes in its MDSP reports, or elsewhere, since those tend to be expressed in just a few words each time unfortunately.

Please elaborate to help the community understand specifically what was done in the wake of the 2020/'481 commitments, and what will be done differently with new commitments such that the latter should be considered credible.

Comment 89

[Mike Shaver (:shaver emeritus)]

(In reply to ngook.kong from comment #85)

(In reply to Tim Callan from comment #72)

(In reply to ngook.kong from comment #60)

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation.
. . .
What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays …

So, to put it plainly, you absolutely expect to continue to delay mandatory revocation in the future and are telling us so right now.

Question 1: Yes or no, is the above statement correct? Please start your response with either the word yes or the word no. You may add other words after, but please avoid the Entrust habit of giving vague evasive responses that dodge the question.

No, your statement is not a fair reflection of what we have said. Your statement implies that we will continue to allow delayed revocations the same way that we have in the past, but that is contrary to what we have said.

You are inserting implications in Tim's question that are not present, in my opinion.

I will ask a narrower question: will Entrust knowingly take on new subscribers for whom delayed revocation would be required in the case of a misissuance or security incident? put differently, will Entrust allow subscribers use web PKI certificates in circumstances known to be incompatible with web PKI revocation requirements?

Comment 90

[ngook.kong]

(In reply to walter.j.marks from comment #80)

After re-reading your post Ngook, I also had something else I wanted to clarify:

Question: To put it on the record: were any operational changes made in regards to the action items given by Entrust in Bug #1651481? Since it seems the community needs to include it, please answer beginning with a yes or no.

No. We believe your question about “operational changes” is specifically talking about processes, and not the actions we’ve already described around encouraging the adoption of automation, etc. On that basis, No, we did not make the operational changes that we should have—we had existing procedures (operations) and in making the commitments in 2020 we assumed that these existing procedures (operations) would be sufficient to support our commitments. We now know we were mistaken in this. To remediate these deficiencies our June 7th report (as updated on June 21st) included several specific “process” and “technology” items in our improvement plan to operationalize our commitments, including creation of a formal incident handling process, as well as support in our backend systems to tag certificates affected by incidents and trigger an immediate indication that affected certificates will be revoked within existing requirements. These and other action items are being tracked in Bug 1901270.

Comment 91

[ngook.kong]

(In reply to Wayne from comment #83)

Will Entrust continue to issue public web PKI certificates to subscribers who they know to be unable to tolerate an appropriately-prompt revocation without catastrophic consequences to society?

Yes, we will continue to issue public web PKI certificates to subscribers who meet the requirements to have publicly trusted certificates issued to them under the applicable rules. We do not believe that the CA/B Forum requirements, or any root embedding program requirements, prohibit issuance to an applicant on the grounds you describe. If this policy was to change we commit to adhering to it.

I will note that previously Entrust stated the following:

For instance, airline ticketing and check-in systems wouldn't be classified as a prohibited use case. While an outage caused by certificate revocation could lead to serious negative impacts for air travelers, it wouldn't directly risk human life. Irresponsible revocation without considering potential consequences could lead to widespread disruption, impacting millions of travelers and causing significant economic harm. This highlights the importance of our collaborative approach with subscribers to ensure a smooth re-issuance process and minimize disruption.

Consider that while not a CA/B F requirement that the scenario creating irresponsible revocation is only possible due to irresponsible issuance. If there is truly a separate compliance team now, then is there a single situation where they are able to overrule other departments in this regard? If so, please provide an example that would previously have been issued without the new compliance team active.

Yes the new compliance team will be fully empowered to overrule other departments if the situation did not meet the rules established by CA/B F. As an example scenario, the new compliance leadership would not allow declaration of mis-issuance but argue that revocation is not needed, they would enforce the rules as written for revocation on mis issuance and use of appropriate CA/B process to discuss any disagreements and propose changes if needed.

Please clarify, are you suggesting that every subscriber who is known to be unable to replace a certificate promptly should be blacklisted by all CAs, or that no CA should be allowed to accept a certificate application from that organization? Would not such a rule need to be clearly articulated in the CA/B Forum requirements and root embedding program requirements so as to apply to all CAs equally before being implemented? For the record, we would have major concerns with such a rule and would not likely be supportive of such a ballot.

The CA/B Forum requirements do not state anything about selling to sanctioned countries. We can agree there are more rules governing issuance than those requirements, correct?

Yes, we agree that CAs must comply with other rules outside of the CA/B Forum requirements. For example, the EV guidelines in section 4.1 prohibits the issuance of certificates to any entity listed on any government denial list or prohibited list (e.g., trade embargo) under the laws of the CA’s jurisdiction.

You write of "pushing customers", but again not what exactly that entails or how the community should observe Entrust's progress on it without waiting for another misissuance event with three month revocation timelines and knowingly and deliberately insufficient description of the reasons for the delays. Is Entrust asking nicely? Requiring subscribers to also approve and prepare an alternative certificate from another CA that they can quickly switch to if their Entrust certificate needs to be revoked?

No. We agree that our process for collecting the reasons for the delays was insufficient, but it was not knowingly and deliberately insufficient. In our June 7th report, we outlined several specific action items to improve how we handle revocation events. One of these is to adopt a new revocation process and ensure that it is communicated to subscribers. We are considering ways to increase visibility of the CA’s right to revoke certificates on short notice beyond our contract language. We also will add a warning to manual order pages and related emails to ensure subscriber understanding of required timelines. These and other action items are being tracked in Bug 1901270.

We do generally recommend that subscribers have a backup CA, but installing a backup certificate in every application would increase cost, effort and complexity. We do not believe this should be an industry requirement.

As previously stated please consider what responsible issuance truly means and how Entrust can be considered stewards of public trust if resiliency is considered a problem. If you are truly stewarding subscribers with critical infrastructure then they will require backup plans and alternative contractors in place to be used at a moment's notice. This would already be part of any contract that asserts usage in that field, and is part of why I find these common assertions across CA's with delayed revocation issues so questionable.

We have considered your view and believe it is a good idea. We also believe that in order to ensure compliance with such a requirement as foundational element of public trust, it should be added explicitly as a rule that applies contractually such as 24/120 h timer.

Comment 92

[ngook.kong]

(In reply to Wayne from comment #86)

(In reply to ngook.kong from comment #84)

We can see that the first quote is from Comment 67, but we are not able to locate the other statements you are referencing, can you please provide the Comment number and the direct quote? Many of these responses must be read in the context of the question they are answering. However, if we did indeed state that we did not make any “unilateral” decisions about revocation, then we agree that this seems incorrect. We may have intended to say that our decisions were per-subscriber and contextual (our own assessment and understanding of the subscriber and context), even when we didn’t have a response from that subscriber.

Absolutely, see: #188705 Comment 33

Entrust does not want to make unilateral decisions to choose which subscribers get an extension and which subscribers do not. We are very much asking for assistance to seek a solution that will successfully address your concerns.

Thank you. In Comment 71, you were asking how these two statements are compatible:

• (From Comment 33) “Entrust does not want to make unilateral decisions to choose which subscribers get an extension and which subscribers do not. We are very much asking for assistance to seek a solution that will successfully address your concerns”.
• (From Comment 67) “in this case, with subscribers who had not responded, we decided to prioritize avoiding unintended harmful consequences."

We see these two statements as compatible, or at least not contradictory, because the statement in Comment 33 is about Entrust does not want to do, and a request for collaboration on a solution. Comment 67 is a statement of what we actually did in the absence of such a solution, and when placed in the difficult situation of needing to make a decision without having (admittedly due in part to issues with our process) sufficient information to properly assess risk. We acknowledge that this is not acceptable and should not have been done as it created a significant issue since all revocation was not done within the required period.

To our knowledge, no subscriber is using a certificate “in or in conjunction with any application in which failure could lead to death, personal injury or severe physical or property damage”. We do not agree with the suggestion that you seem to be making that this language prohibits the use of certificates in critical infrastructure. Further, you seem to be making an argument based on legal interpretation, which we do not believe is appropriate for this forum.

I am quite literally using the precise phrasing from your terms, and on a subject you previously stated in that comment you would not answer. I believe I have made the points about issues with certificate usage in critical infrastructure, and Entrust's awareness of such usage clear at this point so I will move on.

The rationales that we have provided in previous responses.

I do not see a specific case outlined about harm. Subscribers gave very broad rationale hinting at it but that would go against your own terms so this contradiction seems to not be reconcilable. The part I find interesting is literally every subscriber in the breakdown list made the claim of harm, which should not be possible.

In Comment 36 we gave examples where requests for extensions were denied. We do not have any examples where we disputed claims made by subscribers. We believe that our subscribers were acting in good faith.

Okay, then we can agree that Entrust took all reasons in good faith and no checks were performed?

Correct.

Sorry, what categories? We do not understand this question.

Specifically 'Harm' and 'Critical infrastructure', to refresh Entrust's memory please see attachment #9403115: EV Delayed Revocation 2024.05.21.xlsx

Please explain what examples you are asking for.

My point is that of the 33 subscribers listed in that revocation breakdown governing 8580 certificates quite literally all of them claimed 'Harm' and 'Critical Infrastructure'. I am not asking for you to identify a subscriber there, given the scope it should not be impossible to find a clear answer of one of these subscriber's rationale that makes it clear what would go wrong if Entrust revoked.

We believe this is essentially the same as the question asked in Comment 70, and in response to that question we replied that the initial proposed targets for revocation were established based on a good faith discussion about when the certificates could safely be replaced to avoid harm or minimize harm to the subscribers and the relying parties who depend on the servers.

In some cases the work required to enable safe replacement took longer than expected. We do not have specific granular reasons for why this happened, beyond the reasons why delays were required in the first place.

We have acknowledged that there were deficiencies and gaps in how we granted delayed revocation, which may have led to inconsistencies, and we have provided our commitments for how we propose to improve

Okay, we will discuss those more on MDSP.

I am not asking what you are planning to put into policy going forward, but what happened historically in regards to this incident. I was hoping this would also be in your formal report, but I can see nothing like that.

All affected certificates were revoked by June 6, 2024.
Again my question has not been answered:

• Are Entrust capable of giving any reason for these delays?
  See above.
  Entrust please actually read the questions posed. If this is a refusal to ask the questions then please say so against each question so it is clear what your intentions are
  We have read every single question posed and have made good faith efforts to answer.

To repeat the question that somehow keeps getting missed: it took until June 6th for all certificates to be revoked. Can we please just have some explanation as to what caused these additional delays?

There were 9 certificates that were revoked the first week of June. The reason for the delay was due to the subscribers needing more time to coordinate replacement with multiple teams and/or third parties.

(In reply to ngook.kong from comment #85)

So, to put it plainly, you absolutely expect to continue to delay mandatory revocation in the future and are telling us so right now.

Question 1: Yes or no, is the above statement correct? Please start your response with either the word yes or the word no. You may add other words after, but please avoid the Entrust habit of giving vague evasive responses that dodge the question.

No, your statement is not a fair reflection of what we have said. Your statement implies that we will continue to allow delayed revocations the same way that we have in the past, but that is contrary to what we have said.

I do not think the left hand knows what the right hand is doing at this point in time. To clarify that remark let us look at a different comment made today by Entrust:

Will Entrust continue to issue public web PKI certificates to subscribers who they know to be unable to tolerate an appropriately-prompt revocation without catastrophic consequences to society?
Yes, we will continue to issue public web PKI certificates to subscribers who meet the requirements to have publicly trusted certificates issued to them under the applicable rules. We do not believe that the CA/B Forum requirements, or any root embedding program requirements, prohibit issuance to an applicant on the grounds you describe. If this policy was to change we commit to adhering to it.

I do not see a intellectually honest interpretation of the above Yes comment that does not unequivocally conclude that delayed revocations are going to continue for certain subscribers. Far beyond that, it is Entrust's own stated opinion that issuance to subscribers where revocation would result in catastrophic consequences to society will continue until specific requirements appear.

If the compliance team at Entrust does not see the err involved in this rationale then I have some major ethics questions to raise.

Also for clarity is there a general inbox to issue requests for more info towards? I'm never quite sure who to attach to a specific incident. Does the flag help your system for figuring out which comments need answers?

We are tracking all active Bugzilla comments and appreciate keeping the questions in Bugzilla comments. Regarding your request for a general inbox to contact, please email [e-mail]. Yes, flagging the questions on a longer thread helps identify the questions that need answers. Thank you.

Comment 93

[ngook.kong]

(In reply to Zacharias from comment #87)

I opened the attached .xlsx file in this incident and then navigated to the very first certificate listed. It subject is: dev.dpoi.api.td.com a domain that doesn't even seem to have a public DNS entry. Which isn't that suprising given the dev part of the domain name. The listed reason for this certificate is: "Revoking the affected certificates within the prescribed deadline may cause significant harm; the certificates are used in critical infrastructure and cannot be safely replaced prior to the revocation deadline. The Subscriber needs additional time to coordinate replacement with multiple teams, external companies and with change management process/windows."

The third one listed is for: isaetest3-si.delta.com another domain that doesn't appear to have a public DNS entry and from my searches in crt.sh and censys.io hasn't been issued a new certificate.

The seventh certificate is for: preprod.elb.api2.silver-fuel.com, for me there is no public DNS entry. The reason listed is: "
Revoking the affected certificates within the prescribed deadline may cause significant harm; the certificates are used in critical infrastructure and cannot be safely replaced prior to the revocation deadline. The Subscriber needs additional time to coordinate replacement with multiple teams, external companies and with change management process/windows."
The eigth is appears to be another Delta dev environment: rta.arpt-mgmt-dev.aws.delta.com.

The fifteenth is for: blackduck.aa.com, this is hosting software described as: "Black Duck® software composition analysis (SCA) helps teams manage the security, quality, and license compliance risks that come from the use of open source and third-party code in applications and containers."

After the first twenty certificates I did some random sampling and found: webt-uat-wf1.jpmchase.com, another domain I cannot navigate to.

In the attachment for this incident Subscriber 8 (Delta Airlines) has 1684 certificates by my count, and part of the reason for all of them is: "Subscriber advises that it uses a manual process of installation for replacing the affected certificates, which takes more time. ". Quite frankly, that is insane. I sincerely hope that they are lying because the alternative is terrifying, if Entrust made a mistake when translating their stated reasons to their more general version would also feel safer.

Using Censys to try and get an overview Delta has 4849 certificates issued by Entrust.

1. Are dev environment critical infrastructure?

No.

2. Are test environments critical infrastructure?

No.

3. Are preprod environments critical infrastructure?

No.

4. Are uat environments critical infrastructure?

No.

5. Did Entrust evaluate the veracity or even plausibility of their subscribers claimed reasons for why revocation could not be timely?

No, we relied on our subscribers to provide truthful information.

6. If the answer to question 5 is Yes, how was this done?

N/A

7. If the answer to question 5 is No, why was this not done?

We relied on our subscribers to provide truthful information. We will push back harder on our customers moving forward. In fact, we are piloting a new incident handling procedure as part of the revocation underway for Bug 1890685 and have been doing just this in response to subscriber requests for delayed revocation.

8. When a subscriber stated their reasons for not being able to handle a timely revocation, was that reason per certificate even if the subscriber had hundreds of affected certificates?

It was not. That is now part of the current incident handling procedure we are following for revocation related to Bug 1890685.

9. Even if not required to by any rules or regulations, does a CA have any responsibility to ensure some level of certificate agility if they issue thousands of certificates to a customer?

Yes we believe CAs should promote certificate agility, We would appreciate the community’s thoughts on this being a best practice vs rule. And how to codify the rule to ensure consistency across all CAs.

10. Is a "software composition analysis" critical infrastructure?

No.

11. In the future, will Entrust verify the claimed reasons for why delayed revocation is necessary? If so, how?

Yes. As we revoke for Bug 1890685, we are not only asking subscribers who proactively approach us to request delays to provide a detailed justification for the delay, but also vetting that information through a cross-functional team. That team has been responding with additional questions for the subscriber. No delays have been approved.

12. Can you motivate why the following reasons are good reasons to delay revocation?
    - Change management process
    - Change window or freeze

As we revoke for Bug 1890685, we are denying those reasons for delayed revocation. We agree that without more information those reasons are not sufficient to approve a delayed revocation.

Comment 94

[ngook.kong]

(In reply to Mike Shaver (:shaver emeritus) from comment #88)

(In reply to ngook.kong from comment #78)

If you are referring to the action items that were given in response to Bug 1651481, none of those action items included a commitment to creating a new internal policy. We had existing internal policies and procedures and in making the commitments in 2020 we assumed that these existing policies and procedures would be sufficient to support our commitments. We know we were mistaken in this, which is why in our June 7th report one of the specific items in our improvement plan is to adopt a new revocation policy. These and other action items are being tracked in Bug 1901270.

I'm sorry, this is a very challenging position to understand. You say that the 2020 action items from bug 1651481 didn't include a commitment to creating new internal policy. In the bug, Bruce makes this commitment:

To remediate certificate revocation issues:
• We will follow the guidance provided by Mozilla, https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation for miss-issued certificates.
• We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

(There is no explicit listing of "action items" that I can see, so I assume that you mean the remediation plan description.)

What is that if not describing new policy? If it was the existing policy, then the certificates in '481 would have been revoked promptly. They were not, because Entrust's policy at the time was to permit universal delayed revocation. Maybe that previous policy (incredibly) wasn't written down, but it's no less a policy commitment because it doesn't use the specific word "policy". I don't want to start quoting dictionaries, but between this and the denial that redaction is concealment, I'm worried that Entrust is not using important words in the commonly expected ways. This will make it much harder to evaluate the commitments Entrust makes in its MDSP reports, or elsewhere, since those tend to be expressed in just a few words each time unfortunately.

Please elaborate to help the community understand specifically what was done in the wake of the 2020/'481 commitments, and what will be done differently with new commitments such that the latter should be considered credible.

We did not thoroughly follow through on the actions from 2020. That has been acknowledged multiple times across the open bugs. Policies and guidance from Mozilla were not documented into an internal policy but were referenced. Educating subscribers on the need to become more agile and encouraging automation were completed actions but were not as successful as anticipated. In absence of a documented policy on revocations, Leadership was operating with incomplete information when the decision was made to entertain delayed revocation for subscribers.

Knowledge of the BR requirements has been expanded beyond a few colleagues through the reorganization of the certificate solutions compliance team, creation of a certification compliance committee, formal documentation of policies and procedures and other actions that have been included in the Mozilla report action plans. These requirements are now understood by anyone who plays a role in overseeing the handling of a potential incident in the future, including one that results in a mis-issuance warranting revocation.

Comment 95

[ngook.kong]

(In reply to Mike Shaver (:shaver emeritus) from comment #89)

(In reply to ngook.kong from comment #85)

(In reply to Tim Callan from comment #72)

(In reply to ngook.kong from comment #60)

It is our intention to follow the BR timelines for revocation. That said, we cannot promise we will never have a situation where there are exceptional circumstances that warrant approval of a delayed revocation.
. . .
What we can and will commit to is further restricting the approval of delayed revocation requests, reducing the length of time granted for delays, more transparency and detail in our reporting around approved delays …

So, to put it plainly, you absolutely expect to continue to delay mandatory revocation in the future and are telling us so right now.

Question 1: Yes or no, is the above statement correct? Please start your response with either the word yes or the word no. You may add other words after, but please avoid the Entrust habit of giving vague evasive responses that dodge the question.

No, your statement is not a fair reflection of what we have said. Your statement implies that we will continue to allow delayed revocations the same way that we have in the past, but that is contrary to what we have said.

You are inserting implications in Tim's question that are not present, in my opinion.

I will ask a narrower question: will Entrust knowingly take on new subscribers for whom delayed revocation would be required in the case of a misissuance or security incident? put differently, will Entrust allow subscribers use web PKI certificates in circumstances known to be incompatible with web PKI revocation requirements?

No, we will not knowingly take on subscribers who are in this position. We are also making greater efforts moving forward to ensure subscribers understand the revocation requirements and can adhere to them.

Comment 96

[Mike Shaver (:shaver emeritus)]

(In reply to ngook.kong from comment #95)

(In reply to Mike Shaver (:shaver emeritus) from comment #89)

I will ask a narrower question: will Entrust knowingly take on new subscribers for whom delayed revocation would be required in the case of a misissuance or security incident? put differently, will Entrust allow subscribers use web PKI certificates in circumstances known to be incompatible with web PKI revocation requirements?

No, we will not knowingly take on subscribers who are in this position. We are also making greater efforts moving forward to ensure subscribers understand the revocation requirements and can adhere to them.

Thank you. I think this is a very important commitment on Entrust’s part, and I’m quite glad to see it.

Comment 97

[Bruce Morton]

Action Items

Action Item	Kind	Due Date
Implement ACME ARI	Mitigate	2024-07-31
Work with customers to replace and revoked impacted certificates	Mitigate	Done
Implement formal incident response process including incident response communication plan to meet mandatory reporting times	Prevent	Done

Can we have the next update set to 2024-07-31? Thanks.

Comment 98

[Bruce Morton]

Action Items

Action Item	Kind	Due Date
Implement ACME ARI	Mitigate	Done
Work with customers to replace and revoked impacted certificates	Mitigate	Done
Implement formal incident response process including incident response communication plan to meet mandatory reporting times	Prevent	Done

All actions items have been completed. We request that this incident be closed. Thanks.

Comment 99

[Paul van Brouwershaven]

We have no further updates, and all action items have been completed.

We kindly suggest closing this incident.

Comment 100

[Paul van Brouwershaven]

Ben, could you kindly clarify whether we should continue updating this bug while you're evaluating if it can be closed?

Maybe you can set a next update for the meantime?

Comment 101

[Bruce Morton]

Hi Ben, we request this incident be closed. If not please change the next update to 2024-10-31.

Thanks, Bruce.

Comment 102

[Bruce Morton]

We will continue to monitor this bug. As all actions are closed and all comments have been addressed, please set the next update to 2025-03-31.

Comment 103

[Ben Wilson]

We continue work on incident-reporting and compliance requirements aimed at reducing delayed revocation, so this bug will remain open until at least February 1, 2025. Meanwhile, CAs should review https://github.com/mozilla/www.ccadb.org/pull/186.

Comment 104

[Ben Wilson]

Please provide a Closure Summary.

Comment 105

[Bruce Morton]

Incident Report Closure Summary

• Incident Description: Delayed revocation of EV TLS certificates with missing cPSuri.
• Incident Root Cause(s): Certificate revocation was delayed to minimize harm to the Subscribers and Relying Parties.
• Remediation Description: Our revocation position changed and it was decided to revoke all non-expired certificates.
• Commitment Summary: Entrust has sold its public certificate business to Sectigo and is in the process of developing a transition plan with Sectigo. In the interim, Entrust remains committed to compliance with the Baseline Requirements, including strict adherence to 24-hour and 5-day revocation requirements in the event of a mis-issuance.

Comment 106

[Zacharias]

(In reply to Bruce Morton from comment #105)

• Incident Root Cause(s): Certificate revocation was delayed to minimize harm to the Subscribers and Relying Parties.

Is this still the position of Entrust?

I think you have missed a couple of “Why?” when doing the five whys. The root lies deeper, and would address why Entrust took a position, and for a long time kept trying to defend that position, when that was completely opposed to what everyone in the webPKI community (that was willing to engage publicly) argued.

What even is this “harm” that is avoided, some of it is clearly hypothetical:

"in this case, with subscribers who had not responded, we decided to prioritize avoiding unintended harmful consequences"

When I read “Certificate revocation was delayed to minimize harm to the Subscribers and Relying Parties.”, Why? And that comes to mind is: Because Entrust prioritized _____ over following the BRs.

While it’s a bit hard to keep track of the different bugs, scanning though this one I can see discussions about the famous 2020 commitments. The root cause for this bug is of course connected to this as well. Let’s go again: “Certificate revocation was delayed to minimize harm to the Subscribers and Relying Parties.” Why?
Because Entrust did not honor its commitments made in 2020. Why?

Comment 107

[Ben Wilson]

We appreciate the recent questions and discussion, but we will be closing this bug on or about Friday, 21-Feb-2025, as the key issues have already been discussed at length. The outstanding questions appear to revisit points that have already been hashed over before. Given that Entrust is exiting the publicly-trusted PKI business, further debate here would not be productive. If there are other, specific, and unresolved concerns that materially impact the closure of this matter, please contact us directly at certificates_at_mozilla_dot_org. Thanks.