1658794 - Entrust: Late Revocation for Invalid State/Province Issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Dathan Demone]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0

Comment 1

[Dathan Demone]

This bug is related to ttps://bugzilla.mozilla.org/show_bug.cgi?id=1658792

We have been informed by one of our customers that they will not be able to replace all 395 certificates within the 5-day timeframe.

We will provide a summary of the certificates that were not revoked in 5 days, as required by Baseline Requirements. We will also create a timeline for when they will be revoked and provide clear reasons why the certificates could not be revoked within 5 days. We will post an update once we have confirmed which certificates were not revoked by the end of the 5 day period 15 August 2020 17:00 UTC

Comment 2

[Dathan Demone]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

After investigating and contacting certificate subscribers for bug https://bugzilla.mozilla.org/show_bug.cgi?id=1658792, we discovered on 11 August 2020 that a large, international organization would not be able to revoke and replace 395 certificates before the 5-day revocation deadline of 15 August 2020 17:00 UTC. At that time, a decision was made that we would allow the organization more time to replace their certificates and not forcibly revoke these certificates before the deadline, as this would have severe consequences on their global infrastructure.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

The full timeline of events leading up the late revocation can be viewed in the original bug - https://bugzilla.mozilla.org/show_bug.cgi?id=1658792

Here are the additional events that pertain to the specific list of certificates that will be revoked late and a timeline of what is planned to address the issue:

11 August 2020: As described in the original bug, a discussion is held with the organization that has 395 certificates to revoke and replace. During this conversation, they mention that they do not believe they will be able to revoke the certificates before the required deadline of 15 August 2020 17:00 UTC and that a forced revocation would cause a severe impact to their critical infrastructure. The reason why they cannot replace these certificates in time is because of the nature of their business and how there are many certificate owners distributed across international locations. They would need time to communicate this change to everyone. In addition, they must follow their change management and outage process, which requires them to schedule certificate replacement. Due to the nature of some of these systems, the change management process is very strict and cannot be bypassed. Entrust decides that we will not forcibly revoke all of these certificates and allow the organization additional time to properly plan certificate replacement. The organization expresses its view that they are taking this very seriously but just need more time to complete all of the changes.

13 Aug 2020: The organization commits to having as many certificates as possible replaced by 17 August 2020 but states that some will take longer due to the nature of the system and the associated change management processes. They will provide us with a further update during the week of 17 August to let us know when they will be able to replace the remaining certificates. A target of 28 August 2020 is set but that may change once they complete their internal communication.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Based on initial findings and subsequent verification checks, we have stopped issuing these problematic certificates. The last certificate for this specific late revocation issue was issued 17 June 2019.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

For this late revocation issue, there are potentially a total of 395 OV SSL certificates. The customer has agreed to try and revoke as many as possible before the deadline. We will run a fresh report after the deadline to provide the specific list of certificates that have been revoked.
The first certificate was issued on 7 August 2018 and the last certificate was issued on 17 June 2019.

5. The complete certificate data for the problematic certificates.
   A full report of certificates that were not revoked by the deadline will be posted after the original revocation deadline.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   See https://bugzilla.mozilla.org/show_bug.cgi?id=1658792

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Educating Customers on Best Practices to Handle Emergency Certificate Replacement Scenarios
Entrust deals with many large international organizations that issue certificates to many departments and sub organizations. Certificates are often managed by a smaller group that delivers certificates to many individuals within the organization. One of the major challenges in managing the regular certificate lifecycle within these larger organizations is tracking who ultimately owns the certificate, where the certificate is deployed, and the type of system or application that requires this certificate. While knowing this information does not solve every issue related to this late revocation problem, such as expediting change controls/outages process and having resources to generate new CSRs/keys and deploy new certificates, it can certainly help to cut down the amount of time to organize certificate replacement. Our certificate management system includes built-in tracking and reporting capabilities that can help customers find the right information they need during an emergency situations to help expedite the communication process. We need to make sure that our customers are leveraging this as much a possible in the event that there is a requirement to quickly replace certificates for any reason.

Emphasizing the Value in Automation
Certificate lifecycle automation is a valuable tool that can be used to speed up the certificate deployment/replacement process and eliminate the possibility of human error. However, there are many challenges to fully automating PKI within large organizations, including cost and environment complexity. As a CA, we have a responsibility in helping our customers achieve higher levels of automation. We do support many third party integrations and certificate lifecycle automation tools though our APIs, along with other automation protocols that our customers can leverage. While we cannot force our customers to use these tools, we can certainly do more when it comes to emphasizing the need for automation, and these events underscore how automation can help organizations to save resources and reduce their risk of running into issues when undergoing high volume certificate replacement.

Comment 3

[Dathan Demone]

Attached is a list of 384 certificates that were not revoked before the set deadline of 15 August 2020 17:00 UTC

We will follow up shortly with the plan to revoke the remaining certificates as quickly as possible based on feedback from our customer.

Comment 4

[Dathan Demone]

Attachment: Certificate List - Late Revocation.pdf

Comment 5

[Dathan Demone]

Based on multiple discussions with our customer and taking into consideration the negative impact unplanned revocation can have on their infrastructure, we have set a deadline of 5 September 2020 to revoke and replace the remaining certificates.

As of 18 August 19:00 UTC, 383 certificates have not yet been revoked. The customer has said that many certificates have been replaced and they are planning to submit revocation requests for the original certificates shortly.

We will continue to provide updates on how revocation is progressing ahead of the 5 September 2020 deadline.

Comment 6

[Dathan Demone]

There are currently 328 certificates that are pending revocation. We expect many more to be revoked in the near future, as many certificates have been replaced and are just pending installation or confirmation of successful installation. Our support team has been working closely with the customer to help them through the process and we have been communicating with the customer on a daily basis to make sure we can help them replace all of their certificates before the agreed-upon deadline. We will provide another progress report on 1 September 2020

Comment 7

[Dathan Demone]

We are down to 21 certificates that are pending revocation. These remaining certificates will be revoked on or before 5 September

Comment 8

[Dathan Demone]

All of the certificates for this incident have been revoked

Comment 9

[Mathew Hodson]

Whiteboard needs to be updated. Should be [delayed-revocation-leaf] with dashes.

Comment 10

[Ben Wilson]

It appears that the Incident Report and revocations are complete and therefore I intend to close this bug on or about 9-October-2020 unless there are further issues to discuss.