1804753 - Entrust: Delayed Revocation for EV TLS Certificate incorrect jurisdiction

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Bruce Morton]

Attachment: Delayed Revocation.xlsx

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Entrust has filed an incident for issuing EV TLS certificates with incorrect jurisdiction information, https://bugzilla.mozilla.org/show_bug.cgi?id=1802916. Since the incident was discovered in November, revocation for some certificates has been delayed due to blackout periods.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2022-11-28 - Subscribers were contacted and were advised of the incident, the miss-issued certificates, and the planned revocation date
• 2022-11-28 - Starting after the incident was reported, Subscribers provided feedback they required more time to coordinate reissuance within the annual blackout period which generally starts before US Thanksgiving and ends one or 2 weeks after New Year’s day.
• 2022-12-01 17:35 UTC - Compliance discussed U.A.E. issues with Verification
• 2022-12-02 19:15 UTC - Compliance discussed with Verification Subscriber feedback
• 2022-12-06 17:00 UTC - Compliance discussed Subscriber feedback with Support
• 2022-12-06 18:00 UTC - Compliance meeting to discuss revocation plans and status

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Based on the miss-issuance, all discovered jurisdiction issues have been corrected, so certificates with the jurisdiction issue are no longer being issued.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

See item 4.

5. The complete certificate data for the problematic certificates.

The certificates in the attached file were not revoked or expired by the 5-day revocation timeline.

See attached file.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The problem was detected in November and the incident was provided to the Subscribers on 28 November 2022. Entrust was pushed back by many Subscribers for the following reasons:
• Azimut Holding S.p.A. - Large volume of certificates and will need to engage multiple departments to coordinate this effort. Challenging time of year to pull teams together for the various levels of review/approval needed. Requested to complete revocation by 16 December 2023.
• Experian Information Solutions, Inc. - Currently in a code freeze and business protection period until January 7th. Will need to consult with various groups as 3 of the certs are in production and part of the Italy division. Requested to complete revocation by 15 January 2023.
• Fidelity Investments - Large volume of certificates and will need to engage multiple departments to coordinate this effort. Challenging time of year to pull teams together for the various levels of review/approval needed. Planning to complete revocation by end of January 2023.
• First Abu Dhabi Bank PJSC – The 18 certificates are owned by various application teams within FAB and needs to go through a change process with a planned maintenance along with the participation of multiple operations teams to replace the certificate. Being a Bank, the certificates are critical for the trust and functioning of the client/customer facing applications. It is nearing end of year, with holiday season and change freezes around, the approvals will be delayed. Planning to complete revocation by 6 February 2023.
• ING Groep N.V. - Needs extension for the remaining certificates. Planning to complete revocation by 9 December 2022.
• Munich Re Group - EV certificates are issued for critical applications. They are struggling to sort out a clear plan for 1-2 of them. Planning to complete revocation by 16 December 2022.
• Prudential Financial, Inc - Impact is critical to business units and these certificates are vital. Need more time to solidify a plan with international teams and obtain various levels approvals. Planning to complete revocation by 9 December 2023.
• UPL Argentina S.A – Reseller not able to reach the Subscriber. Certificate was revoked on 7 December 2023.

Comment 1

[Bruce Morton]

Prudential Financial certificates were revoked on 2 December 2022.
ING Groep certificate completed revoking certificates on 4 December 2022.
Munich Re Group completed revoking certificates on 16 December 2022.
Azimut Holding certificates were revoked on 17 December 2022.

Comment 2

[Martijn Katerbarg]

Bruce, from the looks of it you were able to get in touch with all but one of these subscribers. We are wondering what your criteria are for granting an extension to the subscribers. Are you able to clarify the reasoning and rationale that led to this decision?
What persuaded you that these were truly exceptional circumstances?

While most of these are stated as subscribers requesting an extension due to end-of-year freezes, we wonder what would happen if one of these subscribers suffered a Key Compromise incident. Are they able to replace certificates within a 24 hour window in that case? Would you grant an extension in that case?

Do you have any plans, or have you had any discussions with these subscribers in order to prevent a similar incident in the future?

Finally, can you give an explanation as to why this incident was not opened within the 5 day revocation window as previously stated you plan on doing?

Comment 3

[Bruce Morton]

(In reply to Martijn Katerbarg from comment #2)

Bruce, from the looks of it you were able to get in touch with all but one of these subscribers. We are wondering what your criteria are for granting an extension to the subscribers. Are you able to clarify the reasoning and rationale that led to this decision?

We were concerned about the timing of the incident. Many companies have blackout periods from before US Thanksgiving through to the New Year. Based on the incident, we thought that a revoked certificate would provide more harm to Relying Parties than a certificate with an EV jurisdiction error.

What persuaded you that these were truly exceptional circumstances?

We are not able to verify that a blackout period exists, so we accepted the request from our Subscribers.

While most of these are stated as subscribers requesting an extension due to end-of-year freezes, we wonder what would happen if one of these subscribers suffered a Key Compromise incident. Are they able to replace certificates within a 24 hour window in that case? Would you grant an extension in that case?

No. Certificates with a compromised private key are revoked within 24 hours of proving key compromise.

Do you have any plans, or have you had any discussions with these subscribers in order to prevent a similar incident in the future?

No. This is a difficult problem to address. We will continue to develop and extend methods for automatic certificate renewal. The challenge is moving Subscribers in this direction.

Finally, can you give an explanation as to why this incident was not opened within the 5 day revocation window as previously stated you plan on doing?

Sorry for the confusion. Our plan was to revoke with 5-days, but would open a delayed revocation incident if we did not.

Please note our focus is to fix the root cause of the original incident, ensure we have a high standard registration matrix, and provide linting against this matrix. The result will be to be to mitigate incorrect jurisdiction information. If we can reduce certificate miss-issuance, then we will also reduce certificate revocation issues.

Comment 4

[Bruce Morton]

Experian certificates were revoked on 15 January 2023.

Comment 5

[Bruce Morton]

First Abu Dhabi Bank PJSC certificates were revoked on 6 February 2023.

Comment 6

[Chris Clements]

Entrust, can you please provide an update to items #6 and #7 in the incident report so that we can clearly understand root cause and remaining next steps?

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.
7. List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

Comment 7

[Bruce Morton]

Fidelity Investments certificate revocation was completed on 15 March 2023.
All certificates have been revoked.

Comment 8

[Bruce Morton]

(In reply to Chris Clements from comment #6)

Entrust, can you please provide an update to items #6 and #7 in the incident report so that we can clearly understand root cause and remaining next steps?

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

Due to the end of year blackout period and the interpreted low severity of the miss-issuance, we agreed to revocation dates with each customer, which could not revoke within 5-days. In one case, a customer did adhere to the committed date. We allowed this customer extended time to replace all miss-issued certificates.

7. List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

Focus first is on limiting incidents. We already do pre-issuance linting using zlint to prevent TLS certificates from being issued. We also perform post-issuance linting which includes zlint, plus Entrust specific tests for our specific certificate profiles, comparison to trusted sources and validated information. Post-issuance checking runs every 30 minutes, so if a certificate gets by pre-issuance linting, we may still have time to notify the Subscriber immediately, so the certificate may not go into service.

Late revocations are base primarily by Subscribers which have not implemented automation. We have increased our efforts to extend implementation of ACME. We are also considering implementing the ACME Renewal Information (ARI) Extension, which allow the certificate to be automatically renewed before the revocation date.

In the shorter term, we will provide Subscribers with automated reminders on the revocation date for miss-issued certificates. We will plan to allow short extensions, based on the severity of the miss-issuance.

Comment 9

[Ben Wilson]

It appears that this incident can be closed. I'll schedule closure for Wed. 19-Apr-2023.