1734917 - IdenTrust: Mis-Issued EV Certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

Attachment: IdenTrust-List of Mis-issued EV Certificates.xlsx

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

IdenTrust: On 9/27/2021 IdenTrust performed an internal review of compliance for the maximum age of documentation that can re-used for re-issuance and domain vetting of EV TLS certificates (EV section 11.14). IdenTrust determined that the requirement for re-vetting documentation that is older than 398 days, which became effective on June 2, 2021, was not implemented for Enterprise customers currently requesting renewal of a certificate via our standard API. In performing an audit of EV certificates issued, we identified a total of 124 certificates, requested by 7 customers. 22 of the certificates were issued between April 5, 2021 and June 1, 2021 where the age of the documentation was older than 13 months. An additional 102 were issued between June 3, 2021 and September 24, 2021 where the age of the documentation was older than 398 days.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   IdenTrust:
   • 9/24/2021: Conducted internal audit review of issued EV certificates and identified a possible issue.
   • 9/27/2021: Confirmation of mis-issuance of certificates due to absence of system queuing for certificate requests processed via IdenTrust API requiring re-validation of vetting documentation older than 398 days for 124 EV certificates issued 4/5/2021 thru 09/24/2021.
   • 9/28/2021: Prepare remediation plan and begin execution of tasks, (See section 7 for list of actions).
   • 9/28/2021: Risk Management Committee (RMC) review of situation and approval of remediation plan.
   • 10/08/2021: Submitted CCADB incident report.
   • 10/TBD/2021: Sent notifications to seven customers impacted by this situation including pertinent information and recommendations

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

IdenTrust: Yes, an immediate process was implemented at our Registration area to ensure that all organizations with active TLS certificates are vetted within 398 days. The API will be augmented to identify and queue certificates in this category for re-vetting. Target implementation date before December 31, 2021; until then organization re-vetting will occur manually
4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

IdenTrust: 124 EV SSL certificates (7 of which were previously revoked) with document age older than 13 months or 398 days, first issued on 4/5/2021, last issued on 9/24/2021.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

IdenTrust: See attached file IdenTrust-List of Mis-issued EV Certificates.xls.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   IdenTrust: The current process to trigger re-validation of documentation and domain validation has been discovered to not work properly when certain conditions are met. Specifically, when certificates are requested via the API to our authoritative Registration system, the re-vetting process was not triggered. This deficiency resulted in in 124 certificates being issued without re-validating vetting documentation after 398 days from the original document validation.
   • List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

IdenTrust:
• Ran queries to identify all impacted certificates totaling 124 and 7 organizations - Completed

• Updated manual process to identify requests in this category and re-validate vetting documentation - Completed

• Perform revalidation of vetting documentation for mis-issued certificates and confirmed that existing organization information is valid - Completed

• Document all remediation activities – Completed

• Implement an automated trigger to place all certificate requests where the documentation review is approaching the 398 day validation date in a separate queue, which will require revalidation of vetting documentation prior to approval of issuance no later than December 31, 2021 – Pending

• Notified 7 customers of incident and requested approval to reissue mis-issued certificates – Completed

Notification includes the following:
- Explanation that the previously submitted documentation has been re-vetted to meet EV document validation requirements including
- Confirmation that all information included in mis-issued certificate is accurate based on the re-validation of documentation
- Request for approval to re-issue all affected certs
- Notification that customers will have 30 days to replace the affected certs with the re-issued certificates on their platforms
- Informed that all mis-issued certs will be revoked at time of customer confirmation of replacement or 30 days from this incident report, whichever comes first

• Compliance will confirm that all mis-issued certificates have been revoked by stated timeline - Pending

Comment 1

[IdenTrust]

Status Update:
• Implement an automated trigger to place all certificate requests where the documentation review is approaching the 398 day validation date in a separate queue, which will require revalidation of vetting documentation prior to approval of issuance no later than December 31, 2021 – Pending – On Track – We will provide an update on October 29, 2021
• Compliance will confirm that all mis-issued certificates have been revoked by stated timeline – Pending – Progressing on reissuing and replacing affected certificates. We will provide an update on November 8, 2021

Comment 2

[Ben Wilson]

I received the following concern: IdenTrust detected and confirmed that it had mis-issued 124 certificates on 2021-09-27, and filed the bug report on 2021-10-08, but its incident report says, "all mis-issued certs will be revoked at time of customer confirmation of replacement or 30 days from this incident report, whichever comes first". According to the BRs, Identrust must revoke the mis-issued certificates within 5 days of discovery. Where does this "30-day" window come from?
Delays in revocation of leaf certificates usually require the filing of another incident in Bugzilla, which are coded in the whiteboard with "[ca-compliance] [delayed-revocation-leaf]".

Comment 3

[Andrew Ayer]

The current process to trigger re-validation of documentation and domain validation has been discovered to not work properly when certain conditions are met. Specifically, when certificates are requested via the API to our authoritative Registration system, the re-vetting process was not triggered. This deficiency resulted in in 124 certificates being issued without re-validating vetting documentation after 398 days from the original document validation.

It sounds like certificates were issued with stale domain validation information. Is this true? If so, the revocation deadline is actually 24 hours, because the domain validation for these certificates cannot be relied upon.

Could this deficiency have allowed certificates to be issued with validation information older than 825 days?

Comment 4

[Andrew Ayer]

Also, why did IdenTrust wait 11 days after confirming the misissuances before filing this incident report?

Comment 5

[IdenTrust]

(In reply to Ben Wilson from comment #2)
IdenTrust
Although Section 4.9.1.1 of the B.R. explicitly lists the reasons under which a subscriber SSL certificates must be revoked within 5 days, the certificates in this incident report comply with all the B.R. SSL requirements; however, due to the “Extended Validation” required checks, these certificates were found to be out of compliance due to expiration of the original approved documentation. In our view, these certificates were miss-issued – thus the incident report – but:

1. Poses minimal security risk as the identity information on each identified certificate has been validated post issuance. All re-validation passed without change to any of the underlying account information that was used for issuing the identified certificates.
2. We balanced the security risk vs. sufficient time for the customers to avoid service disruptions of the externally facing services protected by the identified certs.

The missing additional Incident Report for the delay revocation has been filed: https://bugzilla.mozilla.org/show_bug.cgi?id=1736706

Comment 6

[IdenTrust]

(In reply to Andrew Ayer from comment #3)

It sounds like certificates were issued with stale domain validation information. Is this true?
IdenTrust: No. The organization\account information was stale.

Could this deficiency have allowed certificates to be issued with validation information older than 825 days?
IdenTrust: No, the maximum vetting age of organization\account information was 473 days.

Comment 7

[IdenTrust]

(In reply to Andrew Ayer from comment #4)

Also, why did IdenTrust wait 11 days after confirming the misissuances before filing this incident report?

IdenTrust: Unfortunately, the coordination with customers and internally for establishing a remediation plan took longer than expected. We also had unavailability of key personnel that contributed to the delayed filing.

Comment 8

[IdenTrust]

Status Update:
• Implement an automated trigger to place all certificate requests where the documentation review is approaching the 398 day validation date in a separate queue, which will require revalidation of vetting documentation prior to approval of issuance no later than December 31, 2021:
– On Track - We will provide an update on November 30, 2021

In reference to the 124 miss-issued certificates, all but 3 have been revoked. The remaining 3 are expected to be revoked within the next 24 hours; a confirmation update will follow.

Comment 9

[IdenTrust]

The remaining 3 certificates have been revoked.

Comment 10

[IdenTrust]

We have implemented a visual queue showing upcoming document re-vetting to pro-actively start the re-vetting process before the 398 days expiration. We have also coded automatic validation before new or renew EV SSL issuance; however, due to year-end code-freeze, production release is expected in January 2022.
We will post an update by December 31, 2021 confirming the 2022 production release date.

Comment 11

[IdenTrust]

The tehcnical controls to validate if TLS issuance is allowed based on the age of the previously vetted documents will be deployed on January 22, 2022.

Comment 12

[IdenTrust]

Effective 1/20/2022 we deployed the automated validation across systems to prevent recurrence of this issue; now it is resolved.

Comment 13

[Ben Wilson]

If this can now be closed, I will take a look at closing it next week.