1744627 - IdenTrust: Issuance of OV SSL Certificate with doc vetting older than 398 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 12/3/2021, during account information renewal verification activities, IdenTrust staff discovered that one OV TLS certificate was requested and issued on December 1, 2021 for an organization with account information verified prior to 398 days ago.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

12/1/2021: One OV TLS certificate was requested and issued using account information that was validated prior to 398 days ago.
12/3/2021: IdenTrust staff discovered the miss-issuance and revoked the certificate.
12/3/2021: Confirmed that remediation actions for incident report 1734917 (https://bugzilla.mozilla.org/show_bug.cgi?id=1734917) will prevent further future mis-issuance once the associated technical controls (code fixes) are propagated to production in January 2022.
12/6/2021: As a result of further investigation, IdenTrust staff discovered 2 additional organizations that had account information verified prior to 398 days ago and inactivated those accounts to prevent further mis-issuance.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, we confirmed there are no more organizations in process of certificate issuance with account information verified prior to 398 days ago.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

One OV SSL certificate which was revoked within 24 hours of discovery.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=5716597191

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   As a response to incident report 1734917 and until the associated technical controls (code fixes) described are propagated into production in January 2022, we have established an interim process to actively reverify all organizations with account information older than 398 days. As part of this interim process and due to a bug in a supporting report, any organization that could not be verified was not inactivated for further issuance.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The technical controls that will be deployed in response to incident report 1734917 will avoid this scenario and will not allow OV TLS certificate issuance for any organization with account information verified prior to 398 days. These technical controls will be in place in January 2022.
We will confirm the specific 2022 production deployment date by 12/30/2021.

Comment 1

[Ryan Sleevi]

As part of this interim process and due to a bug in a supporting report, any organization that could not be verified was not inactivated for further issuance.

Can you provide substantive detail? This seems to be core to the incident, and there are no details explaining what happened, why it happened, or why the 2022 work will prevent this. As the goal is to provide information to both assure relying parties and help other CAs prevent similar mistakes, it seems like a detailed breakdown of this “bug” is needed.

Comment 2

[IdenTrust]

(In reply to Ryan Sleevi from comment #1)

As part of this interim process and due to a bug in a supporting report, any organization that could not be verified was not inactivated for further issuance.

Can you provide substantive detail? This seems to be core to the incident, and there are no details explaining what happened, why it happened, or why the 2022 work will prevent this. As the goal is to provide information to both assure relying parties and help other CAs prevent similar mistakes, it seems like a detailed breakdown of this “bug” is needed.
IdenTrust:
The ‘bug’ is in a Report that supports an interim process to identify all organizations that require reverification of accounting information because the available account information was verified prior to 398 days ago. This Report did not identify the Organization associated with the problematic certificate (https://crt.sh/?id=5716597191) even though the account information was verified prior to 398 days ago. This ‘bug’ in the Report only manifests when following conditions are met: (i) Certificate Requests are received via the supported API; and (ii) the Organization does not have any active certificates. As a result of the ‘bug’ the Organization account information was not reverified prior to receipt of the certificate request. Once the problematic certificate was issued, the Organization was listed in the Report. At which point, we reverified the account information – however this was after issuing the problematic certificate and as such necessitating the revocation.

The work scheduled in January 2022 will implement technical controls that will reject certificate issuance if the account information was verified more than 398 days ago.

Comment 3

[Ryan Sleevi]

Thanks. I’m still concerned there’s a fair bit of detail missing that would help understand this incident, as well as understand precisely how it’s being prevented.

Can you share more detail about how the current report is made? Similarly, since you mentioned “via the supported API”, perhaps clarify (with diagrams) the paths of requests that can make it into your system? These are examples that help reveal both the current system and the thought process behind mitigations.

From Bug 1734917, it sounds like your report was simply looking at existing certificates and reminding organizations to provide updated documents. But if that’s the explanation for this issue, then it seriously overlooks that there’s a flaw in the issuance process itself, since one would have expected, at minimum, a human review during the issuance process, to complement the technical control being developed.

The goal here, as I mentioned, is to make sure that there’s enough detail to understand how the mistakes were made: the design flaw of the report, the lack of procedural controls, etc. For example, with more detail, it might suggest that a mistake CAs could make is reporting only on certificates, rather than on underlying validation data from the database. It’s unclear the reasons your report took the design approach it did, which again, is where more detail helps.

Comment 4

[IdenTrust]

(In reply to Ryan Sleevi from comment #3)

Thanks. I’m still concerned there’s a fair bit of detail missing that would help understand this incident, as well as understand precisely how it’s being prevented.

Can you share more detail about how the current report is made? Similarly, since you mentioned “via the supported API”, perhaps clarify (with diagrams) the paths of requests that can make it into your system? These are examples that help reveal both the current system and the thought process behind mitigations.

From Bug 1734917, it sounds like your report was simply looking at existing certificates and reminding organizations to provide updated documents. But if that’s the explanation for this issue, then it seriously overlooks that there’s a flaw in the issuance process itself, since one would have expected, at minimum, a human review during the issuance process, to complement the technical control being developed.

The goal here, as I mentioned, is to make sure that there’s enough detail to understand how the mistakes were made: the design flaw of the report, the lack of procedural controls, etc. For example, with more detail, it might suggest that a mistake CAs could make is reporting only on certificates, rather than on underlying validation data from the database. It’s unclear the reasons your report took the design approach it did, which again, is where more detail helps.

Certificate requests are received via one of the following paths:

1. Website based application forms. This path initiates a combination of technical and human review for issuance.
2. API based requests. This path initiates different workflow depending on the characteristics of the request.
   a. If request is from an established Organization, the issuance workflow consists of rules driven automated decision for issuance.
   b. If request is from any other organization, the issuance workflow is a combination of technical and human review for issuance (same as #1).
   The problematic certificate was received and issued via path #2a.

The report created due to Bug 1734917 gives visibility to our staff on any organization with active certificates on the date the report was executed and calculates the days remaining to re-verify the organization information. In anticipation of continued service and minimizing the time to approve a certificate request, Organizations nearing expiration of their vetting age (398 days in this case) are initiated into a vetting process which requires our staff to re-perform the verification of Organization information (which may require Organizations to resubmit updated account information). As mentioned before, a design flaw in the report resulted in the Organization for the problematic certificate not being listed for initiating the vetting process. The flaw in the report design was to not take into account that an established organization may have let their certificates expire.

Comment 5

[Ben Wilson]

(In reply to IdenTrust from comment #2)

The work scheduled in January 2022 will implement technical controls that will reject certificate issuance if the account information was verified more than 398 days ago.

To be on the safe side, maybe Identrust should implement the controls to make it no more than 395 days, or something to that effect.

Comment 6

[IdenTrust]

(In reply to Ben Wilson from comment #5)

(In reply to IdenTrust from comment #2)

The work scheduled in January 2022 will implement technical controls that will reject certificate issuance if the account information was verified more than 398 days ago.

To be on the safe side, maybe Identrust should implement the controls to make it no more than 395 days, or something to that effect.

IdenTrust:
The report is used to start revetting organizations 45 days prior to the 398 days deadline. In addition, the automated validation stops the issuance process if the due date to revet the organization is over 397 days.

Comment 7

[IdenTrust]

The tehcnical controls to validate if TLS issuance is allowed based on the age of the previously vetted documents will be deployed on January 22, 2022.

Comment 8

[IdenTrust]

Effective 1/20/2022 we deployed the automated validation across systems to prevent recurrence of this issue; now it is resolved.

Comment 9

[Ben Wilson]

If this can now be closed, I will take a look at closing it next week.