1758027 - IdenTrust: Pre-certificates without a final certificate showing OCSP error

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36

Steps to reproduce:

During an internal review, we found instances of “unauthorized" OCSP responses for pre-certificates where a final certificate was never issued. So far, we have discovered 4543 pre-certificates without a final certificate with this issue.

We are still investigating the root cause and will be supplying a full incident report shortly.

Comment 1

[IdenTrust]

We have made significant progress not only discovering the root cause but also updating those ‘abandoned’ pre-certificates to reflect a valid OCSP status. We are still working on timelines to correct the issue for new issuance.
The formal complete incident report will be posted no later than 2022-03-15

Comment 2

[IdenTrust]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

During our internal review on 2202-02-23, we found 4,543 instances where a pre-certificate was generated and the status of each pre-certificate is not available in the OCSP responder. Through further investigation, we discovered additional 124 pre-certificates exhibiting the same issue. In all 4667 cases, the pre-certificate was submitted to certificate transparency logs and the IdenTrust OCSP responders responded with "unauthorized" status.

As pre-certificates are treated the same as certificates under CA/B Forum rules and as such need to have revocation information available, this is a violation of section 4.10.2 of the CA/B Forum Baseline Requirements (quotes added):
The CA SHALL maintain an online 24x7 Repository that application software can use to automatically check the current status of “all unexpired Certificates” issued by the CA.

Also per Mozilla recommended practice guidelines for handling pre-certificates:
A CA must provide OCSP services and responses in accordance with Mozilla policy for all certificates presumed to exist based on the presence of a precertificate, even if the certificate does not actually exist.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2022-2-23 15:51 MST: Discovered the pre-certificates with this discrepancy.
2022-2-24 11:13 MST: Confirmed the discrepancy with other teams proceeded to investigate resolution.
2022-3-3 15:30 MST: Investigation discovered the cause of the issue and tasked engineering to start working on a solution.
2022-3-8 10:47 MST Identified an additional 124 pre-certificates with the same issue
2023-3-11 7:44 MST completed discovery and identified the root cause

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

No, while we deploy a permanent fix, we will be executing a script on a daily basis to search and remediate any unexpired pre-certificate with an OCSP status of “unauthorized”

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

4667 unexpired pre-certificates issued between 2021-04-07 and 2022-03-11

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   See attached excel file

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our current design for TLS issuance process assumes that for every pre-certificate disclosed to CT logs, there will always be a final certificate issued and, thereby, the pre-certificate was not registered with the OCSP DB. However, in the case of high volume requests, there are instances of unexpected errors that have prevented the issuance of the final certificate. Without a final certificate issued, the OCSP responder is unable to locate the associated pre-certificate, thereby responding with the ‘unauthorized’ status.

This issue avoided detection until now because when there was failure to issue a final certificate, the requesting system repeated the request until successful, leaving the initial pre-certificate without the corresponding final certificate.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

To resolve this issue, we have registered all identified pre-certificates in the OCSP DB, and confirmed a valid OCSP response for each pre-certificate.

In order to prevent recurrence, we will update the TLS issuance process to register each pre-certificate into the OCSP DB regardless if the final certificate will be issued. We expect to have this update in production by 2022-05-23.

By 2023-03-21, we will execute a script on a daily basis to search and remediate any unexpired pre-certificate with an OCSP status of “unauthorized”, until the permanent resolution is deployed by 2022-05-23.

Comment 3

[IdenTrust]

Attachment: Identrust Precertificates Missing-OCSP-Entries.xlsx

Comment 4

[IdenTrust]

Effective today, 2022-03-16, we have in place a monitoring utility to check every 24 hours for pre-certificates missing a final certificate remediating the OCSP status. We will provide month-end updates on the progress of the permanent solution.

Comment 5

[IdenTrust]

For the permanent solution, as indicated in item#7 of the report: "In order to prevent recurrence, we will update the TLS issuance process to register each pre-certificate into the OCSP DB regardless if the final certificate will be issued. We expect to have this update in production by 2022-05-23."
This is currently in QA testing and it is still expected to be in Production by 2022-05-23.
We will provide another status update on April 29, 2022.

Comment 6

[IdenTrust]

We are on track to deploy the permanent solution for this issue on 2022-05-21.
We will post a status update by 2022-05-23.

Comment 7

[IdenTrust]

We have successfully deployed the solution mentioned in comment #5 above and consider this issue resolved.

Comment 8

[Ben Wilson]

I will close this on or about Friday 27-May-2022 unless there are additional questions or issues to resolve.