1853447 - IdenTrust: Temporarily Expired CRLs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

Actual results:

On September 8, 2023, IdenTrust found that four CRLs had expired. Our Investigation showed that a job-control utility had failed. The job-control utility was restarted, the system returned to normal operations and performance is being monitored closely. We returned to normal operations shortly after the issue was discovered and fixed. IdenTrust is examining root causes and other details of this incident. We will provide full incident report on or before September 29, 2023.

Comment 1

[IdenTrust]

Here is the incident report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

• On 9-8-2023 noticed that IdenTrust was being flagged in CRL Watch with this error: “error parsing http://validation.identrust.com/crl/trustidevcodesigning4.crl: should have been updated by 2023-09-08 10:05:24 +0000 UTC”.

Section 4.10.2 of the CA/B Forum Baseline Requirements regarding Service Availability was not met, since we were unable to maintain 24/7 availability for some CRLs.

---

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2023-09-08 12:45MT: IdenTrust was flagged in CRL watch
• 2023-09-08 13:30 MT: IdenTrust noticed that we were being flagged in CRL Watch
• 2023-09-08 13:45 MT: We began investigating the scope of affected certificates.
• 2023-09-08 13:57 MT: The job-control utility was restarted
• 2023-09-08 13:59:57MT: New CRLS were generated and loaded successfully.

We confirmed that a major IdenTrust customer was revoking a large number of certs during the time the utility was attempting to create the new CRLs. This introduced some confusion into the system, resulting in a CRL generation slowdown to where these 3 CRLs expired before their replacements were available.

---

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation

• The job-control utility was restarted. The system returned to normal operations shortly after the issue was discovered and fixed.

---

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

• No certificates were affected. The following CRLs were impacted:

http://validation.identrust.com/crl/trustidevcodesigning4.crl:
http://validation.identrust.com/crl/trustidbahca1.crl:
http://validation.identrust.com/crl/trustidcaa13.crl:

---

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

• N/A. No certificates were affected.

---

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now

• The failure occurred because a major IdenTrust customer was revoking a large number of certs during the time the utility was attempting to create the new CRLs. This introduced some confusion into the system, resulting in a CRL generation slowdown to where the 3 CRLs expired before their replacements were available. No other IdenTrust CRLs were impacted.

---

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• We restarted the job-control utility. The system returned to normal operations shortly after the issue was discovered and fixed.
• We have updated internal processes to reduce CRL-creation impacts on customers.
• We have optimized the process for preventing, detecting, and responding to similar situations.

Comment 2

[IdenTrust]

We have nothing else pending to correct on our side regarding this issue and considered it close.

Comment 3

[Ben Wilson]

Thanks. I will close this on Wed 11-Oct-2023 unless there are additional questions to be answered.