1854465 - IdenTrust: Expired ICAs CRLs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.2045.31

Actual results:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

   On 31 August 2023 at 6:08 AM MST, we noticed that an IdenTrust ICA was being
   flagged in CRL Watch with this error: “error parsing
   http://validation.identrust.com/crl/trustidevcodesigning4.crl: should have been
   updated by 2023-08-31 10:05:21 +0000 UTC”

   This is a potential violation of Section 4.10.2 of the CA/B Forum Baseline
   Requirements regarding Service Availability:
   The CA SHALL maintain an online 24x7 Repository that application software can
   use to automatically check the current status of all unexpired Certificates issued
   by the CA.

   IdenTrust monitoring confirmed that one connection node for its CRL repositories
   had not been updated properly with the latest CRLs, resulting in the potential for
   up to one-third of CRL requests to be returned with an “expired CRL” status
   response. The affected node provided expired CRL responses for up to 5 hours
   and 2 minutes, as is described below. However, since the node was part of a
   round-robin configuration, two-thirds of the nodes were unaffected and queries
   to them were responded to using valid CRLs.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable,
   or a document changed, or a bug was introduced, or an audit was done.

   • 2023-08-31 04:05 MDT: Existing CRLs expired and were replaced with newly
   generated ones, except for one node where replacement did not occur.
   • 2023-08-31 06:08 MDT: IdenTrust noticed the error and notified internal
   resources.
   • 2023-08-31 07:45 MDT: Confirmed issue in the monitor system and started an
   investigation, discovering the problem as described above.
   • 2023-08-31 08:07 MDT: Corrected the issue by removing the impacted server
   from the round-robin pool.
   • 2023-08-31 08:45 MDT: Review of the logs showed that the CRL for 1 ICA was
   expired for 5 hours and 2 minutes. The other 10 affected CRLs expired at different
   times within an hour of each other, and the average expiration length for these
   10 was 3 hours and 39 minutes.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation

Certificate issuance was not affected. The CRL replication failure was resolved 
promptly upon diagnosis.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

   No certificates were affected. The following CRLs were impacted:

   CRL Time expired on 08.31 Time before resolution at 8:07 MDT
   http://192.147.157.157/crl/trustidbahca1.crl 4:27:43 MDT 3:39
   http://192.147.157.157/crl/trustidbahca2.crl 4:38:17 MDT 3:29
   http://192.147.157.157/crl/trustidevcodesigning3.crl 3:05:19 MDT 5:02
   http://192.147.157.157/crl/trustidevcodesigning4.crl 4:05:21 MDT 4:02
   http://192.147.157.157/crl/trustidevcodesigning5.crl 4:27:57 MDT 3:39
   http://192.147.157.157/crl/trustidcaas.crl 4:35:21 MDT 3:32
   http://192.147.157.157/crl/saicca1.crl 4:35:23 MDT 3:32
   http://192.147.157.157/crl/trustidcae1.crl 4:37:56 MDT 3:29
   http://192.147.157.157/crl/trustidcao1.crl 4:37:56 MDT 3:29
   http://192.147.157.157/crl/timestamping3.crl 4:05:21 MDT 4:02
   http://192.147.157.157/crl/publicsectorserverca1.crl 4:27:52 MDT 3:40

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

   No such certificates were impacted.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

   The error occurred while updating network equipment to new devices. Although
   successful testing of the new equipment had occurred for two months prior to
   installation, a utility service did not function on the new equipment, and
   detection was not made until the then-current CRLs reached their expiration
   points approximately 8 hours later.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

   IdenTrust removed the affected node from the round-robin pool promptly upon
   discovery. This returned the CRL responder services to their normal availability
   state with respect to current CRLs, thus resolving the issue.

   Situations like this occur infrequently. However, to prevent future incidents of
   this type, IdenTrust is instituting more stringent processes for future equipment
   upgrades and is increasing testing and monitoring in this area. Specific items
   include the following testing steps before and after installation:
   • Documenting necessary configurations that we will refer to for the next device
   upgrade;
   • Testing that new CRLs are generated before expiration of the old ones;
   • Testing that CRLs from our main servers are current-generation and not
   expired;
   • Testing that CRLs from all remote nodes are current generation and not
   expired; and
   • Testing that the same CRLs are being served from all nodes.

   We plan to have all items in place by the end of December 2023, and we will
   update this report monthly until all new steps are in place and proper function is
   ensured. The next update is scheduled for 2023-10-20.

Comment 1

[IdenTrust]

We have implemented stricter procedures for futures equipment upgrades. This includes monitoring improvements both prior to and following equipment installation. There are no outstanding action items for this matter.

Comment 2

[Ben Wilson]

I'll schedule to close this next Wed. 1-Nov-2023 unless there are comments or questions.