IdenTrust: Expired CRL served
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: roots, Assigned: roots)
Details
(Whiteboard: [ca-compliance] [crl-failure])
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0
Actual results:
2023.12.6 - Process alerts indicated that normal CRL checking had failed, resulting in one or more out-of-date CRLs being served. The problem was remediated, and an investigation was started promptly.
2023.12.11 - Investigation results showed that several CRLs subject to CA/B Forum requirements were involved.
We are continuing our research and will provide a full Incident report no later than 2023.12.29.
|
||
Updated•2 years ago
|
Incident Report
Summary
At 22:10:00 UTC on December 6, 2023, alerts highlighted a failure in the regular CRL checking process. Subsequent examination uncovered that 26 CRLs had expired, spanning a duration of 81 to 119 minutes.
The incident constituted a breach of the TLS BR section 4.10.2 regarding 24x7 CRL repositories. The problem was addressed on the same day, and an investigation was started promptly.
Impact
The incident impacted 10 publicly trusted CRLs.
Timeline
Status site observed start: 2023-12-06T22:10:00 UTC (15:10:00 MST)
Status site observed end: 2023-12-06T23:50:00 UTC (16:50:00 MST)
Total observed time service "EXPIRED" CRLs: ~81 to ~119 minutes
Root Cause Analysis
On 2023-12-05:10:00 UTC, a new network file system sharing service was introduced with default directory permissions configured as "Root." The next day, the PKI team in charge of publishing new 30-day CRLs encountered “permission denied” errors. This led to the removal of the new CRLs, leaving the outdated ones in expired status, a situation that was promptly detected by the CRL monitoring systems.
Lessons Learned
What went well
The prompt CRL monitoring system alerts allowed our teams to immediately troubleshoot and fix the issue
What didn't go well
Pre-installation checking to validate that the enabled permissions were sufficient for all teams on the new service.
Where we got lucky
Relatively short period of reporting expired CRLs
Action Items
We've already addressed the situation by discontinuing the recently introduced service and writing new CRLs and updates directly to their intended destinations. This eliminates the possibility of encountering "permission denied" errors associated with a shared file system and mitigates the risk of losing CRLs in case the shared storage becomes inaccessible or is removed.
Appendix
Details of affected certificates
No TLS ICA or Subscriber certificates were involved, only these publicly facing CRLs were affected:
http://validation.identrust.com/crl/trustidbahca1.crl
http://validation.identrust.com/crl/saicca1.crl
http://validation.identrust.com/crl/trustidcaas2.crl
http://validation.identrust.com/crl/trustidbahca2.crl
http://validation.identrust.com/crl/trustidcao1.crl
http://validation.identrust.com/crl/hydrantidcao1.crl
http://validation.identrust.com/crl/timestamping3.crl
http://validation.identrust.com/crl/trustidevcodesigning4.crl
http://validation.identrust.com/crl/trustidcaa14.crl
http://validation.identrust.com/crl/trustidevcodesigning3.crl
We have no pending remediation actions for this issue.
|
|
||
Comment 3•1 year ago
|
||
I will close this on or about next Wed. 24-Jan-2024.
|
|
||
Updated•1 year ago
|
|
|
||
Updated•1 year ago
|
Description
•