2087789 - HARICA: 2 truncated CRLs published

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[HARICA]

Preliminary Incident Report

Summary

• Incident description: On 2026-10-29 at approximately 23:31 UTC, two Certificate Revocation Lists (CRLs) were issued and published in a truncated format. HARICA’s internal monitoring system generated an alert regarding this anomaly on 2026-10-30 at 02:52 UTC. The CA engineering team acknowledged the alert at 08:00 UTC and immediately launched an investigation to troubleshoot the issue.
  Shortly after, at 08:25 UTC, a third party contacted our general support email to report the anomaly. They detected the issue via their independent CRL-monitoring service, which downloads CRLs from the CRL Distribution Point (CRLDP) and parses them as DER.
  The engineering team successfully corrected the problem at the origin site by 09:06 UTC. At 09:16 UTC, the corrected CRLs were successfully propagated and published to our Content Delivery Network (CDN).
  Later that day, at 13:49 UTC, a second third party filed a report via the formal Certificate Problem Report process regarding the truncated CRLs; however, the incident had already been fully remediated by this time.
  The initial investigation revealed that the truncation was created due to a temporary exhaustion of disk space that occurred precisely during the generation of the two affected CRLs. This underlying storage issue has been resolved.
  A comprehensive incident report will be published no later than 2026-11-11.

• Relevant policies: Sections 4.9.7 and 4.10.2 of the TLS Baseline Requirements

• Source of incident disclosure: The incident was detected by HARICA and was already being investigated when a third-party reported the issue via the general support email address.

Comment 1

[HARICA]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000035
• Incident description: On 2026-10-29, HARICA published two truncated Certificate Revocation Lists (CRLs) to its Content Delivery Network (CDN). The affected CRLs had successfully passed linting, but their subsequent copy operation failed partially due to a temporary exhaustion of available disk space on the filesystem used by the CRL publishing process. As a result, incomplete CRL files were published on the CDN.
• Timeline summary:
  • Non-compliance start date: 2026-10-29
  • Non-compliance identified date: 2026-10-30
  • Non-compliance end date: 2026-10-30
• Relevant policies: Sections 4.9.7 (CRL issuance frequency) and 4.10.2 (Service availability) of the CA/Browser Forum TLS Baseline Requirements, Version 2.3.1.
• Source of incident disclosure: Self Reported: The incident was detected by HARICA and was already being investigated when a third-party reported the issue via the general support email address.

Impact

• Total number of certificates: N/A
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: N/A
• Incident heuristic: Two CRLs generated and published on 2026-10-29 at approximately 23:31 UTC were affected by this incident. Both CRLs were published in truncated format and subsequently replaced with corrected versions on 2026-10-30 at 09:16 UTC.
• Was issuance stopped in response to this incident, and why or why not?: Certificate issuance was not stopped, as the incident affected the generation and publication of CRLs rather than the certificate issuance process.
• Analysis: N/A
• Additional considerations: N/A

Timeline

All times are in UTC unless said otherwise:

• 2026-10-29 13:02: A CA administrator begins an administrative operation on a server where online CRLs are issued and gradually consumes the available space on a filesystem used by the CRL publishing process.
• 2026-10-29 15:50: An alert for another service (unrelated to the CRL) is triggered because a filesystem was full.
• 2026-10-29 23:31: The copy operation of the two successfully linted CRLs fails partially due to insufficient available filesystem space, resulting in truncated CRLs being published to the CDN. NON-COMPLIANCE BEGINS
• 2026-10-29 23:57: HARICA’s internal monitoring system generates a warning regarding CRLs missing the expected refresh interval
• 2026-10-30 02:52: HARICA’s internal monitoring system generates an alert regarding the truncated CRLs
• 2026-10-30 07:27: The infrastructure engineering team acknowledges the space exhaustion issue
• 2026-10-30 07:30: The infrastructure engineering team frees up space
• 2026-10-30 08:25: A third party reports the truncated CRLs to HARICA’s general support email address, following detection through their independent CRL-monitoring service.
• 2026-10-30 08:30 - 09:06: The CA engineering team issues new CRLs for all online CAs and updates the origin site.
• 2026-10-30 09:16: The corrected CRLs are successfully propagated and published to HARICA’s Content Delivery Network (CDN). NON-COMPLIANCE ENDS
• 2026-10-30 13:49: A second third party submits a Certificate Problem Report concerning the truncated CRLs. The issue has already been remediated by this time.
• 2026-10-31 18:02: Submission of the Preliminary Incident Report to Bugzilla.
• 2026-11-02 to 2026-11-11: Detailed Root Cause Analysis, Lessons Learned and Remediation Action Plan; compilation of the Full Incident Report (this report).

Related Incidents

Bug	Date	Description
1954580	2025-03-17	Database outage due to file system filled to 100% of capacity, causing failure in OCSP
1900129	2024-05-31	CRL monitoring failure to detect a CRL configuration error caused by human mistake

Root Cause Analysis

Contributing Factor #1: Exhaustion of filesystem space during CRL publication

• Description: During the publication of the two affected CRLs on 2026-10-29, an administrative operation consumed the available filesystem space used by the CRL publishing process. The administrative operations were performed in a location that shared the same filesystem as the CRL publishing process, rather than in a separate filesystem designated for administrative operations.

  As a result, the filesystem space for the copy operation of the already linted CRLs was insufficient to complete the operation successfully, contributing to the publication of truncated CRLs.

• Timeline:

  • 2026-10-29 13:02: A CA administrator begins a long running operation that gradually consumes the available disk space.
  • 2026-10-29 23:31: The copy operation of the two successfully linted CRLs failed partially due to insufficient available filesystem space, resulting in truncated CRLs being published to the CDN.
  • 2026-10-30 02:52: HARICA's internal monitoring system generated an alert related to the affected CRLs. No alert was identified for the disk space exhaustion itself.

• Detection: The filesystem space exhaustion was not detected through a dedicated alert because the filesystem monitoring check operates at the host level, covering all filesystems under a single check. At the time of the incident, this check was already in a CRITICAL state due to another filesystem that was almost full. Consequently, when the filesystem used by the CRL publishing process also ran out of space, no new alert was generated because the overall check state did not change.

• Interaction with other factors: The filesystem space exhaustion caused the copy operation of the linted CRLs to fail partially, resulting in truncated CRLs being published to the CDN.

• Root Cause Analysis methodology used: 5-Whys

Contributing Factor #2: Partial failure of the linted CRL copy operation

• Description: During the publication of the two affected CRLs on 2026-10-29, the copy operation of the successfully linted CRLs failed partially, due to insufficient available filesystem space in the destination directory. Despite the incomplete copy operation, the resulting truncated CRL files were published to the CDN.
• Timeline:
  • 2026-10-29 approximately 23:31: The two affected CRLs successfully passed linting, but their subsequent copy operation failed partially due to insufficient available filesystem space. The resulting truncated CRLs were published to the CDN.
  • 2026-10-30 09:06: The issue was corrected at the origin site.
  • 2026-10-30 09:16: The corrected CRLs were propagated and published to the CDN
• Detection: The partial failure of the copy operation was identified through HARICA's internal monitoring system after the truncated CRLs had been published. The existing safeguards did not detect or prevent the incomplete copy operation from resulting in the publication of truncated CRLs.
• Interaction with other factors: The partial failure of the copy operation was a direct consequence of the filesystem space exhaustion (CF-1) and resulted in truncated CRLs being published to the CDN.
• Root Cause Analysis methodology used: 5-Whys

Lessons Learned

• What went well: HARICA's internal monitoring system detected the publication of the truncated CRLs, allowing the engineering team to investigate and address the issue. The corrected CRLs were subsequently published to the CDN.
• What didn’t go well:
  • The administrative operation exhausted the available filesystem space used by the CRL publishing process, and the corresponding alert was suppressed.
  • The partial failure of the CRL copy operation was not detected or prevented before the truncated files were published to the CDN.
• Where we got lucky: The issue was limited to 2 CRLs.
• Additional: -

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Create a dedicated filesystem for temporary administrative operations, clearly separated from filesystems used by critical PKI processes.	Prevent	Root Cause # 1	Dedicated filesystem created, its intended use documented, and access and usage guidelines communicated to relevant administrators.	2026-11-11	Completed
Eliminate the unprotected copy step by linting CRLs immediately before uploading them to the CDN.	Prevent	Root Cause # 2	CRL linting is performed as the final step before each CRL is uploaded to the CDN.	2026-11-20	In progress
Create a dedicated filesystem for CRL generation and other critical PKI processes/applications, ensuring that administrative operations cannot consume the resources required by these processes.	Prevent	Root Cause # 1, #2	Dedicated filesystem provisioned and configured; isolation from administrative temporary operations verified through testing.	2026-11-27	In progress
Research and adopt an enhanced storage monitoring solution.	Detect / Mitigate	Root Cause # 1	Accurate detection of similar issues, providing sufficient lead time to address them proactively and minimize their potential impact.	2027-01-22	In progress

Appendix

N/A; this issue does not affect certificate issuance.

Comment 2

[HARICA]

We kindly request setting the next update to 2026-11-20 and we're monitoring this bug for any additional questions or comments..