SSL.com: Inconsistent published CRL for one ICA
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: secauditor, Assigned: secauditor)
Details
(Whiteboard: [ca-compliance] [crl-failure])
Preliminary Incident Report
Summary
- Incident description:
For one of our TLS issuing CAs, inconsistencies were observed in the published CRL, where revoked certificate serial numbers were intermittently missing from or duplicated within the CRL.
- Relevant policies:
Baseline Requirements v2.3.0, Section 4.10.1:
"Revocation entries on a CRL or OCSP Response MUST NOT be removed until after the Expiry Date of the revoked Certificate."
- Source of incident disclosure: Third Party Reported.
SSL.com will post our Full Incident Report on or before 2026-10-16.
|
||
Updated•16 days ago
|
Full Incident Report
Summary
-
CA Owner CCADB unique ID: A002038
-
Incident description:
-
On 2026-04-03, SSL.com completed a mass revocation of 1.7 million certificates. The details of this mass revocation are captured in Bug 2029230. 1,298,817 of these certificates were issued by the Cloudflare TLS Issuing ECC CA 3. By default, our software processed populating the revoked certificate information across several CRLs and in no guaranteed order. For CRLs with more than 500,000 entries, this lack of ordering caused some entries to be duplicated or missed on one CRL but to appear correctly on a later CRL. Only CRLs of the Cloudflare TLS Issuing ECC CA 3 met this condition. Our software does provide a configuration parameter to force a guaranteed order, but it is disabled by default.
-
Timeline summary:
- Non-compliance start date: 2026-04-20 16:53 UTC - First CRL published that dropped an unexpired serial number.
- Non-compliance identified date: 2026-09-30 12:56 UTC - Email reporting a CRL entry missing or duplicated issue was received.
- Non-compliance end date: 2026-10-01 15:35 UTC - New CRL for the only CA with a CRL large enough to be affected by the bug was generated and published. Non-impacted servers were patched later.
-
Relevant policies:
Baseline Requirements v2.3.0, Section 4.10.1:
"Revocation entries on a CRL or OCSP Response MUST NOT be removed until after the Expiry Date of the revoked Certificate."
-
Source of incident disclosure: Third Party Reported.
Impact
- Total number of certificates: . The only CRLs affected were from CN=Cloudflare TLS Issuing ECC CA 3. There were 4 revoked certificates across 16 incorrect removal events (a CRL omitting a previously listed, unexpired entry), and every one of them reappeared on a later CRL. In total 16 CRLs in the range of CRL #203 to #356 were affected, and none from #357 on.
- Total number of "remaining valid" certificates: All certificate serials in this incident were already revoked.
- Affected certificate types: All certificate serials incorrectly dropped from CRLs are DV TLS.
- Incident heuristic: All CRLs generated from Cloudflare TLS Issuing ECC CA 3 from 2026-04-20 16:53 until 2026-10-01 15:35.
- Was issuance stopped in response to this incident, and why or why not?: No, since there were no misissuances involved in this incident, issuance was not stopped.
- Analysis: N/A, no revocations delayed.
- Additional considerations: OCSP responses remained correct throughout. The issue only affected CRLs. In the same range of affected CRLs, 26 CRLs listed one serial number twice, always with the same revocation date and reason, so no entry carried conflicting information.
Timeline
All times are in UTC unless otherwise noted.
2020-12 - EJBCA version 7.4.3.2 was released, including a parameter described as whether EJBCA should request ordered fetching of revoked certificates when generating CRLs.
2021-03-21 - Production deployment of EJBCA version 7.4.3.2 was completed.
2026-04-03 15:57 - Mass revocation CRL for Cloudflare TLS Issuing ECC CA 3 published from bug.
2026-04-20 16:53 - First CRL published that dropped an unexpired serial number.
2026-09-30 12:56 - Email reporting a CRL entry missing and duplicated issues was received.
2026-09-30 14:26 - CRL #356 was generated, still dropping an unexpired serial number.
2026-09-30 14:52 - CA Admin Team began research into the issue with the CRL reported.
2026-09-30 22:46 - The inconsistency was confirmed from historical CRLs.
2026-09-30 23:53 - An archival bug in the software was identified as a contributory factor.
2026-10-01 01:00 - Multiple fixes were determined as options.
2026-10-01 10:16 - Fixes were reviewed and configuration change was staged.
2026-10-01 10:53 - Reporter notified.
2026-10-01 12:20 - Deployment of the corrected configuration to production began.
2026-10-01 15:21 - Deployment was completed and verified on impacted servers.
2026-10-01 15:35 - New CRL was generated and published.
2026-10-02 14:00 - Deployment started on non-impacted servers.
2026-10-02 16:40 - Deployment was completed on both impacted and non-impacted servers.
2026-10-02 20:22 - Preliminary Incident Report posted.
2026-10-16 – The full incident report is posted.
Related Incidents
| Bug | Date | Description |
|---|---|---|
| 2075655 | 2026-09-25 | Actalis: Incorrect Revocation dates in CRL entries |
| 2044788 | 2026-06-03 | Let's Encrypt: CRLs Temporarily Missing Revoked Serials |
| 2029230 | 2026-04-03 | SSL.com: Incorrect Open MPIC Lambda implementation by EJBCA ACME Service. |
| 1954861 | 2025-03-18 | Let's Encrypt: Early CRL Removal Incident |
| 1943379 | 2025-01-23 | Actalis: CRL with duplicate serial number in revokedCertificates |
Root Cause Analysis
Contributing Factor #1: EJBCA Parameter addition defaulted to off
- Description:
In EJBCA version 7.4.3.2 (released December 2020), a new parameter was introduced and described as whether EJBCA should request ordered fetching of revoked certificates when generating CRLs. Microsoft SQL Server 2016 was known to return duplicated and/or missing entries when multiple batches were read. The parameter default was set to off. SSL.com release review team was using a different SQL database provider, so this new parameter was allowed to default and was not fully tested to see if there were other potential undocumented impacts with the current database. This defect only manifested with sufficiently large CRL lists and had not occurred for our production environment until after the 1.7 million certificate mass revocation event.
- Timeline:
2020-12 - EJBCA version 7.4.3.2 was released, including a parameter described as whether EJBCA should request ordered fetching of revoked certificates when generating CRLs.
2021-03-21 - Production deployment of EJBCA version 7.4.3.2 was completed.
2026-04-03 15:57 - Mass revocation CRL for Cloudflare TLS Issuing ECC CA 3 published from bug.
2026-04-20 16:53 - First CRL published that dropped an unexpired serial number.
2026-09-30 12:56 - Email reporting a CRL entry missing and duplicated issues was received.
026-09-30 14:26 - CRL #356 was generated, still dropping an unexpired serial number.
2026-09-30 14:52 - CA Admin Team began research into the issue with the CRL reported.
2026-09-30 22:46 - The inconsistency was confirmed from historical CRLs.
2026-09-30 23:53 - An archival bug in the software was identified as a contributory factor.
2026-10-01 01:00 - Multiple fixes were determined as options.
2026-10-01 10:16 - Fixes were reviewed and configuration change was staged.
2026-10-01 12:20 – Deployment of the corrected configuration to production began.
2026-10-01 15:21 - Deployment was completed and verified on impacted servers.
2026-10-01 15:35 - New CRL was generated and published.
2026-10-02 14:00 - Deployment started on non-impacted servers.
2026-10-02 16:40 - Deployment was completed on both impacted and non-impacted servers.
- Detection: A report was received about a potential CRL compliance issue. SSL.com was not monitoring CRLs specifically for dropping or duplication of serial numbers.
- Interaction with other factors: Contributing Factor #2 kept the list of revoked certificates large enough to need more than one page of results, which is the condition under which the non-ordered query can miss or duplicate entries.
- Root Cause Analysis methodology used: 5-Whys
Contributing Factor #2: Archival status change bug delayed expired revoked certificate removal from CRL
- Description: On each full CRL generation cycle, the CA software transitions currently expired revoked certificates after one additional CRL entry to an archived status in the database. The certificate archived status prevents the CRL generation process from pulling that certificate information for the next CRL generation cycle. Because of a software bug, this archival process was limited to 10 seconds per CRL generation instead of the longer time intended. With one CRL generated per day, only around 2,000 expired certificates were archived each day, while hundreds of thousands were still pending archival after April's mass revocation (about 860,000 on 2026-10-02).
- Timeline:
2026-04-03 15:57 - Mass revocation CRL published from bug
2026-04-20 16:53 - First CRL published that dropped an unexpired serial number.
2026-09-30 14:52 - CA Admin Team began research into the issue with the CRL reported.
2026-09-30 23:53 - An archival bug in the software was identified as a contributory factor.
- Detection: Investigation into the potential CRL compliance issue reported by the third party led to finding this bug. SSL.com was not monitoring at this time for ensuring that removal of serial numbers for expired certificates were being removed when expected.
- Interaction with other factors: Because the expired entries were not removed in time, the CRL stayed large, long after the certificates from the mass revocation had expired, so every daily CRL generation remained exposed to the non-ordered query of Contributing Factor #1.
- Root Cause Analysis methodology used: 5-Whys
Lessons Learned
- What went well:
- The research into the reported CRL issue brought attention to the database certificate status and led us to find the archival bug which is being addressed in the action items as well.
- A configuration parameter had been created to address the bug with query result ordering and was available for use.
- What didn’t go well:
- Internal monitoring suites did not include detailed per certificate monitoring at large scale to alert on the intermittent issue.
- Where we got lucky:
- There was only one Issuer’s CRL with a high enough volume to be impacted by this EJBCA bug.
- OCSP responses were unaffected by this incident.
- Additional: N/A.
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Deploy an EJBCA configuration parameter to force an ordered query for certificates to be entered into the CRL | Prevent | Root Cause #1 | Compliance testing to confirm CRL generated with correct serial numbers present and not duplicated. | 2026-10-01 | Complete |
| Manually archive the expired revoked certificates in the database. | Mitigate | Root Cause # 2 | Verification in the production database comparing CRLs before and after the archive is completed. | 2026-10-30 | Ongoing |
| Create and deploy an EJBCA patch to remove the time limitation on the certificate archiving procedure. | Prevent | Root Cause # 2 | Simulations to test in non-production environments | 2026-12-04 | Ongoing |
| Add additional CRL monitoring to perform the following actions: - Monitor every published CRL for entries removed before the expiry of their certificates and for duplicate entries - Monitor every published CRL for its size and for expired entries that are not being removed | Detect | Root Causes # 1 and #2 | Simulations to confirm monitoring alerts are working in non-production environments. | 2026-12-04 | Ongoing |
Appendix
N/A.
Full Incident Report
Summary
-
CA Owner CCADB unique ID: A002038
-
Incident description:
-
On 2026-04-03, SSL.com completed a mass revocation of 1.7 million certificates. The details of this mass revocation are captured in Bug 2029230. 1,298,817 of these certificates were issued by the Cloudflare TLS Issuing ECC CA 3. By default, our software processed populating the revoked certificate information across several CRLs and in no guaranteed order. For CRLs with more than 500,000 entries, this lack of ordering caused some entries to be duplicated or missed on one CRL but to appear correctly on a later CRL. Only CRLs of the Cloudflare TLS Issuing ECC CA 3 met this condition. Our software does provide a configuration parameter to force a guaranteed order, but it is disabled by default.
-
Timeline summary:
- Non-compliance start date: 2026-04-20 16:53 UTC - First CRL published that dropped an unexpired serial number.
- Non-compliance identified date: 2026-09-30 12:56 UTC - Email reporting a CRL entry missing or duplicated issue was received.
- Non-compliance end date: 2026-10-01 15:35 UTC - New CRL for the only CA with a CRL large enough to be affected by the bug was generated and published. Non-impacted servers were patched later.
-
Relevant policies:
Baseline Requirements v2.3.0, Section 4.10.1:
"Revocation entries on a CRL or OCSP Response MUST NOT be removed until after the Expiry Date of the revoked Certificate."
-
Source of incident disclosure: Third Party Reported.
Impact
- Total number of certificates: . The only CRLs affected were from CN=Cloudflare TLS Issuing ECC CA 3. There were 4 revoked certificates across 16 incorrect removal events (a CRL omitting a previously listed, unexpired entry), and every one of them reappeared on a later CRL. In total 16 CRLs in the range of CRL #203 to #356 were affected, and none from #357 on.
- Total number of "remaining valid" certificates: All certificate serials in this incident were already revoked.
- Affected certificate types: All certificate serials incorrectly dropped from CRLs are DV TLS.
- Incident heuristic: All CRLs generated from Cloudflare TLS Issuing ECC CA 3 from 2026-04-20 16:53 until 2026-10-01 15:35.
- Was issuance stopped in response to this incident, and why or why not?: No, since there were no misissuances involved in this incident, issuance was not stopped.
- Analysis: N/A, no revocations delayed.
- Additional considerations: OCSP responses remained correct throughout. The issue only affected CRLs. In the same range of affected CRLs, 26 CRLs listed one serial number twice, always with the same revocation date and reason, so no entry carried conflicting information.
Timeline
All times are in UTC unless otherwise noted.
2020-12 - EJBCA version 7.4.3.2 was released, including a parameter described as whether EJBCA should request ordered fetching of revoked certificates when generating CRLs.
2021-03-21 - Production deployment of EJBCA version 7.4.3.2 was completed.
2026-04-03 15:57 - Mass revocation CRL for Cloudflare TLS Issuing ECC CA 3 published from bug.
2026-04-20 16:53 - First CRL published that dropped an unexpired serial number.
2026-09-30 12:56 - Email reporting a CRL entry missing and duplicated issues was received.
2026-09-30 14:26 - CRL #356 was generated, still dropping an unexpired serial number.
2026-09-30 14:52 - CA Admin Team began research into the issue with the CRL reported.
2026-09-30 22:46 - The inconsistency was confirmed from historical CRLs.
2026-09-30 23:53 - An archival bug in the software was identified as a contributory factor.
2026-10-01 01:00 - Multiple fixes were determined as options.
2026-10-01 10:16 - Fixes were reviewed and configuration change was staged.
2026-10-01 10:53 - Reporter notified.
2026-10-01 12:20 - Deployment of the corrected configuration to production began.
2026-10-01 15:21 - Deployment was completed and verified on impacted servers.
2026-10-01 15:35 - New CRL was generated and published.
2026-10-02 14:00 - Deployment started on non-impacted servers.
2026-10-02 16:40 - Deployment was completed on both impacted and non-impacted servers.
2026-10-02 20:22 - Preliminary Incident Report posted.
2026-10-16 – The full incident report is posted.
Related Incidents
| Bug | Date | Description |
|---|---|---|
| 2075655 | 2026-09-25 | Actalis: Incorrect Revocation dates in CRL entries |
| 2044788 | 2026-06-03 | Let's Encrypt: CRLs Temporarily Missing Revoked Serials |
| 2029230 | 2026-04-03 | SSL.com: Incorrect Open MPIC Lambda implementation by EJBCA ACME Service. |
| 1954861 | 2025-03-18 | Let's Encrypt: Early CRL Removal Incident |
| 1943379 | 2025-01-23 | Actalis: CRL with duplicate serial number in revokedCertificates |
Root Cause Analysis
Contributing Factor #1: EJBCA Parameter addition defaulted to off
- Description:
In EJBCA version 7.4.3.2 (released December 2020), a new parameter was introduced and described as whether EJBCA should request ordered fetching of revoked certificates when generating CRLs. Microsoft SQL Server 2016 was known to return duplicated and/or missing entries when multiple batches were read. The parameter default was set to off. SSL.com release review team was using a different SQL database provider, so this new parameter was allowed to default and was not fully tested to see if there were other potential undocumented impacts with the current database. This defect only manifested with sufficiently large CRL lists and had not occurred for our production environment until after the 1.7 million certificate mass revocation event.
- Timeline:
2020-12 - EJBCA version 7.4.3.2 was released, including a parameter described as whether EJBCA should request ordered fetching of revoked certificates when generating CRLs.
2021-03-21 - Production deployment of EJBCA version 7.4.3.2 was completed.
2026-04-03 15:57 - Mass revocation CRL for Cloudflare TLS Issuing ECC CA 3 published from bug.
2026-04-20 16:53 - First CRL published that dropped an unexpired serial number.
2026-09-30 12:56 - Email reporting a CRL entry missing and duplicated issues was received.
026-09-30 14:26 - CRL #356 was generated, still dropping an unexpired serial number.
2026-09-30 14:52 - CA Admin Team began research into the issue with the CRL reported.
2026-09-30 22:46 - The inconsistency was confirmed from historical CRLs.
2026-09-30 23:53 - An archival bug in the software was identified as a contributory factor.
2026-10-01 01:00 - Multiple fixes were determined as options.
2026-10-01 10:16 - Fixes were reviewed and configuration change was staged.
2026-10-01 12:20 – Deployment of the corrected configuration to production began.
2026-10-01 15:21 - Deployment was completed and verified on impacted servers.
2026-10-01 15:35 - New CRL was generated and published.
2026-10-02 14:00 - Deployment started on non-impacted servers.
2026-10-02 16:40 - Deployment was completed on both impacted and non-impacted servers.
- Detection: A report was received about a potential CRL compliance issue. SSL.com was not monitoring CRLs specifically for dropping or duplication of serial numbers.
- Interaction with other factors: Contributing Factor #2 kept the list of revoked certificates large enough to need more than one page of results, which is the condition under which the non-ordered query can miss or duplicate entries.
- Root Cause Analysis methodology used: 5-Whys
Contributing Factor #2: Archival status change bug delayed expired revoked certificate removal from CRL
- Description: On each full CRL generation cycle, the CA software transitions currently expired revoked certificates after one additional CRL entry to an archived status in the database. The certificate archived status prevents the CRL generation process from pulling that certificate information for the next CRL generation cycle. Because of a software bug, this archival process was limited to 10 seconds per CRL generation instead of the longer time intended. With one CRL generated per day, only around 2,000 expired certificates were archived each day, while hundreds of thousands were still pending archival after April's mass revocation (about 860,000 on 2026-10-02).
- Timeline:
2026-04-03 15:57 - Mass revocation CRL published from bug
2026-04-20 16:53 - First CRL published that dropped an unexpired serial number.
2026-09-30 14:52 - CA Admin Team began research into the issue with the CRL reported.
2026-09-30 23:53 - An archival bug in the software was identified as a contributory factor.
- Detection: Investigation into the potential CRL compliance issue reported by the third party led to finding this bug. SSL.com was not monitoring at this time for ensuring that removal of serial numbers for expired certificates were being removed when expected.
- Interaction with other factors: Because the expired entries were not removed in time, the CRL stayed large, long after the certificates from the mass revocation had expired, so every daily CRL generation remained exposed to the non-ordered query of Contributing Factor #1.
- Root Cause Analysis methodology used: 5-Whys
Lessons Learned
- What went well:
- The research into the reported CRL issue brought attention to the database certificate status and led us to find the archival bug which is being addressed in the action items as well.
- A configuration parameter had been created to address the bug with query result ordering and was available for use.
- What didn’t go well:
- Internal monitoring suites did not include detailed per certificate monitoring at large scale to alert on the intermittent issue.
- Where we got lucky:
- There was only one Issuer’s CRL with a high enough volume to be impacted by this EJBCA bug.
- OCSP responses were unaffected by this incident.
- Additional: N/A.
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Deploy an EJBCA configuration parameter to force an ordered query for certificates to be entered into the CRL | Prevent | Root Cause #1 | Compliance testing to confirm CRL generated with correct serial numbers present and not duplicated. | 2026-10-01 | Complete |
| Manually archive the expired revoked certificates in the database. | Mitigate | Root Cause # 2 | Verification in the production database comparing CRLs before and after the archive is completed. | 2026-10-30 | Ongoing |
| Create and deploy an EJBCA patch to remove the time limitation on the certificate archiving procedure. | Prevent | Root Cause # 2 | Simulations to test in non-production environments | 2026-12-04 | Ongoing |
| Add additional CRL monitoring to perform the following actions: - Monitor every published CRL for entries removed before the expiry of their certificates and for duplicate entries - Monitor every published CRL for its size and for expired entries that are not being removed | Detect | Root Causes # 1 and #2 | Simulations to confirm monitoring alerts are working in non-production environments. | 2026-12-04 | Ongoing |
Appendix
N/A.
Description
•