1906690 - Actalis: CRL distribution point with ldap scheme

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Marco Menonna]

Attachment: ldap_affected_certs.txt

Today, upon technical checks on our certificate corpus, we found that we have issued some TLS certificates with a defect in their CRLDistributionPoints extension.
In particular, these certificates contain, in addition to a URL with "http" scheme, also a URL with "ldap" scheme which has been deprecated starting from BRs v2.0.
We will revoke these certificates within 5 days as requested by the BRs.

The list of impacted certificates is attached to this bug.

Additional details and a full incident report will follow in the next few days

Comment 1

[Marco Menonna]

Please be informed that all affected certificates were revoked this morning. A full incident report, detailing the events and measures taken, will be provided shortly.

Comment 2

[Marco Menonna]

Incident Report

Summary

While performing routine checks on our certificates with a variety of linters, we found that we issued some TLS Server certificates with an ldap-scheme URL in their CRLDistributionPoints extension after the date when non-http schemes have been deprecated (since the BRs v2.0).

Impact

The impacted certificates were issued by a dedicated technically constrained SubCA. We found a total of 28 certificates affected by the issue. As soon as we became aware of the problem we intervened on the configuration of the affected SubCA by correcting the certificate profile in order to avoid further misissuances. All affected active certificates were proptly revoked after their replacement.

Timeline

All times are CEST.

2023-09-15:

• 00:00 The CABF BRs v2.0 enter into force, prohibiting non-HTTP URLs in CRLDistributionPoints

2024-07-08:

• 09:00 While performing routine checks on our certificates with a variety of linters, we found that we issued some TLS Server certificates with an ldap-scheme URL in their CRLDistributionPoints extension.
• 09:15 We notified all the internal stakeholders of a possible compliance problem.
• 09:20 We collectively reviewed the BR 2.0 and found confirmation that ldap CDPs are no longer allowed since 2023-09-15. (This was already applied for the vast majority of TLS certificates issued under our Root)
• 09:30 We performed a thorough scan of all our SSL certificates and confirmed that the problem was confined to a single technically constrained, customer-dedicated, SubCA. (This was expected, as Actalis decided to no longer include ldap URLs in its certificates several years ago.)
• 10:00 After an emergency meeting with internal stakeholders, we immediately intervened on the affected SubCA by correcting the profile of the certificates in order to prevent further erroneous emissions.
• 10:30 We notified our customer the need to revoke the affected certificates, therefore inviting them to arrange for their replacement.
• 15:45 A preliminary Incident Report was posted on Bugzilla
• 17:40 We completed issuing replacement for all still active affected certificates, re-using the previous CSRs.

2024-07-09:

• Supported the customer in the certificate replacement process, helping them to verify its progress.

2024-07-10:

• Continued to support the customer in the certificate replacement process, helping them to verify its progress.

2024-07-11:

• 16:55 We revoked a part of the still active affected certificates.

2024-07-12:

• 10:50 We revoked all remaining still active affected certificates.
• 14:35 An update has been posted for Bug 1906690 informing that all certificates have been revoked

Root Cause Analysis

The error was caused by a combination of factors including

1. Our main pre-issuance linter (Zlint) does not yet detect this type of problem.
2. Issuance settings defined for the SubCA reflects specific customisations needs to meet client requirements contractually defined.
3. In this case, considering that the vast majority of our emissions were not affected by that specific CABF BRs v2.0 change (because we already removed the ldap URLs from our TLS certificates several years ago), a misalignment for the SubCA occurred.

Lessons Learned

• Beyond the human checks that can be (and are) carried out manually, with all the related limitations, it is better to use a multiplicity of linters to have sufficient confidence that certificates are fully compliant with the BRs. We are well aware of this, and in fact we have been using more than one linter for years, however this incident reinforces the need to further strengthen our linting process.

What went well

• The vast majority of our certificates are not impacted by the issue.
• We revoked all affected active certificates within 5 days of discovery.
• The problem was internally discovered.

What didn't go well

• Our current linting tools, while good overall, had a gap that we were unaware of.

Where we got lucky

• A very small number of certificates were impacted, all issued to the same customer.

Action Items

Action Item	Kind	Due Date
Fix configuration of TLS Server certificate profiles on the impacted SubCA removing LDAP-based CDPs	Prevent	Done
Enhancing the communication process between compliance, legal department and other internal stakeholders in order to clarify that, in the face of each update of the BR, even if not already formalized but only anticipated, it is necessary and important to examine all existing customized solutions to verify that there are no incompatibilities.	Prevent	Done
Develop and deploy a new specific lint in our proprietary linter to detect CDPs with non-HTTP schemes.	Prevent	2024-07-31
Develop a new specific lint for Zlint to detect CDPs with non-HTTP scheme, if not already in the works.	Prevent	2024-07-31
Complete integration of further linters (e.g. Pkilint) with our certificate issuance process.	Prevent	2024-10-30

Appendix

Details of affected certificates

See attached file.

Comment 3

[Marco Menonna]

This is to update that we have completed following action items:

Action Item	Kind	Due Date
Fix configuration of TLS Server certificate profiles on the impacted SubCA removing LDAP-based CDPs	Prevent	Done
Enhancing the communication process between compliance, legal department and other internal stakeholders in order to clarify that, in the face of each update of the BR, even if not already formalized but only anticipated, it is necessary and important to examine all existing customized solutions to verify that there are no incompatibilities.	Prevent	Done
Develop and deploy a new specific lint in our proprietary linter to detect CDPs with non-HTTP schemes.	Prevent	Done
Develop a new specific lint for Zlint to detect CDPs with non-HTTP scheme, if not already in the works.	Prevent	Done
Complete integration of further linters (e.g. Pkilint) with our certificate issuance process.	Prevent	2024-10-30

Comment 4

[Ben Wilson]

Please provide a status update on "Complete integration of further linters (e.g. Pkilint) with our certificate issuance process."
Thanks,
Ben

Comment 5

[Marco Menonna]

To date we have completed an initial assessment of how we could integrate it into our processes and conducted a trial; we will continue working towards actual implementation in production; at the moment we do not see any problems in meeting the anticipated deadline.

Comment 6

[Marco Menonna]

As an update on our actions, we have made good progress with the integration of 'pkilint' and we are now working on addressing a performance hit (on the overall processing flow) that has occurred as a result of its integration.

Comment 7

[Rob Stradling]

Marco, just out of interest, has Actalis looked at pkimetal at all? pkimetal integrates multiple linters, including Zlint and pkilint, and in my own testing I've observed pkimetal boost pkilint's performance by up to 20x. (See code here for details on how pkimetal achieves this).

Comment 8

[Marco Menonna]

Thank you for your suggestion. We have considered that option; however, at this time, we prefer not to commit ourselves to an external service. That said, pkimetal is an excellent initiative, and we will continue to monitor its progress as a potential option for the future

Comment 9

[Marco Menonna]

This is to inform you that pkilint is now integrated into our certificate issuance process and all action items have been completed as follows:

Action Item	Kind	Status
Fix configuration of TLS Server certificate profiles on the impacted SubCA removing LDAP-based CDPs	Prevent	Done
Enhancing the communication process between compliance, legal department and other internal stakeholders in order to clarify that, in the face of each update of the BR, even if not already formalized but only anticipated, it is necessary and important to examine all existing customized solutions to verify that there are no incompatibilities.	Prevent	Done
Develop and deploy a new specific lint in our proprietary linter to detect CDPs with non-HTTP schemes.	Prevent	Done
Develop a new specific lint for Zlint to detect CDPs with non-HTTP scheme, if not already in the works.	Prevent	Done
Complete integration of further linters (e.g. Pkilint) with our certificate issuance process.	Prevent	Done

Comment 10

[Adriano Santoni]

We do not have any further updates and would like to ask that this bug be closed.

Comment 11

[Ben Wilson]

Dear Marco,
Even though this has not yet been officially formalized as a bug-closure requirement, could you please provide a closing summary?
Thanks,
Ben

A closing summary should briefly:

• describe the incident, its root cause(s), and remediation;
• summarize any ongoing commitments made in response to the incident; and
• attest that all Action Items have been completed.

Here is a markdown template if needed:

Incident Report Closure Summary

• Incident Description: [Two or three sentences summarizing the incident.]
• Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
• Remediation Description: [Two or three sentences summarizing the incident's remediation.]
• Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 12

[Marco Menonna]

Incident Report Closure Summary

Incident Description:

During routine certificate validation checks, it was discovered that some TLS server certificates contained an ldap-scheme URL in their CRLDistributionPoints extension. This issue arose despite the deprecation of non-HTTP schemes per BRs v2.0, which took effect on September 15, 2023. A total of 28 certificates were affected, all issued by a technically constrained SubCA.

Incident Root Cause(s):

The incident resulted because our main pre-issuance linter (Zlint) did not detect the presence of non-HTTP URLs in CRLDistributionPoints and because the specific SubCA had legacy issuance settings reflecting contractual customization for clients.

Remediation Description:

Immediate actions were taken to correct the certificate profile of the affected SubCA, preventing further misissuance. All active affected certificates were proptly revoked after their replacement.
In addition, Actalis has completed all remediation actions for the in order to development and deployment of additional linting mechanisms to prevent similar issues in the future and for the integration of Pkilint into the certificate issuance process to enforce pre-issuance compliance checks.

Commitment Summary:

In addition of the completed actions our committment is to enhance and keep use a multiplicity of linters to have sufficient confidence that certificates are fully compliant with the BRs, to continuously refining automated linting mechanism to improve misissuance detection, to establish training programs to keep our teams informed of evolving BR requirements and to establish a governance framework for regular review and updates of certificate issuance processes

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 13

[Ben Wilson]

I'll close this early next week unless there are additional issues or concerns to discuss.