1922906 - FNMT: LDAP URI in CRL Distribution Points Extension

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Amaya Espinosa]

Preliminary Incident Report

FNMT has received a notification concerning potentially mis-issued TLS server certificates that include an LDAP URI in their CRL Distribution Points (CDP) extension. We have stopped the issuance of certificates while we diagnose the problem in order to take the appropriate measures.
Additional details and a full incident report will follow in the next few days

Comment 1

[Amaya Espinosa]

Attachment: Affected Certificates F1.txt

Comment 2

[Amaya Espinosa]

Attachment: Affected Certificates F2.txt

Comment 3

[Amaya Espinosa]

Incident Report

Summary

La FNMT has received a notification regarding a set of SSL/TLS Certificates that could potentially be miss-issued in accordance with the section 7.1.2.11.2 (“CRL Distribution Points") of the CAB/Forum Baseline Requirements.
After review, it has been detected that certificates have been issued including a LDAP URI in their CRL Distribution Points (CDP) extension.

Impact

The affected certificates types were issued by the Subordinate CA Componentes Informaticos.

• Total number of affected certificates: 759
• Date of first affected certificate issued: 2024-01-23 09:51:25 UTC
• Date of last affected certificate issued: 2024-03-18 08:34:23 UTC.

Timeline

All times are UTC.

• September 15th, 2023:
  • BR for TLS 2.0.0 has become effective
• March 18th, 2024:
  • 9:00 UTC: The certificate profiles issued by CA Componentes Informaticos were updated to align with those of CA Servidores Seguros, and LDAP CRL distribution point was consequently removed.
• October 4th, 2024:
  • 16:00 UTC: Compliance staff process an email received through the incidents mailbox to report a possible error in the issuance of some certificates with respect to the CRL Distribution Points Extension field in order to check and confirm the problem.
  • 16:08 UTC: start investigating the issue, checking the set of SSL/TLS certificates indicated by e-mail and CAB/Forum Baseline Requirements.
  • 16:20 UTC: Compliance staff confirms non-compliance.
  • 16:50 UTC: After an emergency meeting with the TSP Management Committee of the FNMT, it was decided to suspend the SSL/TLS certificates issuance service.
  • 17:07 UTC: A reply was sent to the email that had been received.
  • 17:30 UTC: We initiated a preliminary investigation to determine the scope of the affected certificates.
• October 5th, 2024:
  • 7:30 UTC: we continued our comprehensive investigation and examined the following aspects:
    - checking for any additional certificates that may be affected.
    - analyzing the configuration of services, and
    - reviewing the certificate profiles of all our SSL/TLS certificates issuing CAs, to identify any potential non-compliance issues
  • 9:00 UTC: Activities are initiated to contact affected subscribers of the initially confirmed certificates to inform them that their certificates will be revoked.
  • 17:53 UTC: A preliminary incident report was posted on Bugzilla
• October 6th, 2024:
  • 11:15 UTC: we have confirmed that the problem was confined to a single SubCA.
  • 14:25 UTC: A full list of affected certificates is obtained, revealing that 759 are affected in total
  • 14:40 UTC: The FNMT TSP Management Committee is informed the scope. Due to the great impact that these revocations will have on subscribers (mainly public administration and some critical infrastructure) and their end users, FNMT will revoke all affected certificates within 5 days.
• October 07th,2024:
  • 6:15 UTC: Subscriber notifications begin in order to inform them about the incidence. We inform subscribers they need to revoke their certificates and about the procedure to obtain a new one.
  • 11:55 UTC: Once the CA’s configuration was confirmed to be correct, the issuing services were reactivated.
• October 08th, 2024:
  • FNMT‘s Contact and Support Centre are helping our customers to clarify and solve doubts about the incident.
• October 09th, 2024:
  • 12:42 UTC: The 3 certificates identified in the initial notice are revoked.
    - https://crt.sh/?serial=398C92065BDB51A565F44079E0072965
    - https://crt.sh/?serial=45E05DB4C14E58C465F43F0002C71927
    - https://crt.sh/?serial=533E8869E7DC39A365F7DE7E3688E12D
• October 11th, 2024:
  • 14:00 UTC: All affected certificates have been revoked

Root Cause Analysis

The error resulted from a combination of several factors. We present the results of the “5 whys” root cause analysis that been carried out:

Why was there a problem?

Because we were issuing certificates with LDAP URI in CRL Distribution Points Extension when it was not allowed.

Why were we issuing certificates with this field?

Because during the BR 2.0.0 requirements review process, we did not identify that ldap should be removed from the CRL distribution point extension and therefore it was not included in the profile field checklist we have for reviewing our profiles.

Why was this non-compliance not detected?

The limited time available to implement the solution for #bugzilla 1875942 in order to reissue certificates to our customers led us to focus on adjusting the certificate profiles for the detected non-compliance, coupled with the fact that the issue was not detected with the linting tools used at that time.
We assumed everything was correct and concluded the review process for BR 2.0.0. During the subsequent reviews, we focused on the redline, so the issue was not detected in the following compliance checks.

Why didn’t lint tools detected?

Although local instances of linter were deployed at the end of February to automate the checking process with bi-weekly sampling. The scripts that run linters are an ad-hoc development. During the analysis it was detected that there was a problem in the execution of the linter script and it was not being performed properly for certificates issued by CA Componentes Informáticos.

Why was the problem in linter execution not detected?

Certificate linting scripts, the non-identification of the certificate type and there was insufficient error checking implemented prevented the detection.

Lessons Learned

What went well

• The CA Servidores Seguros are not impacted by the issue.

What didn’t go well.

• The problem was detected by a third party.
• Not detected the problem in linter script execution
• Non-compliance monitoring systems did not alert us properly

Where we got lucky:

• The number of affected certificates is limited. TLS certificates issued after 18 March 2024, 9:00 UTC no longer contain the LDAP URL

Action Items

Action Item	Kind	Due Date
Update our checklist for review of certificate profiles to identify which ldap is prohibited	Prevent	Done
Improve error handling in the script	Detect	11/20/2024
Develop a detailed procedure for the use and configuration of the linter scripts	Detect	11/20/2024
Non-compliance monitoring systems review and update	Detect	11/20/2024

Appendix

The files with the affected certificates is attached. We provide the certificate serial number.

Comment 4

[Amaya Espinosa]

This is our weekly update, the pending actions is progressing as expected. We will continue to monitor this incident and provide updates.

Comment 5

[Amaya Espinosa]

Here is an update to our action items.
We are working on the review and update of the non-compliance monitoring systems, but it is running behind schedule

Action Items

Action Item	Kind	Due Date
Update our checklist for review of certificate profiles to identify which ldap is prohibited	Prevent	Done
Improve error handling in the script	Detect	Done
Develop a detailed procedure for the use and configuration of the linter scripts	Detect	Done
Non-compliance monitoring systems review and update	Detect	12/16/2024

Comment 6

[Amaya Espinosa]

Work is still in progress on the updates to the monitoring systems; however, we will need some additional time to complete them.

Action Items

Action Item	Kind	Due Date
Update our checklist for review of certificate profiles to identify which ldap is prohibited	Prevent	Done
Improve error handling in the script	Detect	Done
Develop a detailed procedure for the use and configuration of the linter scripts	Detect	Done
Non-compliance monitoring systems review and update	Detect	01/31/2025

Comment 7

[Amaya Espinosa]

Here’s an update below.
New monitors have been added to Site24x7 to enhance the detection of potential erroneous emissions.

Action Items

Action Item	Kind	Due Date
Update our checklist for review of certificate profiles to identify which ldap is prohibited	Prevent	Done
Improve error handling in the script	Detect	Done
Develop a detailed procedure for the use and configuration of the linter scripts	Detect	Done
Non-compliance monitoring systems review and update	Detect	Done

We have nothing further to update. Please let us know if there are additional questions or if we can close this case.

Comment 8

[Ben Wilson]

Thank you for your work.

Even though this is not an official bug-closure requirement until March 1, 2025, could you please provide a "Closure Summary"?

A Closure Summary should briefly:

• describe the incident, its root cause(s), and remediation;
• summarize any ongoing commitments made in response to the incident; and
• attest that all Action Items have been completed.

Here is a markdown template if needed:

Incident Report Closure Summary

• Incident Description: [Two or three sentences summarizing the incident.]
• Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
• Remediation Description: [Two or three sentences summarizing the incident's remediation.]
• Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 9

[Amaya Espinosa]

Thank you for your message. Below is the requested summary.

Incident Report Closure Summary

• Incident description: The FNMT-RCM issued SSL/TLS certificates that included an LDAP URI in the CRL Distribution Points (CDP) extension, which does not comply with section 7.1.2.11.2 (“CRL Distribution Points") of the CAB/Forum Baseline Requirements version 2.0.0, effective since September 15, 2023. A total of 759 affected certificates were identified, issued between January 23 and March 18, 2024.
• Root Cause(s) of the Incident: During the review process of version 2.0.0 of the Baseline Requirements, it was not identified that LDAP URIs had to be removed from the CDP extension; therefore, this requirement was not included in the certificate profile checklist. Additionally, the linting tools used at that time did not detect this non-compliance due to issues in the execution of the linting scripts for certificates issued by the CA Componentes Informáticos.
• Remediation Description: All affected certificates were revoked within 5 days. The certificate profile checklists were updated, error handling in the linting scripts was improved, and a guide for their use and configuration was developed.
• Commitment Summary: The FNMT is committed to improving our non-compliance monitoring system and keeping our linting tools up to date with the latest versions to proactively detect and prevent potential non-compliance issues.

All actions described in this report have been completed as indicated, and we request the closure of this report.

Comment 10

[Ben Wilson]

I will close this on or about Friday 7-Feb-2025.