1875942 - FNMT: Certificates issued included Policy qualifiers other than id-qt-cps

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Amaya Espinosa]

On January 22nd, FNMT has realized that we has issued a number of TLS certificates since September 15th 2023 included Policy Qualifiers other than the id-qt-cps. This is not compliance BR 7.1.2.7.9

The types of certificates affected are as follows:

• FNMT’s OID: 1.3.6.1.4.1.5734.3.16.1.1 - QEVCP-w (AC Servidores Seguros Tipo1)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.16.1.2- QEVCP-w (AC Servidores Seguros Tipo1)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.16.1.3 - QEVCP-w (AC Servidores Seguros Tipo1)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.16.2.1 – OVCP (AC Servidores Seguros Tipo2)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.16.2.2 – OVCP (AC Servidores Seguros Tipo2)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.16.2.3 – OVCP (AC Servidores Seguros Tipo2)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.9.17 – OVCP (AC Componentes Informáticos)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.9.18 – OVCP (AC Componentes Informáticos)
• FNMT’s OID: 1.3.6.1.4.1.5734.3.9.21 - QNCP-w (AC Componentes Informáticos)

We have already suspended the service of issuing this type of certificates until we solve the incident.
This is a preliminary report. We will provide through this bug a full report of the incident asap.

Comment 1

[Amaya Espinosa]

Incident Report

Summary

On January 22nd, 2024 FNTM detects an alert from our monitoring systems of a possible regulatory non-compliance. We become aware that since September 15th, 2023 it has issued certificates that included Policy Qualifiers other than id-qt-cps.

More specifically, these certificates have the id-qt-unotice type qualifier (OID: 1.3.6.1.5.5.7.7.2.2). It contravenes BR 7.1.2.7.9 of the CAB/Forum Baseline Requirements v 2.0.0.

The affected certificate types are issued by the following SubCAs:

• AC Componentes Informáticos
• AC Servidores Seguros Tipo1
• AC Servidores Seguros Tipo2

Impact

The total number of affected certificates was 712, issued between 2023-09-15 8:01:18 UTC and 2024-01-22 10:01:46 UTC. They are distributed as follows:

• Affected certificates from "AC Componentes"
  • Total number of affected certificates: 394
  • Date of first affected certificate issued: 2023-09-15 08:01:18 UTC
  • Date of last affected certificate issued: 2024-01-22 10:01:46 UTC
• Affected certificates from "AC Servidores Seguros Tipo1"
  • Total number of affected certificates: 260
  • Date of first affected certificate issued: 2023-09-15 10:52:31 UTC
  • Date of last affected certificate issued: 2024-01-22 9:29:46 UTC
• Affected certificates from "AC Servidores Seguros Tipo2"
  • Total number of affected certificates: 58
  • Date of first affected certificate issued: 2023-09-18 06:30:59
  • Date of last affected certificate issued: 2024-01-18 06:52:12

Timeline

All times are UTC.

• September 15th, 2023: Effective date of Certificate Profiles Update defined in the Baseline Requirements v2.0.0.
• January 22nd, 2023:
  -19:15 UTC: The technical Area detected an alert displayed on the monitoring system.
  -19:20 UTC: Compliance staff is notified in order to check and confirm the issue.
  -19:50 UTC: The non-compliance issue is confirmed by Compliance staff.
  -19:54 UTC: The FNMT TSP Management Committee is informed and decides to suspend the TLS certificates issuance service of affected SubCAs.
  -20:00 UTC: We begin a preliminary investigation the scope of affected certificates.
• January 23th,2023:
  -8:17 UTC: Certificate profiles are updated and published in our website (“AC Componentes Informaticos Profiles v.1.21”, “AC Servidores Seguros Tipo1 Profiles v.1.7” and ““AC Servidores Seguros Tipo2 Profiles v.1.7”)
  -8:20 UTC: Reactivation of issuance services.
  -9:03 UTC: Our initial investigation reveals there are 712 affected certificates. Subscriber notifications begins in order to inform them about the incidence. We inform subscribers they need to revoke their certificates and about the procedure to obtain a new one. Due to the great impact that these revocations will have on subscribers (mainly public administration and some critical infrastructure) and their end users, FNMT will revoke all affected certificates within 5 days.
• January 24th, 2023: FNMT ‘s Contact and Support Centre are helping our customers to clarify and solve doubts about the incident.

Root Cause Analysis

Following our protocol for the compliance surveillance of all applicable technical requirements and regulatory changes, our external Compliance Office detected the BR v2.0.0 release and we reviewed it. Due to a misinterpretation in the attributes defined for the subscriber certificate profile, we did not identify this discrepancy with our profiles and the need to update them deleting the id-qt-unotice type qualifier.

Lessons Learned

What went well

• The FNMT's continuous monitoring system detected compliance issue.
• Communication with the customer has been fluid and transparent, our Contact and Support Center has been in permanent contact with all our affected customers.
• The issues identified were resolved speedily.

What didn’t go well

• The review of the new BR v2.0 requirements, must be performed by two members of the compliance team, but in this case, due to the restructuring of the compliance area, it was only performed by one person.
• We trusted that the linter tools that we use are up to date with the latest requirements.

Where we got lucky

Action Items

Action Item	Kind	Due Date
An internal control shall be established to ensure that peer review of all applicable requirements is performed in any case	Prevent	Done
Monthly control of the versions and updates of linter tools in order to know and be able to assess risks	Mitigate	01/02/2024
Evaluate the use of Digicert/pkilint tool for additional sampling of issued certificates	Detect	01/02/2024

Appendix

The file with the affected certificates is attached. We provide the certificate serial number.

This bug will be updated to confirm revocation of all affected certificates

Comment 2

[Amaya Espinosa]

Attachment: List of Certificates affected AC Componentes.txt

Comment 3

[Amaya Espinosa]

Attachment: List of Certificates affected AC Servidores Tipo1.txt

Comment 4

[Amaya Espinosa]

Attachment: List of Certificates affected AC Servidores Tipo2.txt

Comment 5

[Amaya Espinosa]

All affected certificates have been revoked within 5 days

Comment 6

[Ben Wilson]

Can you please provide a much more thorough root cause analysis? https://en.wikipedia.org/wiki/Root_cause_analysis The statement, "Due to a misinterpretation in the attributes defined for the subscriber certificate profile, we did not identify this discrepancy with our profiles and the need to update them deleting the id-qt-unotice type qualifier.", is insufficient and does not constitute a root cause analysis.

Comment 7

[Amaya Espinosa]

Thanks for your feedback, Ben. Let me provide more information on root cause and update actions items.

Root Cause Analysis

The Ceres Department's compliance team is responsible for overseeing compliance with applicable requirements, such as BRs, EV Guidelines, eIDAS, ETSIs, RFCs, Root Programs and corresponding national regulations.

In addition, we have an external Compliance Office that, on a monthly basis, reviews these regulations and generates a report pointing out any updates or variations in the baseline requirements.

In this particular case, this report highlighted the publication of BR 2.0.0, with the required action of reviewing and verifying profiles. However, during the SC62 review regarding our profiles, it was not identified that the change affected the "certificatePolicies" extension.

The main cause of the problem was the lack of change detection during the profile review by the Compliance team. Although we have internal tools, such as a compliance matrix, that facilitate these regulatory reviews. The absence of a specific "checklist" for the review of profile fields/attributes has been identified.

Normally, this review is performed in pairs, but on this occasion it was performed by a single person, which impeded the change from being detected in the second review.

The root cause, failure in the profile review process and reliance on a single person were the reasons for overlooking the change in the "certificatePolicies" extension.

In addition, not having an updated version of the linters prevented early detection of this error. In our monitoring of crt.sh (via zlint, cablint and x509lint), we lack control over updates. Therefore, we have approved the deployment of local instances of zlint and pkilint, which will allow us to have version and update control.

These tools will be used to implement an automated testing process using bi-weekly sampling, which will serve as a complementary measure to our current monitoring.

Action Items

Action Item	Kind	Due Date
An internal control shall be established to ensure that peer review of all applicable requirements is performed in any case	Prevent	Done
Monthly control of the versions and updates of linter tools in order to know and be able to assess risks	Mitigate	Done
Checklist for review of certificate profiles	Prevent	09/02/2024
Tools for additional sampling of certificates issued against pkilint and zlint	Detect	27/02/2024

Comment 8

[Amaya Espinosa]

We are working on the action items.
The checklist for the review of the certificate profiles will be slightly delayed. Updating the action items.

Action Items

Action Item	Kind	Due Date
An internal control shall be established to ensure that peer review of all applicable requirements is performed in any case	Prevent	Done
Monthly control of the versions and updates of linter tools in order to know and be able to assess risks	Mitigate	Done
Checklist for review of certificate profiles	Prevent	2024-02-27
Tools for additional sampling of certificates issued against pkilint and zlint	Detect	2024-02-27

Comment 9

[Amaya Espinosa]

We have completed the implementation of all our AIs.

We have defined a specific "checklist" for the review of profile fields/attributes and we have started using the tools for additional bi-weekly sampling of certificates issued against pkilint and zlin.

Comment 10

[Ben Wilson]

Hello Amaya,
There has been no activity in this bug for some time. Have you found success with your remediation items? Can this now be closed?
Thanks,
Ben

Comment 11

[Amaya Espinosa]

Hi Ben,
Yes, we have. Please, we would like to request that this bug be closed now.
Thanks.

Comment 12

[Ben Wilson]

I will close this on or about Wed. 28-Aug-2024.