Closed Bug 1651637 Opened 5 years ago Closed 4 years ago

Firmaprofesional: Failure to revoke ICAs within 7 days: OCSP EKU

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mprieto, Assigned: mprieto)

Details

(Whiteboard: [ca-compliance] [ca-revocation-delay])

Attachments

(1 file)

Firmaprofesional_new_ICA_certs_new_keys.zip
7.09 KB, application/x-zip-compressed
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36

We acknowledge that Firmaprofesional will not be able to revoke ICAs disclosed in the Incident Report, in https://bugzilla.mozilla.org/show_bug.cgi?id=1649943, within 7 days.
We will update the incident report in https://bugzilla.mozilla.org/show_bug.cgi?id=1649943 in a few days, as we are continuing our investigations.

Type: defect → task
Assignee: bwilson → mprieto
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [delayed-revocation-ca]
Whiteboard: [ca-compliance] [delayed-revocation-ca] → [ca-compliance] [delayed-revocation-ca] Next update 22-July-2020

Repasting Comment 7 from Bug 1649943 here, which is relevant:

We maintain dates for the issuance of ICA certificate with the same keys and without the EKU OCSP Signing, which we attach, and for the revocation of current certificates with EKU OCSP Signing.

We will ask this week to add to OneCRL the current ICA certificates and the new ones, except INFRAESTRUCTURA, as a mitigation means for OneCRL clients.

Issuance of new certificates with new keys: estimated week of July 27, 2020.

Once the new certificates appear in the Spanish TSL (this is a requirement for most of our clients who also rely in the TSL), we will begin the rollover of the still valid leaf certificates to the new ICA certificates with new key pairs and when the rollover is completed: revocation of the ICA certificates with the same keys and key destruction ceremony.

Clarifications:

  • Firmaprofesional has exclusive control of the private keys of all the CAs that are part of its hierarchy and thus declares it in its CPS
  • None of the affected ICA certificates of Firmaprofesional has the keyUsage digitalSignature
  • Only the ICA INFRASTRUCTURE issues SSL certificates.
  • The new INFRASTRUCTURE certificate with new keys will be the first in a series of CA Intermedia (ICA) certificates aimed at issuing secure server certificates that will follow this scheme:
    ** The ICA certificate will last no more than 3 years.
    ** ICA certificates will be renewed annually, with new keys
    ** For easier identification of the certificates, the year of issue will be added to the certificate, for example "AC FIRMAPROFESIONAL - Secure Web 2020"

Repasting Comment 7 from Bug 1649943 here, which is relevant:

We maintain dates for the issuance of ICA certificate with the same keys and without the EKU OCSP Signing, which we attach, and for the revocation of current certificates with EKU OCSP Signing.

We will ask this week to add to OneCRL the current ICA certificates and the new ones, except INFRAESTRUCTURA, as a mitigation means for OneCRL clients.

Issuance of new certificates with new keys: estimated week of July 27, 2020.

Trying to make sure I understand this:

  • You've issued new certificates with the old keys, without the EKU
  • You've requested Mozilla to add to OneCRL those with the EKU
  • You'll create new intermediates with new keys on 2020-07-27
  • At some point, time unknown, you'll perform the actual revocation (see below)

Once the new certificates appear in the Spanish TSL (this is a requirement for most of our clients who also rely in the TSL), we will begin the rollover of the still valid leaf certificates to the new ICA certificates with new key pairs and when the rollover is completed: revocation of the ICA certificates with the same keys and key destruction ceremony.

Why do these certificates need to appear in the TSL? Is your root not in the TSL? Can you provide a reference to the TSL you're specifically mentioning?

It sounds like this has identified a potential root cause for Firmaprofesional's failure to abide by the BR timeframes: a dependency on an external system limits the ability to replace intermediates as required by the Baseline Requirements. It would be important to understand and see how this limitation is being addressed.

This is also a very unclear timeline. This could be something measured in days, or could be measured in months, and I'm concerned that simply saying "This is an external dependency we can't influence" would be a move in the wrong direction. Certainly, more data is needed here to understand the current architecture of Firmaprofessional, the relevant understanding about the how and why the TSL affects intermediates' timelines, the expected resolution (and what steps will be taken if it doesn't happen as expected), and a timeline for this.

Clarifications:

  • Firmaprofesional has exclusive control of the private keys of all the CAs that are part of its hierarchy and thus declares it in its CPS

How is this relevant to Relying Parties? I would argue it's not, and it doesn't do anything to mitigate this issue, without a discussion about how these keys are protected and managed specific to the risk being discussed here.

  • None of the affected ICA certificates of Firmaprofesional has the keyUsage digitalSignature

As has been mentioned elsewhere, this is irrelevant for virtually all clients, including Mozilla. I understand that individuals at Firmaprofesional believe this should address any risk, but it's an interpretation not shared by any PKI expert that has implemented a certificate verification library, apparently, so this seems a very ad-hoc and problematic interpretation if it's being suggested as a mitigation.

  • Only the ICA INFRASTRUCTURE issues SSL certificates.

How or why is this relevant?

  • The new INFRASTRUCTURE certificate with new keys will be the first in a series of CA Intermedia (ICA) certificates aimed at issuing secure server certificates that will follow this scheme:
    ** The ICA certificate will last no more than 3 years.
    ** ICA certificates will be renewed annually, with new keys
    ** For easier identification of the certificates, the year of issue will be added to the certificate, for example "AC FIRMAPROFESIONAL - Secure Web 2020"

I think this is quite encouraging, and it seems that this is part of an (unspecified) root cause, which is that presently, rotating intermediates is hard, and regularly rotating intermediates is being seen as a solution.

However, this only applies to the SSL/TLS issuance, and thus doesn't really mitigate the systemic risk, which in this case, is posed by CAs not issuing SSL/TLS but that can influence the security of those that do.

Have you re-examined how you administer and segment your certificate hierarchy? Consider this discussion with PKIoverheid that looks to more systemically lay out risks and long-term mitigations.

Flags: needinfo?(mprieto)

Trying to make sure I understand this [...]

You are right.

Also find our query to add certificates to OneCRL, as a palliative measure here: https://bugzilla.mozilla.org/show_bug.cgi?id=1655074

Why do these certificates need to appear in the TSL? Is your root not in the TSL? Can you provide a reference to the TSL you're specifically mentioning?

This is the same for other European CAs issuing Qualified Certificates pursuant eIDAS, such as PKIoverheid. They cited EUTL, European Trust List. But to be more specific the EUTL is a list of TSL (Trusted Services List), pursuant ETSI TS 119 612 V2.2.1 (2016-04); Electronic Signatures and Infrastructures (ESI); Trusted Lists (https://www.etsi.org/deliver/etsi_ts/119600_119699/119612/02.02.01_60/ts_119612v020201p.pdf).

These Trusted List do not include root CA certificates but ICAs and bound to them, the Trust Services Associated with these ICAs.
For example PKIoverheid are issuing Qualified Certificates for Electronic Signature and this is why they mention the EUTL and S/MIME (“For the S/MIME ICAs a timeline is being investigated, as these also depend on EUTL listing.”)

Firmaprofesional, and other CAs, such as QuoVadis, not only issues Qualified Certificates for Electronic Signature, but also Qualified Certificates for Electronic Seals, Qualified Website Authentication Certificates (the European way to name EV certificates+some extra requirements) and Qualified Time Stamping Authority. And this is why we have to deal with extremely thorough audits every year.

This all is absolutely transparent. You can get the EUTL here:

You can also get the Spanish TSL here (https://sede.serviciosmin.gob.es/Prestadores/Paginas/Inicio.aspx):

Why European CAs issuing Qualified Certificates (Qualified Trust Services Providers) have to wait until the corresponding ICA certificate is included in the TSL? Because we are not allowed to issue leaf certificates with this ICA until its certificate appears in the TSL. So we can not begin a rollover until the new ICA certificates are included in the TSL.

Maybe this way it’s a little bit slower but this also means that not only CA/B Forum but also European Member States and the corresponding Supervisory Body are watching us, even for the issuance of web server certificates.

Have you re-examined how you administer and segment your certificate hierarchy? Consider this discussion with PKIoverheid that looks to more systemically lay out risks and long-term mitigations.

Thanks for sharing the suggestions and discussion for future PKI designs. In fact, the new ICA AC Firmaprofesional - Secure Web 2020 is intended to issue only SSL certificates and, as PKIoverheid, we are also considering other designs such as separating use cases per root.

As planned, Yesterday we issued the new certificates with new CN and new key pair.

Find them attached.

Whiteboard: [ca-compliance] [delayed-revocation-ca] Next update 22-July-2020 → [ca-compliance] [delayed-revocation-ca]

As planned, on August 6, the following certificates with offending EKU were revoked:

Find the ARL at: http://crl.firmaprofesional.com/fproot.crl

Once we issued the new ICAs, weeks ago, with the same keys and with a new key pair, we sent them to the Spanish Supervisory Body to be included in the Spanish TSL. Unfortunately a new version of the Spanish TSL has not been yet released.

We are in touch with the Spanish SB , trying to speed this process up."

Flags: needinfo?(mprieto)

Today the Spanish TSL has been updated, unfortunately without including the new ICA certificates issued by Firmaprofesional.

We are in touch with the Spanish Supervisory Body trying to know when the TSL will be updated with Firmaprofesional ICAs certificates.

There are no more updates to date.

Today the Spanish TSL has been updated including the necessary ICAs of the Firmaprofesional root that enable us to begin the rollover of the aprox. 6.500 end-entity certificates, including TSA certificates, Qualified Electronic Signature certificates, QWACS, SSL-OV, Qualified Electronic Seal certificates among other.

Our plan is to finish the roll-over in no more than three months with the aim of finalizing it within this year.

We had already started all the communications with the affected clients, so the rollover can begin just Today.

We will inform periodically about the progress pf the rollover.

We keep on track.

We have begun issuing SSL certificates from ICA "AC Firmaprofesional - Secure Web 2020" and reissuing and revoking the certificates issued from ICA "AC Firmaprofesional - INFRAESTRUCTURA". A side effect is that all those 2-years validity will be reissued as 1.year validity certificates, against clients convenience but in favor of security.

The new "FIRMAPROFESIONAL CLOUD QUALIFIED TSU - 2020" TSA certificate is also within the Sapnish TSL and we will begin to serve time-stamp tokens the following 212st of October. We needed to give time to customers to propagate the new QTSA certificates within their systems.

We still keep our aim of finalizing it within this year.

We still keep on track.

We initiated the issuance of Qualified timestamps last week without problems.

Finally Today we have revoked the following ICA certificates:

Find the ARL at: http://crl.firmaprofesional.com/fproot.crl

Still valid are:

This certificate will be superseded by https://crt.sh/?id=3226381768: "AC Firmaprofesional - Secure Web 2020"

These two certificates will be superseded by https://crt.sh/?caid=180578: "SIGNE Autoridad de Certificacion - 2020"

After the revocation of all of them we will perform the key destruction procedure.

We still keep our aim of finalizing it within this year.

We keep on track.

We have ended the renewal of all SSL certificates with ICA "AC Firmaprofesional - Secure Web 2020" and still revoking the certificates issued from ICA "AC Firmaprofesional - INFRAESTRUCTURA", since some clients have not changed the old certificates with the new ones.

We have begun to serve time-stamp tokens since 21st of October and some customers have asked for time to re-seal time-stamped documents.

We still keep our aim of finalizing it within this year.

We keep on track.

The change of the old SSL certificates with the new ones is still in process for some late customers.

The re-sealed of time-stamped documents is still in process.

We still keep our aim of finalizing it within this year.

Whiteboard: [ca-compliance] [delayed-revocation-ca] → [ca-compliance] [delayed-revocation-ca] Next update 2021-01-04

Today we have just revoked the last end entity certificate, except the OCSP ones of each affected CA, that is:

  1. https://crt.sh/?id=3149354218: “AC Firmaprofesional - INFRAESTRUCTURA”. Certificate w/o offending EKU but with same key pair than https://crt.sh/?q=10601239: “AC Firmaprofesional - INFRAESTRUCTURA” (with offending EKU and revoked Today)
  2. https://crt.sh/?q=240192053: “SIGNE Autoridad de Certificacion” with offending EKU
  3. https://crt.sh/?id=3149354217: “SIGNE Autoridad de Certificacion” w/o offending EKU but with same key pair than the one above

Next steps:

  1. next Monday 28th of December we will:
    1. revoke the remaining OCSP certificates
    2. issue a lastCRL according with ETSI ETSI EN 319 411-2 V2.2.2 (2018-04) for each CA
    3. revoke the afore mentioned ICA certificates.
  2. End January: due to the current dates and the conditions in Spain because of COVID it is not possible to find an external auditor to come to our data center to witness the key destruction ceremony. We will postpone the key destruction to end of January 2021.

Today we have revoked the following certificates:

  1. https://crt.sh/?id=3149354218: “AC Firmaprofesional - INFRAESTRUCTURA”. Certificate w/o offending EKU but with same key pair than https://crt.sh/?q=10601239: “AC Firmaprofesional - INFRAESTRUCTURA” (with offending EKU and revoked Today)
  2. https://crt.sh/?q=240192053: “SIGNE Autoridad de Certificacion” with offending EKU
  3. https://crt.sh/?id=3149354217: “SIGNE Autoridad de Certificacion” w/o offending EKU but with same key pair than the one above

All the ICA certificates with the offending EKU OCSPSigning have been revoked, so did the ICA certificates without the offending EKU but with the same key pair.

Due to scheduling problems we have had to move the key destruction ceremony to the first half on February.

Last Friday, 5th of February Firmaprofesional performed the key destruction ceremony.

The process was as follows:

  • 4 people, face to face, in our, main datacenter:
    • Security manager, Firmaprofesional
    • CA Administrator, Firmaprofesional
    • Firmaprofesional relevant witness, Firmaprofesional
    • External auditor
  • The destroyed keys had the following SKI:
6215abb5b30879a587fe80d922f08efc8f11fd79
7b20b90938e580e2d7da1de6942de9cbcb7bf960
35e78be37a9e08347c6ad3ba623801ff8297649b
90ad1a31fd47456d32444318859764c738ae6da8
  • Backups were identified so the auditor can check its location and that copies of the private keys are destroyed
  • The verification of the execution environment is carried out by the auditor.
  • A secure erasure of the CA's private key was carried out, using the functions provided by the hardware cryptographic device (HSM, in this case, nCipher), so that the rest of the keys managed by the device are not affected.
    rocs> list keys
        key_pkcs11_****
        ...
        
    rm key_pkcs11_****
    rm key_...
    ...
  • The removal of private keys was checked, listing the keys again, so the auditor could check that the identified keys were erased.
  • All backup copies of the CA's private key, previously identified and shown to the auditor, were safely deleted.

All of the above was collected in a key destruction ceremony minute, signed by the participants, and which, together with evidence of physical access to the datacenter and that which the eIDAS auditor may request, will be delivered to him in the next eIDAS audit.

Please provide an update. Thanks.

Flags: needinfo?(chemalogo)
Whiteboard: [ca-compliance] [delayed-revocation-ca] Next update 2021-01-04 → [ca-compliance] [delayed-revocation-ca] Next update 2021-04-23

All the required steps have been done, including key destruction.

Flags: needinfo?(chemalogo)

Was this key destruction ceremony witnessed by an independent auditor, and could you provide us with the audit report (if such issued)?

Last Friday, 5th of February Firmaprofesional performed the key destruction ceremony.

The process was as follows:

4 people, face to face, in our, main datacenter:
Security manager, Firmaprofesional
CA Administrator, Firmaprofesional
Firmaprofesional relevant witness, Firmaprofesional
External auditor

[...]

All of the above was collected in a key destruction ceremony minute

Similarly to https://bugzilla.mozilla.org/show_bug.cgi?id=1652603#c14 (closed bug), the minute contains sensitive data that cannot be disclosed.
This is why we added:

and which, together with evidence of physical access to the datacenter and that which the eIDAS auditor may request, will be delivered to him in the next eIDAS audit

Our eIDAS audit is expected to be released by first half of June.

Are there any reasons why this bug cannot be closed now? I will call this back up for review to be closed on next Wednesday, 12-May-2021. Thanks.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-ca] Next update 2021-04-23 → [ca-compliance] [ca-revocation-delay]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: