Open Bug 1538638 Opened 7 months ago Updated 2 months ago

Firmaprofesional: AC Firmaprofesional - INFRAESTRUCTURA insufficient serial number entropy

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: chemalogo, Assigned: chemalogo)

Details

(Whiteboard: [ca-compliance])

Attachments

(1 file, 6 obsolete files)

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36

Steps to reproduce:

Issue SSL certificates with insufficient serial number entropy

Actual results:

Issued certificates had 63 bits instead of 64 bits of entropy

Expected results:

Issued certificates sholud have had at least 64 bits

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2019-03-15 09:00 CET, after reviewing ongoing discussions and incident reports published on mozilla.dev.security.policy about 64 bit entropy for serial number generation, we started investigating our systems for possible violation of BR v.1.6.3 §7.1.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  • 2019-03-15 09:00 CET – identified relevant ongoing discussions on m.d.s.p and incident reports published.
  • 2019-03-15 09:00 CET – started investigation whether the issue affected our systems.
  • 2019-03-16 14:00 CET
    ** conclusion of investigation is that certificates issued by Firmaprofesional’s “AC Firmaprofesional - INFRAESTRUCTURA” are affected by this issue, having only 63 bits of effective entropy.
    ** Stopped of issuance of SSL certificates
    ** Development of fixes started immediately.
    Since then we have been testing the fixes under the QA environment, testing different alternatives taking into account the options of the software manufacturer. Correction is planned to be deployed in production on 2019-03-25 at 14:00 CET.

We still are carefully evaluating scenarios for the replacement of the certificates.
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped the issuance of SSL certificates on 16-March-2019.

We will update today to the latest branch 6 version of EJBCA to be able to configure a longer serial number setting in a per-CA basis.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

All SSL certificates issue. 293 are still valid.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?Identity=%25&iCAID=994

293 are still valid.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The serial numbers generation logic implemented by EJBCA, combined with EJBCA’s default settings, caused serial numbers to have less entropy than we expected. We relied on EJBCA to generate 64-bits random serial numbers, and since the serial numbers had the expected length and did look random we did not realize that they contained less than 64 bits of randomness.

That said, of course it is not a fault of PrimeKey nor EJBCA.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Serial numbers size will be increased to 128 bits, 127 bits of entropy to fulfill CA/B forum requirements.

Assignee: wthayer → chemalogo
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]
  1. Were the corrective actions deployed?
  2. What is the status of the remaining certificate? The provided list is insufficient for determining the affected certificates and their status.
Flags: needinfo?(chemalogo)

(In reply to Ryan Sleevi from comment #2)

  1. Were the corrective actions deployed?
    Yes, they were. All certificates issued from 16th of March on are certificates fully compliance
  1. What is the status of the remaining certificate? The provided list is insufficient for determining the affected certificates and their status.
    The list was a comprehensive list, so it identified all of the affected certificates.

As you can see, Firmaprofesional does not issue too many certificates but some of them are used in very critical environments, such as the regional system for electronic medical prescriptions in Catalonia or Castille (two main regions in Spain). This is why we have to assess very carefully with the customers when we can revoke a certificate, based on risk analysis, taking into account the impact of keep a 63-bit S/N entropy certificate valid versus jeopardising the customer business or a whole solution for the citizenship.

After negotiations with our customers, we have started to reissue and revoke all of the affected certificates and we will have finished revoking all of them by the 31st of July 0f 2019.

Flags: needinfo?(chemalogo)

(In reply to Ryan Sleevi from comment #2)

  1. Were the corrective actions deployed?

Yes, they were. All certificates issued from 16th of March on are certificates fully compliance

  1. What is the status of the remaining certificate? The provided list is insufficient for determining the affected certificates and their status.

The list was a comprehensive list, so it identified all of the affected certificates.
As you can see, Firmaprofesional does not issue too many certificates but some of them are used in very critical environments, such as the regional system for electronic medical prescriptions in Catalonia or Castille (two main regions in Spain). This is why we have to assess very carefully with the customers when we can revoke a certificate, based on risk analysis, taking into account the impact of keep a 63-bit S/N entropy certificate valid versus jeopardising the customer business or a whole solution for the citizenship.

After negotiations with our customers, we have started to reissue and revoke all of the affected certificates and we will have finished revoking all of them by the 31st of July 0f 2019.

The list was to a crt.sh query that changes over time. That doesn’t provide any insight into the problem or whether it was comprehensively identified -e.g. additional certificates could be logged over time. The incident report guidelines provide clear expectations.

So you revoked zero?

Flags: needinfo?(chemalogo)

(In reply to Ryan Sleevi from comment #6)

So you revoked zero?

No. We have already revoked some of them.

I can provide a specific list next Monday.

Flags: needinfo?(chemalogo)

Thanks. Please carefully review https://wiki.mozilla.org/CA/Responding_To_An_Incident in doing so.

I want to highlight that it's extremely concerning the lack of updates and the lack of progress on this. This is not acceptable going forward. It is in Firmaprofessional's best interests to comprehensively document their response, reasoning, and handling of this issue. To understand what a responsible level of documentation looks like, please examine those requirements carefully, as well as incidents from other CAs, like https://bugzilla.mozilla.org/show_bug.cgi?id=1551374 . Please also carefully examine all of the other serial entropy issues, to make sure you are not repeating mistakes that other CAs have made. This is the opportunity Firmaprofessional has to fix the perception of the handling of this incident.

Flags: needinfo?(chemalogo)
Flags: needinfo?(chemalogo)

First of all, thanks for the advice and resources.

To begin with, Firmaprofesional has failed communicating accordingly. That’s a fact that we can not deny. We should have been more proactive and accurate and we will try to fix it now with the commitment of keeping been proactive, accurate and transparent pursuant https://wiki.mozilla.org/CA/Responding_To_An_Incident.

Corrective actions:
1.- “We will update today to the latest branch 6 version of EJBCA to be able to configure a longer serial number setting in a per-CA basis.”

We updated EJBCA and serial number entropy on March, 2019, the 16th. We configured S/N to be 128 bits. Hence 127 entropy bits

2.- “We still are carefully evaluating scenarios for the replacement of the certificates.”

As said last Friday, Firmaprofesional issues certificates to be used in very critical environments, such as the regional system for electronic medical prescriptions in Catalonia or Castille (two main regions in Spain), or for professional associations of medicine doctors, pharmacists or nurses and they also use these certificates for electronic medical prescriptions, or other Public Administrations who base their electronic transactions with the citizenship on these certificates.

Due to the above, from our point of view, the situation is exceptional and, based on an internal risk analysis the harm caused by following the requirements outweighed the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement temporarily and perform a “per client” approach to revoke them all by 31st of July 2019.

The timeline and next steps are:

  1. To provide a comprehensive list of the 293 affected certificates, two times a week (Mon and Thu) to show the progress, until all of them are revoked, starting today: Incident_Report_On_Serial_number_entropy-ra_190708r01.pdf
  2. To speed up the revocation process for clients less vulnerable, with the goal of revoking all the certificates as soon as we can, always before 31st of July 2019. You will find that we have revoked some certificates today.

The list includes the following columns:

  1. Id_crt_sh: crt.sh ID. Some of them are not available. Bear in mind that we started to feed CT Logs automatically on 28th of September, 2017
  2. X509_sn: the serial number of the certificate
  3. X509_not_before: self-explanatory
  4. X509_not_after: self-explanatory
  5. Status:
    1. Expired: expired without being revoked
    2. Revoked
    3. Reissued_ (revocation_pending): already issued the new certificate but waiting for the customer to publish the new one before revoking the affected one.
    4. Still_valid
  6. Date: date of revocation or reissuance.

This issue is a finding in our last audits, already informed in CCADB:

Additional measures:

  • We have simplified the procedure to take decision on Root Program issues from a decision of the steering committee to a direct decision of the Compliance Manager.
  • We will update during July the contract template for SSL certificates to add a clause stating that Firmaprofesional will revoke a certificate with findings regarding Root Programs in no more than five days.
  • We are also evaluating the use of ACME protocol to issue certificates.

Thanks. This is a much better, and clearer, update, and I appreciate you taking the time to provide it.

With respect to the following proposed change:

We will update during July the contract template for SSL certificates to add a clause stating that Firmaprofesional will revoke a certificate with findings regarding Root Programs in no more than five days.

The Baseline Requirements require (using Version 1.6.5), in Section 9.6.1, that:

The CA represents and warrants to the Certificate Beneficiaries that, during the period when the Certificate is
valid, the CA has complied with these Requirements and its Certificate Policy and/or Certification Practice
Statement in issuing and managing the Certificate.

and

Subscriber Agreement: That, if the CA and Subscriber are not Affiliated, the Subscriber and CA are
parties to a legally valid and enforceable Subscriber Agreement that satisfies these Requirements, or,
if the CA and Subscriber are the same entity or are Affiliated, the Applicant Representative
acknowledged the Terms of Use;

and

Revocation: That the CA will revoke the Certificate for any of the reasons specified in these
Requirements.

The description makes it sound like the changes to the contract mean that there's presently no "legally valid and enforceable Subscriber Agreement that satisfies these Requirements" that "the CA will revoke the Certificate for any of the reasons specified in these Requirements". Is that a correct understanding?

I'm asking to understand if this was the result of a differing interpretation of the BR obligations, which may highlight an opportunity to improve and clarify them to be clearer, if so.

Flags: needinfo?(chemalogo)

The description makes it sound like the changes to the contract mean that there's presently no "legally valid and enforceable Subscriber Agreement that satisfies these Requirements" that "the CA will revoke the Certificate for any of the reasons specified in these Requirements". Is that a correct understanding?

In fact there is a legally valid and enforceable Subscriber Agreement that satisfies these Requirements, but in our case doing two hops: the Subscriber Agreement enforces the CPS and CP by reference, and the CPS and CP enforces the BR by reference.

After our experience, in our opinion it would be better, to bring directly to the Subscriber Agreement at least the requirements that could be more controversial in a negotiation with the client. Maybe this is worth to be discussed in the CA/B Forum public mailing list.

I'm asking to understand if this was the result of a differing interpretation of the BR obligations, which may highlight an opportunity to improve and clarify them to be clearer, if so.

We think that binding the Subscriber with the Requirements in the way we do (by reference through the CPS, and these by reference in th Subscriber Agreement) is align with the BR, but if not, then maybe we have misinterpreted them.

Flags: needinfo?(chemalogo)

Thanks. I agree with your analysis; namely, that it's acceptable, but that it would be beneficial if it was more explicit and up-front to the subscriber. I would wholly support a modification of 9.6.3 of the BRs to make it more explicit that the reasons for revocation generalize to any of the reasons in the CP/CPS, and not merely the specific reasons enumerated there, which can help draw clearer attention to the fact that revocation is required not just for actions the Subscriber may take, but for any issues the Issuing CA may have caused or encountered.

Thanks Ryan . After resolving the current issue we can propose a ballot on making more explicit and up-front to the subscriber some requirements.

Find attached an update of the report: Incident_Report_On_Serial_number_entropy-ra_190711r02.pdf

As agreed, find attached the Monday's update report: Incident_Report_On_Serial_number_entropy-ra_190715r01.pdf

To confirm: Your stated plan is to revoke all (293) certificates by 2019-07-31, correct?

The PDF you've attached cuts off a number of fields, and thus it's difficult to assess progress or what remains.

Flags: needinfo?(chemalogo)

(In reply to Ryan Sleevi from comment #18)

To confirm: Your stated plan is to revoke all (293) certificates by 2019-07-31, correct?

Yes, correct.

The PDF you've attached cuts off a number of fields, and thus it's difficult to assess progress or what remains.

I do not know to which fields you are referring to. This report has the same fields than the first one, Incident_Report_On_Serial_number_entropy-ra_190708r01.pdf , and they are:

  1. Id_crt_sh: crt.sh ID. Some of them are not available. Bear in mind that we started to feed CT Logs automatically on 28th of September, 2017
  2. X509_sn: the serial number of the certificate
  3. X509_not_before: self-explanatory
  4. X509_not_after: self-explanatory
  5. Status:
    1. Expired: expired without being revoked
    2. Revoked
    3. Reissued_ (revocation_pending): already issued the new certificate but waiting for the customer to publish the new one before revoking the affected one.
    4. Still_valid
  6. Date: date of revocation or reissuance.

Nevertheless, we can provide more information to make it easier to assess progress.

Flags: needinfo?(chemalogo)
Attachment #9076626 - Attachment is obsolete: true
Attachment #9077434 - Attachment is obsolete: true
Attachment #9078164 - Attachment is obsolete: true

As agreed, find attached the Thursday's update report: Incident_Report_On_Serial_number_entropy-ra_190718r01

Attachment #9079066 - Attachment is obsolete: true

find attached a new version of the report: Incident_Report_On_Serial_number_entropy-ra_190722r01

As agreed, find attached the Thursday's update report: Incident_Report_On_Serial_number_entropy-ra_190725r02.pdf.

Attachment #9079809 - Attachment is obsolete: true
Attachment #9080617 - Attachment is obsolete: true

As agreed, find attached the Monday's update report: Incident_Report_On_Serial_number_entropy-ra_190729r01

We are quite proud of announcing that all the affected certificates have been revoked or have expired, and some of them, when clients asked for it, have been reissued, with two exceptions:

  1. https://crt.sh/?id=1003306510 ; ridm.firmaprofesional.com
  2. https://crt.sh/?id=804284087 ; api.andbank-biid.com

These two certificates are protecting the above URLs and are being used to implement the security feature called “Certificate Pinning” in two critical mobile Apps. You can read more about Certificate Pinning here and [here]q(https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning).

The two Apps are:

  1. MobileID (https://play.google.com/store/apps/details?id=cat.bcn.idbcn&hl=en): app of the Barcelona City Council address to all the citizens in Barcelona and other municipalities to interact with the city council.
  2. AndBank (https://play.google.com/store/apps/details?id=air.com.inversis.AndbankSmartphone&hl=e): app of the leading bank in Andorra, address to their private banking customers .

This is an advanced security practice implemented in Mobile Apps that communicates with a server via an SSL connection.

We ask to have more time to revoke these two certificates, that, in any case will be revoked during August 2019.

The https://crt.sh/?id=1003306510 ; ridm.firmaprofesional.com certificate has been already revoked.

The last certificate has been revoked. With this, we finally have revoked all the affected certificates.

You need to log in before you can comment on or make changes to this bug.