Closed Bug 1647121 Opened 2 years ago Closed 1 year ago

Izenpe: Failure to provide a preliminary report within 24 hours.

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: fozzie, Assigned: o-garcia)

Details

(Whiteboard: [ca-compliance])

I sent a problem report to info@izenpe.com concerning two invalid EV certificates (https://misissued.com/batch/111/) with "France" in the stateOrProvinceName field on the 19th of June at 23:45 UTC, as of the 21st of June 09:40 UTC I have yet to receieve a response from them.

Assignee: bwilson → o-garcia
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

Hi, this is the incidence report:

  • How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Email received from bugzilla, yesterday (Sunday 21th) at 22:00. We first received an email from George last Saturday 20 at 1:45, reporting a misissuance of two certificates. This email was sent to the contact of the CPS, but the contact email is really a list that is also used for some other subjects, that was incorrectly configured, and it didn't arrive to the correct people.

  • A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

We've done today (June 22th) a training to all operators doing the validations

  • Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This application we use now creates a new CSR from the information in the request form, filled by the applicant.
On the other side, we finished issuing certificates to entities outside Spain more than one year ago.

  • A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

2 certificates were affected, both issued in 2018

  • The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=1111091670
https://crt.sh/?id=1045678964

  • Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In both cases the request form included "Île-de-France" as the region of Paris (L), and we reduced the stateOrProvinceName to "France" . At that time we didn't have an application to request SSL certificates, as we have now. This application creates a new CSR from the information in the request form, filled by the applicant.

  • List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

For the problem of the contact email:

  1. We've already asked the creation of a new contact email account, just for security incidences. We'll update the CPS with this email contact. It'll be created by the end of this week. Meanwhile we've already fixed the list of contacts in that list. The CPS must be approved by the Security Committee, but we expect to have published the new version by next week.

For the problem with the certificates:

  1. We've done today (June 22th) a training to all operators doing the validations
  2. We've developed a script that sends an email to validation operators for each certificate issued. The operator will manually review that the issued certificate is the same he validated

Regards

Both affected certificates will be revoked before next Wednesday at 1:45
Regards

(In reply to Oscar Garcia from comment #1)

Email received from bugzilla, yesterday (Sunday 21th) at 22:00. We first received an email from George last Saturday 20 at 1:45, reporting a misissuance of two certificates. This email was sent to the contact of the CPS, but the contact email is really a list that is also used for some other subjects, that was incorrectly configured, and it didn't arrive to the correct people.

This is what Izenpe has disclosed in CCADB as the problem reporting address, as per https://ccadb-public.secure.force.com/mozilla/CAInformationReport

Could you highlight which CPS you believe is relevant? When I look at Website CPS, v1.6, which also discloses the same e-mail address, I can see a number of violations of the Baseline Requirements and Mozilla Root Policy requirements, and so I'm hoping I'm just looking at the wrong document. This was accessed from http://www.izenpe.com/s15-content/en/contenidos/informacion/doc_especifica/en_def/index.shtml .

In the event it's this document, from http://www.izenpe.com/s15-content/en/contenidos/informacion/descarga_certificados/es_url/index.shtml , then I see that same email address disclosed in Section 1.5.2, which is where Relying Parties should look, according to Section 4.9.3 of the Baseline Requirements.

So this doesn't feel like a very reassuring statement, and more root cause analysis is needed.

  • Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This application we use now creates a new CSR from the information in the request form, filled by the applicant.
On the other side, we finished issuing certificates to entities outside Spain more than one year ago.

It seems like this should be included in the timeline, which per Responding to An Incident, A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  • Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In both cases the request form included "Île-de-France" as the region of Paris (L), and we reduced the stateOrProvinceName to "France" . At that time we didn't have an application to request SSL certificates, as we have now. This application creates a new CSR from the information in the request form, filled by the applicant.

I'm having trouble seeing how this relates to or answers the question. It seems there are important details missing to understanding how that was reduced to the stateOrProvinceName, what the validation policies were at the time, why that reduction happened, why multiple approvers allowed that reduction, why that's been mitigated since the issuance, and why Spain wouldn't be similarly affected by a reduction.

  • List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

For the problem of the contact email:

  1. We've already asked the creation of a new contact email account, just for security incidences. We'll update the CPS with this email contact. It'll be created by the end of this week. Meanwhile we've already fixed the list of contacts in that list. The CPS must be approved by the Security Committee, but we expect to have published the new version by next week.

This is seeking to fix an issue that Izenpe should have fixed some time ago. It doesn't, however, address why it wasn't fixed, or how similar issues like this are being prevented in the future.

The opportunity of incident reports isn't just to correct mistakes, but to learn from them, in order to prevent them. It's not clear that there's been a thoughtful examination about how this happened, why it was overlooked (especially given the broad attention to this matter has received from other CAs), and what controls exist going forward to prevent future mistakes.

For the problem with the certificates:

  1. We've done today (June 22th) a training to all operators doing the validations
  2. We've developed a script that sends an email to validation operators for each certificate issued. The operator will manually review that the issued certificate is the same he validated

This doesn't meet the standard of these incident reports. Saying "We've done more training" provides no new information to the community, nor helps any CA learn from this. What was the old training? What was the new training? Why would the new training address this issue? What steps are there to make sure the training is correct and sufficient? What steps are there to prevent future training failures?

The goal would be to allow any other CA within the Mozilla Program to independently evaluate this report and adopt similar solutions. A solution of "Don't make the same mistake" doesn't really mitigate the systemic issues at play here, and that's the goal.

Flags: needinfo?(o-garcia)

Our current CPS is http://www.izenpe.com/contenidos/informacion/descarga_certificados/es_url/adjuntos/DOC_P_CPS_v6.3.pdf. The contact is info@izenpe.com, as indicated in https://ccadb-public.secure.force.com/mozilla/CAInformationReport, which is really a distribution list. As we indicated in comment #1 the problem has been that the list didn't include the correct persons, and that was the reason to generate the preliminary report of those two misissued certificates. Although it's not the official way, we have a 24x7 support center available in many ways, as you can see in https://www.izenpe.com/s15-content/en/contenidos/informacion/cau_izenpe/en_cau/cau_izenpe.html, so it's difficult that someone can't notify a security incidence.
Mitigation actions we're going to undertake:

  • Create a specific distribution list to notify security incidences: it'll be created this week. After that we must include into a new CPS version, that must be approved by the Security Committee. We expect to have it published by the end of next week.
  • We'll include a test of this distribution list in the tests included in our Contingency Plan. That'll be done by next week, and applied in tests planned by next month

About the problem of incorrect stateOrProvinceName in those two certificates, two years ago we didn't have an authomatic process to check Locality and Province names, and two operators verified that "Île-de-France" was the correct region, but at the time of issuance the third operator (the one who issued the certificate) reduced the region name to "France". This is something difficult to happen now, because:

  • Applicant must fill a form where localities and regions are fixed (list got from the Spanish Ministry)
  • The CSR is re-created from the information of the request form
  • We don't issue SSL certificates to entities out of Spain
  • We've developed a script that sends an email to validation operators for each certificate issued. The operator will manually review that the issued certificate is the same he validated
  • Our validation team is really small, and members are continuously trained
  • As specified by BRs, every term 3% of all issued certificates are reviewed

In the medium term we plan to integrate our web application into our PKI system, but we don't have a date yet.

Flags: needinfo?(o-garcia)

Please file a new, fully complete and compliant final incident report. Here are the instructions:
https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Flags: needinfo?(o-garcia)

Created a new bug with a complete report for the issue of incorrect stateOrProvinceName -> https://bugzilla.mozilla.org/show_bug.cgi?id=1653284

About the issue of providing a preliminary report within 24 hours, today (July 16th) we have updated our CPS with the new incidencias@izenpe.eus contact. This contact is a distribution list, to not depend on just one person. We've already notified to CCADB.

Flags: needinfo?(o-garcia)

I intend to close this bug on or about 14-Sept-2020.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.