Closed Bug 1653284 Opened 4 years ago Closed 3 years ago

Izenpe: incorrect value in stateOrProvinceName

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: o-garcia, Assigned: o-garcia)

Details

(Whiteboard: [ca-compliance] [ev-misissuance])

This bug is a continuation of https://bugzilla.mozilla.org/show_bug.cgi?id=1647121

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Investigation made during the analysis for https://bugzilla.mozilla.org/show_bug.cgi?id=1647121. We were reported in that bug that the following certificates had an incorrect stateOrProvinceName value:
https://crt.sh/?id=1111091670
https://crt.sh/?id=1045678964
During the analysis we have found that there are 16 live certificates with an incorrect stateOrProvinceName

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

June 22th organized training for all validation personnel and RA operators
June 22th developed and put in production a script that sends an email to validation operators for each certificate issued. The validation agent will manually review that the issued certificate is the same he validated
June 24th reported certificates were revoked, and Izenpe began an investigation
July 7th Required all customers to use the web application to request certificates
July 15th Found that there are 16 affected certificates. Izenpe will revoke all of them before the following 5 days

  1. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Since last July 7th all requests go through our web application, where localities and provinces are taken from the Spanish Ministry official list, so there’s no risk to put a province by hand.

  1. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

16 certificates with stateOrProvinceName as “FRANCE”, instead of “Île-de-France”

  1. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=976518444
https://crt.sh/?id=955588634
https://crt.sh/?id=1154593917
https://crt.sh/?id=1130629695
https://crt.sh/?id=923040929
https://crt.sh/?id=967383323
https://crt.sh/?id=1154423592
https://crt.sh/?id=962032211
https://crt.sh/?id=1137176095
https://crt.sh/?id=1137206205
https://crt.sh/?id=1109849921
https://crt.sh/?id=1231234643
https://crt.sh/?id=1279443215

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In all cases the request form included "Île-de-France" as the region of Paris (L), and we reduced the stateOrProvinceName to "France".
Fifteen months ago, we didn't have an automatic process to check Locality and Province names, and two operators verified that "Île-de-France" was the correct region, but at the time of issuance the third operator (the one who issued the certificate) reduced the region name to "France".

  1. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
  • Require all customers to use the web application to request certificates -> done (July 7th)
  • Applicant must fill a form where localities and regions are fixed (list got from the Spanish Ministry) -> done (June 21th)
  • We don’t issue certificates to entities out of Spain -> done (November 15th, 2019)
  • The CSR uploaded by the applicant is re-created from the information of the request form. We only get the public key from the CSR sent by the applicant, we create a new CSR with this public key, and the validated information -> done
  • We've developed a script that sends an email to validation operators for each certificate issued. The operator will manually review that the issued certificate is the same he validated -> done (June 21th)
  • By the end of September we plan to integrate our web application into our PKI system to reduce the risk of manual errors to the minimum.

For your existing certificates, have you only checked for disrepencies with "France" in the province name or have you checked against something like ISO 3166-2?

Assignee: bwilson → o-garcia
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

What are your plans regarding the 13 other certificates identified under 5 above? Can you supplement your response to 7 (Steps to Resolve), to explain what will be done with those customers and their certificates? Thanks.

Flags: needinfo?(o-garcia)

(In reply to george from comment #1)

For your existing certificates, have you only checked for disrepencies with "France" in the province name or have you checked against something like ISO 3166-2?

We don't issue to many certificates, so it hasn't been difficult to review the stateOrProvinceName of all our certificates. In any case, as indicated above we have replaced the form in our application from a text box by a list of fixed official values, got from the Spanish Ministry. And at this moment we don't issue SSL certificates to entities out of Spain.

Flags: needinfo?(o-garcia)

(In reply to Ben Wilson from comment #2)

What are your plans regarding the 13 other certificates identified under 5 above? Can you supplement your response to 7 (Steps to Resolve), to explain what will be done with those customers and their certificates? Thanks.

All of those certificates will be revoked today

Yesterday (July 7th) were revoked all affected certificates

Sorry, obviously the date was incorrect. Yesterday (July 21th) all certificates were revoked

You opened this incident on the 16th July 2020 at 15:18 UTC the revocation date for https://crt.sh/?id=976518444 seems to say 22nd July 2020 at 07:23 UTC. So either these certificates were not revoked within 5 days or there seems to be a 16 hour delay with your OCSP responder, can you clarify what time you actually revoked these certificates?

The revocation date is correct, the delay of 16 hours have been due to an internal communication mistake

What communication mistake occurred? If I requested an OCSP response for https://crt.sh/?id=976518444 before 2020-07-22 07:23:14 UTC would it have shown as Good? All of the certificates mentioned above seem to be revoked within 10 minutes after 07:23:14.

Flags: needinfo?(o-garcia)

There's no problem with the OCSP response, it's OK. All certificates were revoked at the time indicated by the OCSP answer, you can check it with the CRL too, if you don't trust our OCSP.
I'll give your a timeline of what happened, to try to explain what I mean with "communication mistake":

  • July 16th -> reported all affected certificates. All of them are for the same customer, so the impact for them is really high. Same day we reported the customer about the need to revoke. Due to the impact of the revocation, the customer requires us to wait until the last available day. We internally ask the revocation team to revoke by July 20th, but mistakenly we didn't clearly specify that it was a requirement of CABF, and it couldn't wait.
  • July 22th -> at first working hour all certificates are revoked. I update this bug being sure that they had been revoked the day before (without reviewing, that's my fault)

As improvement measure, we're reviewing our internal procedures to give more clear instructions to our revocation team, when the cause is a security incidence.

Flags: needinfo?(o-garcia)

Could you provide a bit of description about how key lifecycle works at Izenpe?

For example, you mention having a "revocation team", which I imagine is separate from your team (which is... compliance?), and I'm not sure what other teams may exist. It's clear that not all teams understand the Baseline Requirements, but that shouldn't be an issue, because your CP/CPS should have incorporated the same. That is, if your revocation team understands your CP/CPS, it shouldn't have mattered that it was a requirement of the CABF?

It suggests that, perhaps, there's a disconnect in incident handling, in which different sub-teams may be involved in different tasks of incident response (e.g. the initial incident report, the compliance review, the revocation). And that seems like it may lead to future incidents.

You mentioned a review of your internal procedures, but I think it'd be useful if you could describe how things are implemented at Izenpe presently. This would allow the public to highlight concerns or risks that might also be worth addressing as part of your review of internal procedures.

From the description in Comment #10, it does sound like the delayed communication was, itself, a separate CABF BR incident. In which case, everything above would be best addressed as part of a new incident report, "Failure to revoke within 5 days", as a new bug. You can simply mention that bug on this issue, and file a new 7 point incident report, and trying to address the concerns above as part of your description (e.g. within questions 6 and 7).

The goal of that new bug is to help build a comprehensive understanding about Izenpe's current design and structure, how things went wrong, and how the systemic issues are being identified and fixed, in a way that other CAs can learn from and implement.

Flags: needinfo?(o-garcia)

We have two different teams:

  • Validation agents -> compliance review
  • Issuance/revocation agents -> have access to the PKI console to issue or revoke certificates

And these are the main steps our internal procedure for security incidences:

1.- Event detection: as soon as it's received, it's registered centraliced in an internal database. In case it affects to a critical service (defined by our BIA) this incidence has an specific treatment. In this case it came from bugzilla. The CSO follows all the process.

2.- Event identification: we identify and clasify the event into one of the following:

  • Failure or Security Incident (Breakdown): An event in which the security of an asset has been compromised.
  • Threat (problem): Detection of an agent that may cause damage to one of the Izenpe assets.
  • Weakness (problem): Detection of a possibility of failure in some of the assets of Izenpe.
  • Malfunction (failure): Error in any of the Izenpe computer assets, which may lead to a weakness or a security incident.
  • Improvement (request): Detection of an improvement in the application of security measures.
  • Privacy: all those security violations that cause the destruction, loss or accidental or illicit alteration of personal data transmitted, stored or otherwise processed, or the unauthorized communication of or access to such data.
    In case it's necessary to activate the Contingency Plan we have specfic steps.
    In this case it was clasified as threat.

3.- Communication: depending on the type of event it's notified to the corresponding entities (i.e.: CABForum, Spanish Ministry, third parties, etc.). In this case it was notified to bugzilla and to the customer.

4.- Event treatment: once the event has been identified and clasified, the CSO is responsible for the event treatment, since the beginning till the closure. In case there's a risk of CA key compromise there's an specific procedure. In this case Izenpe, accordingly with the customer, having in mind the impact of the revocation over this entity, decided to wait until the last available day to revoke. The CSO communicated the need to revoke all those certificates to the issuance/revocation team, by an automatic generated email. In that email it was specified that it's due to a security incidence, and it includes (among other information) event type, criticity and risk level. It was clasified as a threat (we don't have evidence of compromise), so although in the email it was specified that all certificates had to be revoked by the July 21th, they were revoked the next day at first working hour.

5.- Close: security events, once closed, can be used for awareness-raising practices, as examples of what might have happened, how to respond to them and how to avoid them in the future. In particular, detected safety events may be used within the internal security training plan.

As an improvement measure we're going to define the new type "SSL certificate misissuance" into the clasification of our database, to be sure that it's treated correctly. It'll be done by next August 4th.

Flags: needinfo?(o-garcia)

(In reply to Oscar Garcia from comment #0)

16 certificates with stateOrProvinceName as “FRANCE”, instead of “Île-de-France”

https://crt.sh/?id=976518444
https://crt.sh/?id=955588634
https://crt.sh/?id=1154593917
https://crt.sh/?id=1130629695
https://crt.sh/?id=923040929
https://crt.sh/?id=967383323
https://crt.sh/?id=1154423592
https://crt.sh/?id=962032211
https://crt.sh/?id=1137176095
https://crt.sh/?id=1137206205
https://crt.sh/?id=1109849921
https://crt.sh/?id=1231234643
https://crt.sh/?id=1279443215

Only 13 certificates are listed above :

  • 12 certificates with stateOrProvinceName = France
  • 1 certificate (crt ID 1454136779) with stateOrProvinceName = Île de France

I can find the complete list of revoked certificates for that date in your CRL : http://crl.izenpe.com/cgi-bin/crlsslev
From this revocation list I found the total of 16 certificates revoked on 2020-07-22, all from the same organization.

But I notice the 3 new certificates (missing from your list) actually have the correct stateOrProvinceName = Île-de-France

I have also found another list of revoked certificates for that date in another CRL : http://crl.izenpe.com/cgi-bin/crlsslev2
From this list revocation list I see 7 certificates revoked on 2020-07-22, all from the same organization.

But I notice the 7 new certificates actually have the correct stateOrProvinceName = Île-de-France

Looks like you've just revoked all certificates emitted for the organization with serialNumber = 35600000000048
Can you please confirm there was another issue and explain what was it.

Flags: needinfo?(o-garcia)

(In reply to frntn from comment #13)

(In reply to Oscar Garcia from comment #0)

16 certificates with stateOrProvinceName as “FRANCE”, instead of “Île-de-France”

https://crt.sh/?id=976518444
https://crt.sh/?id=955588634
https://crt.sh/?id=1154593917
https://crt.sh/?id=1130629695
https://crt.sh/?id=923040929
https://crt.sh/?id=967383323
https://crt.sh/?id=1154423592
https://crt.sh/?id=962032211
https://crt.sh/?id=1137176095
https://crt.sh/?id=1137206205
https://crt.sh/?id=1109849921
https://crt.sh/?id=1231234643
https://crt.sh/?id=1279443215

Only 13 certificates are listed above :

  • 12 certificates with stateOrProvinceName = France
  • 1 certificate (crt ID 1454136779) with stateOrProvinceName = Île de France

I can find the complete list of revoked certificates for that date in your CRL : http://crl.izenpe.com/cgi-bin/crlsslev
From this revocation list I found the total of 16 certificates revoked on 2020-07-22, all from the same organization.

But I notice the 3 new certificates (missing from your list) actually have the correct stateOrProvinceName = Île-de-France

I have also found another list of revoked certificates for that date in another CRL : http://crl.izenpe.com/cgi-bin/crlsslev2
From this list revocation list I see 7 certificates revoked on 2020-07-22, all from the same organization.

But I notice the 7 new certificates actually have the correct stateOrProvinceName = Île-de-France

Looks like you've just revoked all certificates emitted for the organization with serialNumber = 35600000000048
Can you please confirm there was another issue and explain what was it.

There's a mistake in the list provided in comment #0, the last one is not affected by this incidence. The correct one is:

https://crt.sh/?id=976518444
https://crt.sh/?id=955588634
https://crt.sh/?id=1154593917
https://crt.sh/?id=1130629695
https://crt.sh/?id=923040929
https://crt.sh/?id=967383323
https://crt.sh/?id=1154423592
https://crt.sh/?id=962032211
https://crt.sh/?id=1137176095
https://crt.sh/?id=1137206205
https://crt.sh/?id=1109849921
https://crt.sh/?id=1231234643

So the affected certificates are 12. About the rest of certificates revoked for that organization, it's a consecuence of a directive from the senior management of Izenpe. As we indicated in comment #0, we have decided to not issue to entities out of Spain, because we're validating the localities and provinces taking the information from a national oficial source. But we're no just stop issuing certificates for entities out of Spain, but to reduce the risk surface (obviously we're confident that all certificates are correct), senior management have decided to revoke all certificates issued to any entity out of Spain. Those ten certificates with stateOrProvinceName = Île-de-France are correct, but there are some other certificates from the same entity that will be revoked before the end of 2021.
The business of Izenpe is dedicated to the local public administration, and we want to focus our efforts to that goal.

Flags: needinfo?(o-garcia)

Those ten certificates with stateOrProvinceName = Île-de-France are correct, but there are some other certificates from the same entity that will be revoked before the end of 2021.

That... sounds like a very long time. It sounds not much different from natural expiration. Have I misunderstood?

Since you mentioned a mistake in the list provided by Comment #0, could you redo the Incident Report, with the appropriate timelines and response? I think you've shared useful information here, but it would be good to make sure it's structured in a way that's representative for your CA.

Flags: needinfo?(o-garcia)

Here it is a reconstruction of the Incident Report with corrected data and incorporating information gathered in the bug.

Regarding the decision of revoking certificates issued to non-spanish Organizations, naturally, some of them will expire before this year ends, but these are approximately only one third of the total, we still have another two thirds to face.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

2020/06/21 - 19:59 UTC. Bugzilla bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1647121) triggered this incident. Two certificates were reported initially and further analysis detected another 13.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2019/11/15. Izenpe decides not to issue any certificate for organizations out of Spain.
2020/06/22. Organized training for all validation personnel and RA operators.
2020/06/22. Developed and put in production a script that sends an email to validation operators for each certificate issued. The validation agent will manually review that the issued certificate is the same he validated.
2020/06/24. Initially reported certificates were revoked, and Izenpe began an investigation.
2020/07/7. Required all customers to use the web application to request certificates.
2020/07/16. Another 13 certificates were found with the stateOrProvinceName field value equals to France.
Izenpe’s senior management decide to revoke all certificates issued to foreign entities as soon as possible.
2020/07/22. Remaining 13 certificates were revoked. Additionally, another dozen were revoked according to the previous decision.

  1. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Since last July 7th all requests go through our web application, where localities and provinces are taken from the Spanish Ministry official list, so there’s no risk to put a province by hand.

  1. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
Problem # certficates affected First Ocurrence Last Ocurrence
Invalid Value for stateOrProvinceName field (=France) 15 2018/10/31 2019/02/25
  1. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=1111091670
https://crt.sh/?id=1045678964
https://crt.sh/?id=976518444
https://crt.sh/?id=955588634
https://crt.sh/?id=1154593917
https://crt.sh/?id=1130629695
https://crt.sh/?id=923040929
https://crt.sh/?id=967383323
https://crt.sh/?id=1154423592
https://crt.sh/?id=962032211
https://crt.sh/?id=1137176095
https://crt.sh/?id=1137206205
https://crt.sh/?id=1109849921
https://crt.sh/?id=1231234643

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In all cases the request form included "Île-de-France" as the region of Paris (L), and we reduced the stateOrProvinceName to "France".
Fifteen months ago, we didn't have an automatic process to check Locality and Province names, and two operators verified that "Île-de-France" was the correct region, but at the time of issuance the third operator (the one who issued the certificate) reduced the region name to "France".

  1. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• Require all customers to use the web application to request certificates -> done (July 7th)
• Applicant must fill a form where localities and regions are fixed (list got from the Spanish Ministry) -> done (June 21th)
• We don’t issue certificates to entities out of Spain -> done (November 15th, 2019)
• The CSR uploaded by the applicant is re-created from the information of the request form. We only get the public key from the CSR sent by the applicant, we create a new CSR with this public key, and the validated information -> done
• We've developed a script that sends an email to validation operators for each certificate issued. The operator will manually review that the issued certificate is the same he validated -> done (June 21th)
• By the end of September we plan to integrate our web application into our PKI system to reduce the risk of manual errors to the minimum.
• By the end of 2020, all certificates issued to foreign companies, will be revoked.

Whiteboard: [ca-compliance] → [ca-compliance] Next Update 1-October-2020

Hi Ben, sorry. We have had a delay in our process of deploying and testing the application. We have agreed with our software developer as the end date on October 19

We already have in production environment all changes to integrate the web application with the PKI system. We have developed two applications; one for the customer, where all requests are registered, and the other one for Izenpe operators. The internal frontend is able to issue the certificate once all validations are correct. A qualified certificate with two authentication factors are required to access to both applications.
The system manages all evidences in the life-cycle of each certificate.
This change also includes that all validations are made automatically, so we've reduced the risk of human mistakes.

Thanks

Flags: needinfo?(o-garcia)
Whiteboard: [ca-compliance] Next Update 1-October-2020 → [ca-compliance] Next Update 2020-12-01

I will close this case on or about Wed. 9-Dec-2020 unless there are any other issues to discuss.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next Update 2020-12-01 → [ca-compliance] [ev-misissuance]
You need to log in before you can comment on or make changes to this bug.