1598608 - Izenpe: intermediate certificates not revoked within BR time period

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Oscar Garcia]

This is an incident report for 4 intermediate certificates not listed in audit reports and not revoked within the BR time period.
The intermediate certificates are 4 of the 7 intermediate certificates not listed in audit reports and should therefore have been revoked. This was reported as a bug earlier; ‘Izenpe: CA certificates not listed in audit report’ (Bug 1596744).

• How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of this problem by a note of Ryan Sleevi on the bug referenced above (1596744)

• A timeline of the actions your CA took in response. A timeline is a date-and-timestamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

The first incident (7 intermediate certificates not listed in audit reports) was reported November 15th 2019 and at that point in time our intention was to revoke all those subCA certificates.
We have revoked 3 of the 7 of them today November 22th. The reason for not revoking the last 4 intermediate certificates is due to the impact that the revocation will generate, because those subCAs issue subscriber certificates that are used by all citizens and entities with their procedures with the Administration. That change must be done with a reasonable minimum of guaranties.
November 20th Ryan Sleevi clarified on mozilla.dev.security.policy the expectations for a CA with intermediate certificates not listed in the audit report, and he noted in Bug 1596744. Based on this clarification, we decided to create an incident report (this one) since the 4 intermediate certificates were not revoked within the BR time period.
Our understanding is that according to section 4.9.1.2 of BRs this situation doesn’t fit to any of causes described there. However, we understand that the expectations from the community differs from our view and respect this.
We plan to revoke all pending subCAs next November 29th.

• Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The 4 intermediate certificates were issued between 2008 and 2009.
We do no longer issue intermediate certificates “within scope” of our audits without including them in the audit reports.
We will also do our best to ensure that any revocation of intermediate certificates will be within the BR time period.

• A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

4 intermediate certificates not intended to issue SSL/TLS certificates.

• The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The 4 intermediate certificates still under investigation are:
CA Teknikoa - CA Tecnica
https://crt.sh/?id=12724269
Herritar eta Erakundeen CA - CA de Ciudadanos y Entidades (4)
https://crt.sh/?id=9035578
Herritar eta Erakundeen CA - CA de Ciudadanos y Entidades (3)
https://crt.sh/?id=9035577
EAEko HAetako langileen CA - CA personal de AAPP vascas (2)
https://crt.sh/?id=9035576

• Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We were aware of the mistake looking at the CCADB ALV results, and comparing them with Kathleen clarifications, where we realised that although all CA certificates were covered in the audit, the fingerprint of all SHA1 subCA certificates were not included in the report.

• List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We do not consider the failure to include all the intermediate certificates in the audit report to be a security issue. Therefore, we do not consider the failure to revoke these intermediate certificates within the BR time period to be a security issue.
However, we do understand that these failures are compliance issues and will ensure that all affected intermediate certificates are revoked.
Since these intermediate certificates are still extensively used, we must ensure that revoking them will not cause any main issues for customers still is using these old intermediate certificates.
We have identified customers still using the affected intermediate certificates, are in the process of contacting them, and advise them to replace the intermediate certificates with the latest generation to avoid any issues at time of revocation.

Comment 1

[Oscar Garcia]

All pending subCAs have been revoked:

CA Teknikoa - CA Tecnica
https://crt.sh/?id=12724269

Herritar eta Erakundeen CA - CA de Ciudadanos y Entidades (4)
https://crt.sh/?id=9035578

Herritar eta Erakundeen CA - CA de Ciudadanos y Entidades (3)
https://crt.sh/?id=9035577

EAEko HAetako langileen CA - CA personal de AAPP vascas (2)
https://crt.sh/?id=9035576

Comment 2

[Ryan Sleevi]

Oscar: Thanks for the update.

I think one of the key objectives of bugs like this is to understand what steps the CA is taking to ensure there are not future delays in revoking subscriber certificates or intermediate certificates, whether used for TLS or used for other purposes.

You briefly mention this, in

We will also do our best to ensure that any revocation of intermediate certificates will be within the BR time period.

One of the reasons for asking CAs to create bugs is so that we can better track how CAs are doing, individually, in achieving this, as well as better share information, as an industry, on how to ensure this.

You mention that these intermediates were challenging because of their use for non-TLS purposes. How has Izenpe adjusted its PKI design to account for this? What opportunities exist to help ensure timely revocation with minimal disruption? For TLS cases, has Izenpe considered supporting automated issuance, so that revocation of an intermediate can be done timely with minimal disruption?

These are the sorts of things we should be trying to understand through bugs like this. The steps, both technical and procedural, to help minimize the disruption or reasons for delay, or to better understand and disseminate some of the challenges.

Comment 3

[Oscar Garcia]

Izenpe is involved some time ago in a project to divide the tree of CAs, so that we have a root for all subCAs issuing TLS certificates (we don’t have EV code signing profile), and some other different ones for the rest of CAs. We’ll keep the current notified root CA in the CCADB as it is. We expect to have it by the end of 2020. That would reduce the risk in case we have any incidence with TLS certificates, and obviously the impact to our customers would be substantially reduced.
According the automated issuance, we are finishing the development of a new web application to request and manage all TLS certificates (DV, OV and EV). That application will automate most of all verification we're currently doing manually. We expect to put it in production next January. With that application the human action will be reduced to the minimum.

Regards

Comment 4

[Ryan Sleevi]

Oscar: Are you still on track to have the new system in place by end-of-month?

Comment 5

[Oscar Garcia]

Hi Ryan, we plan to put the new system in the production environment by next Wednesday 29th. We'll update this bug once it's up and running.
Regards

Comment 6

[Oscar Garcia]

Yesterday we put in production our new application to manage all web certificates.
Thanks

Comment 7

[Ryan Sleevi]

Wayne: I think all remediation steps are completed here.

• The affected certificates have been revoked
• Izenpe has restructured its TLS PKI to prevent further delays

A longer-term remediation is to structure its non-TLS PKI, estimated EOY 2020, to prevent risk of delays from non-TLS compliance. However, I don't think that needs to keep this bug open.

Comment 8

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.