1674082 - Dhimyotis / Certigna: Certificates issued with validity periods greater than 398-days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[r.delval]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36

Steps to reproduce:

Context reminder
We create this ticket in connection with the identification of 76 certificates and pre-certificates issued between September 1 and September 28, 2020 for which the fixed lifetime was 398 days, but without taking into account the second inclusive. The ticket for reporting this non-compliance is available at this address https://bugzilla.mozilla.org/show_bug.cgi?id=1667744, and this second ticket was created after internal consolidation of the detailed history of actions carried out.
Almost all of the certificates affected by this non-compliance are certificates issued to French administrations and ministries, these being mainly used for services and teleservices of the French State.

Actual results:

Operations following the report
The non-compliance was identified and escalated on September 28 afternoon.
On September 29, 2020, we notified our national Supervisory body of this non-compliance and of our intention to revoke the certificates concerned. As a reminder, the supervisory body in France is ANSSI (National Information System Security Agency), an organization attached to the Secretary General of Defense and National Security.

Our assessment body was also informed of the subject and observed its treatment during our qualification audit which took place from October 2 to 8, 2020.

The Certificates Managers for the relevant certificates were notified of our wish to revoke their certificates on September 29. Almost half of the certificates were revoked before the October 2. However, it turns out that many of the remaining certificates were already deployed on government teleservices for which a revocation would have had an impact on these services, sometimes managed by externals providers.

Here is the revocation history for the 76 affected certificates and pre-certificates:

• 1 certificate was already revoked on September 9;
• 28 certificates were revoked on D + 4 ;
• 17 certificates were revoked before D + 10;
• 30 certificates were revoked before D + 25.

Remediation and improvement of revocation practices

Our certificate profiles have been corrected on September 29 in order to issue certificates whose lifespan can not exceed now 397 days, in accordance with the recommendations.
After discussion with ANSSI on the circumstances which led us to delay the revocation of certain certificates for a few days, ANSSI confirmed to us that we could now request its support, if similar circumstances occur, in order to guarantee compliance with revocation deadlines and anticipate the possible impact on State services.

We also made all our staff aware of this event and in particular on:

• Compliance with revocation deadlines and recourse to ANSSI to educate those Certificates Managers to take this deadline into account.
• Reminder of the need to take into account the inclusive second when changing the lifetime of a certificate profile.

We have also added and performed additional checks (manual and automatic) to our change management procedures and control processes to strengthen our oversight on these points.
We remain at your disposal for any further information.

Comment 2

[Ryan Sleevi]

I fail to see how

Compliance with revocation deadlines and recourse to ANSSI to educate those Certificates Managers to take this deadline into account.

Will prevent this, given that the stated cause was:

However, it turns out that many of the remaining certificates were already deployed on government teleservices for which a revocation would have had an impact on these services, sometimes managed by externals providers.

It seems like Dhimyotis/Certigna could have revoked the existing certificates, as scheduled.

Equally, once the decision was made to delay revocation, an incident report should have been created regarding this, and yet, wasn't. This is in clear violation of the expectations in https://wiki.mozilla.org/CA/Responding_To_An_Incident

Given that there's been ample discussion of these expectations, both within m.d.s.p. and in the incidents of other CAs, this seems an inexcusable oversight, and so it's important to know what steps are being taken to ensure better adherence to and understanding of expectations is being done.

Comment 3

[r.delval]

As it was said in https://bugzilla.mozilla.org/show_bug.cgi?id=1667744#c8 and https://bugzilla.mozilla.org/show_bug.cgi?id=1685142#c1 the incident report was not correctly written.

You will find a more complete report where, I hope, you will find why we delayed certificate revocation and how we will treat it if there will be a new incident

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We have been warned by the opening of incident https://bugzilla.mozilla.org/show_bug.cgi?id=1667744 where certificates were generated with incorrect duration.
We tried to comply with the 5 days revocation delay but failed for a number of certificates.
Ben WIlson reminded us that in case of delay for revocations, a new incident report had to be created (https://bugzilla.mozilla.org/show_bug.cgi?id=1667744#c6)

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

28/09/2020 16:49 CEST : The bug on certificate duration was notified to our security team.
29/09/2020 01:00 CEST : We updated our certificate generation service to solve this error.
29/09/2020 17:16 CEST : we notified our national Supervisory body of this non-compliance and of our intention to revoke the certificates concerned.

30/09/2020 : We have contacted all certificate managers to notify them of the certificate duration issue and the need to revoke them promptly (phone calls and emails).
01/10/2020 : Some certificate managers inform us that several certificates are used on platforms of French government services where a service interuption would be highly damaging and where they can't replace the certificate immediatelly.
01/10/2020 : We saw we will not be able to revoke all certificates in the 5 days period. We decided to contact all certificate managers in order to obtain their revocation timeline (as quick as possible).

02/10/2020 : 29 certificates were revoked and replaced
15/10/2020 : 25 certificates were revoked and replaced
23/10/2020 : 22 certificates were revoked and replaced

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Has explained in https://bugzilla.mozilla.org/show_bug.cgi?id=1667744#c10, we stopped certificate issuance as soon as we detected the incident. Certificate issuance was re-enable the next day, after patch publication.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

76 certificates issued between 02/09/2020 and 28/09/2020
There were 75 certificates mississued available at https://crt.sh/?cablint=1456&iCAID=11279&minNotBefore=2020-09-01&opt=cablint plus this one https://crt.sh/?id=3417309879&opt=cablint for a total of 76 certificates
You will find an attached file with the crt.shID and certificate revocation date.

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

see link above for crtsh ID and the attached file for each certificate revocation date

6. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The certificates impacted by exceeding the 5 days revocation period were certificates used by the French government services.
Some certificate managers inform us service interruption would be highly damaging and they can't replace the certificate immediately (before 2020-10-03). This kind of application is managed with other service provider and they needed more time to contact the entity in charge of the services for replacing certificates (key generation, online order, authorization validation, installation of the new certificat and finally revocation of old certificate)

Considering that the initial incident was not a "security incident", we decided to give more time to revok all certificates. The fact here is we missed to open a new bug for explaining revocation delay.
After discussion with certificate managers, we ask them to give us a planning for certificates replacement before 2020-10-15.
Only one customer can't done it before 2020-10-15 because he had to replace more than 30 certificates and there were some difficulties in reaching all the people involved in the certificate deployment chain.
We defined the last deadline will be on 2020-10-23 and finnally we had revoked all certificates on 2020-10-22.

We contacted again our national Supervisory body (ANSSI) to confirm all certificates were revoked but we missed the 5 day revocation delay because certificates manager had difficulty replacing their certificates quickly.
The ANSSI, which is also the security supervisory body for French government entities confirmed to us that we could now request its support, if similar circumstances occur, in order to guarantee compliance with revocation deadlines and anticipate the possible impact on State services.
if this kind of incident occurs again, we will must revoke certificates in conformity with the standards for wich we are qualified, the 5 days revocation delay will must be respected

This incident highlighted that our incident response procedure was incomplete because it was not specified the need of incident report as it is specified in https://wiki.mozilla.org/CA/Responding_To_An_Incident (and more specifically, even if this is not considered a "security incident"). So, we updated this procedure.

This incident also highlighted the obstacles that may face some certificates managers when they must quickly replace a certificate. We are planning changes to our PKI tools in 2021, and in particular with features that allow certificates to be replaced more quickly in some cases (automation, shortcuts on customer area, ...).

To conclude, with Ryan comment #2, we could have revoked all certificates but as it's said in Mozilla wiki : "Mozilla recognizes that in some exceptional circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline". We considered ourselves to be in exactly this type of situation without 'security breach'.
But we aknowledge that we missed the bug creation as stated in the same document (for any circumstences).

Comment 4

[r.delval]

Attachment: list-cert-revoked.xlsx

Comment 5

[Ben Wilson]

It appears that the certificates have been revoked and that there has been adequate discussion captured for future reference. I intend to close this on or about 5-Mar-2021, unless there are additional items to address.