1651651 - Actalis: Failure to revoke within 7 days: OCSP EKU issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Adriano Santoni]

This bug is related to Actalis not revoking within 7 days, as per the BR, the ICAs affected by https://bugzilla.mozilla.org/show_bug.cgi?id=1649961

We will post an incident report as soon as possible.

Comment 1

[Ben Wilson]

An update is needed.

Comment 2

[Adriano Santoni]

We have just updated the related incident with the final plan, https://bugzilla.mozilla.org/show_bug.cgi?id=1649961.

For this bug specifically, we will post the preliminary incident report early in the first couple of days of next week.

Comment 3

[Adriano Santoni]

Here's our incident report.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

This bug is related to Actalis not revoking within 7 days, as per the BR, the ICAs affected by https://bugzilla.mozilla.org/show_bug.cgi?id=1649961

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1572638

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

After the event of last year (see https://bugzilla.mozilla.org/show_bug.cgi?id=1534295), we realized the need for strong improvements to our management of this type of events, in particular as regards the communication with our customers and our internal organization. To this end, we implemented the measures documented on https://bugzilla.mozilla.org/show_bug.cgi?id=1572638

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

This bug is related to the intermediate CA certificates (ICAs) affected by the https://bugzilla.mozilla.org/show_bug.cgi?id=1649961, in that we decided - also in consideration of the measures we adopted to mitigate the related security risks - not to revoke those ICAs within the terms provided for in the BRs, as that would have caused a huge disruption to many critical services.

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

There are no end-entity certificates directly involved in this bug, however the need to revoke the ICAs affected by https://bugzilla.mozilla.org/show_bug.cgi?id=1649961 impacts essentially all of the still-active end-entity certificates issued by Actalis, that is approximately 430,000 (430k).

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As already explained on https://bugzilla.mozilla.org/show_bug.cgi?id=1649961, we decided not to revoke within 7 days the affected ICAs because that would certainly have caused a huge disruption to many critical services, compared with the highly unlikely security risks consequent to a deferment of ICAs revocation.

We decided to implement (and have assessed by our auditor) new specific security measures to protect our own ICA keys from abuse, and at the same time allow our customers the time necessary to replace all their TLS certificates (with new ones issued by new ICAs).

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The corrective measures taken last year (https://bugzilla.mozilla.org/show_bug.cgi?id=1572638) are working, in our opinion. The internal management of the incident has been faster and more effective. Within a few days from the incident inception, a task force was formed, new ICAs were created and deployed, ad-hoc risk mitigation tasks have been completed and are already under assessment by our auditor, the necessary communications to customers were finalized and sent, and we have already started proactively helping our customers to replace their certificates. All customers were contacted over the phone to ensure they had received and understood the communication, resolving any type of discussion referring to the contract and to our CPS. We settled a precise date (30/10/2020) by which all the certificate re-issuances (replacement) deemed necessary by them must be completed and after which the old ICAs will be revoked (and the related keys destroyed).

A further measure that we intend to adopt, and which we think may contribute to reducing the time necessary for the early reissuance of many certificates in unexpected situations (such as https://bugzilla.mozilla.org/show_bug.cgi?id=1649961), consists in proposing to our main customers, starting from Q4 2020, to automate their enrollment processes through the ACME protocol, for which we have developed the necessary support.

Comment 4

[Ryan Sleevi]

Setting the Next Update for the end of month, to track progress on the 2020-10-30 complete revocation.

In the interim, however, it'd be useful to have very specific and precise details about:

We decided to implement (and have assessed by our auditor) new specific security measures to protect our own ICA keys from abuse, and at the same time allow our customers the time necessary to replace all their TLS certificates (with new ones issued by new ICAs).

Comment 5

[Adriano Santoni]

The related inspection report, including technical details, will be shared soon (see bug 1649961).

Comment 6

[Ben Wilson]

Please make sure you address: https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

• The rationale [for the delay, including] an explanation for why the situation is exceptional. (I think this has been provided)
• A clear timeline describing if and when the problematic certificates will be revoked or expire naturally.
• Work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.
• An analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.

Comment 7

[Adriano Santoni]

• A clear timeline describing if and when the problematic certificates will be revoked or expire naturally.

The problematic certificates are the following 5 intermediate CA certificates:

https://crt.sh/?id=1298407195&opt=cablint
https://crt.sh/?id=1298407200&opt=cablint
https://crt.sh/?id=1305420938&opt=cablint
https://crt.sh/?id=2029983391&opt=cablint
https://crt.sh/?id=3059419298&opt=cablint

All the above intermediate CA certificates were issued between 19-Mar-2019 and 20-Sep-2019.

All of them would expire naturally on 22-Sep-2030.

Considering the strong mitigation actions implemented and assessed by our auditor (we therefore believe the residual risk of misuse of the ICAs private keys is negligible), the pandemic situation (we have many customers in the Healthcare sector) with many companies who have workers on standby, and the number of impacted end entity certificates (more than 430.000 to be reissued) we have defined, approved and we commit to this plan of revocations of end-entity certificates issued by those ICAs:

• 80% by the 30th of August
• 98% by the 30th of September
• 99,99% by the 30th of October

Revocation and key destruction for all the affected ICAs is planned by the 6th of November, the exact date still to be confirmed as ICA keys destruction will be done in the presence of our auditor.

• Work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.

We have engaged our auditor since July 9; they carried out an inspection on July 30 and 31, to assess the risk mitigation controls we have implemented, and are now preparing their inspection report; once this report is ready, we will share it with all the stakeholders, and will attach it to this bug.

• An analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.

We believe that the measures we adopted in the aftermath of the past incident (see bug 1572638) are proving effective. We have nearly a half million end entity certificates to revoke and replace, before we revoke the affected ICAs, and we committed to complete this task by the end of October. We will also try to convince our bigger customers to move to certificate enrollment automation (based on the ACME protocol), in the next months, so to further reduce the time needed for unscheduled certificate replacement in situations like this, in addition to other benefits.

Comment 8

[Ben Wilson]

Thanks. This is very helpful. I'll set a next update for Sept. 15.

Comment 9

[Adriano Santoni]

Attachment: LC0455 - ACTALIS SPA 11-08-20.pdf

I am attaching to this bug the statement of our auditor (BVI) regarding the risk mitigation measures we adopted. The detailed audit report, containing technical details, will also be attached as soon as it is released. We expect it by the end of next week.

Comment 10

[Adriano Santoni]

Revocation of all affected certificates is proceeding according to our plan.

To date, we have revoked approximately 88% of all affected certificates.

Furthermore, our auditor BVI just published their detailed report at the following link:
https://www.bureauveritas.it/sites/g/files/zypfnx256/files/media/document/Client_Report_Actalis-Statement-SECINC.pdf

Comment 11

[Adriano Santoni]

I will post another update by the end of September.

Comment 12

[Adriano Santoni]

While steadily progressing with our revocation plan, we had to temporarily re-enable the possibility of issuing new certificates from one of the affected ICAs (https://crt.sh/?id=1305420938) in order to issue one single certificate with a validity of 30 days.

Due to TLS Server certificate pinning to the above mentioned ICA, a major banking institution was forced to renew the certificate of the server used by one of their mobile applications (https://crt.sh/?id=1904999129) which would otherwise expire on Sunday 13 September. The reissuance of said certificate using a new ICA would disrupt their service, impacting millions of users. The new certificate (https://crt.sh/?id=3368606397), issued by the old ICA, guarantees the continued availability of their services.

This has been agreed with all internal stakeholders, starting from our Directorate General and including the NOS department that is monitoring the use of our HSM, and is an exception that does not imply any changes to our revocation plan which will culminate in the revocation of our old ICAs (including the above) by the date we have already indicated.

Comment 13

[Ryan Sleevi]

Adriano: Previously, in Bug 1534295 , we discussed pinning in the context of delayed revocations. You can find my concerns expressed in https://bugzilla.mozilla.org/show_bug.cgi?id=1534295#c23

What resources have you provided your customers regarding the ill-advisability of pinning, precisely because of situations like the one described in that bug or in the exceptional case mentioned here in Comment #12? CAs are perhaps best placed to advise their customers that pinning is harmful, and I'm curious if you can point to any existing references from Actalis as to that point.

Comment 14

[Adriano Santoni]

Ryan, sorry about the delay. Starting from September 2019 and still during the current year we had meetings with individual customers who use mobile applications, in the banking and government sectors, to discuss the problem of pinning. In these meetings, customers generally declared that certificate pinning is imposed on them by their company policies which refer to security best practices for mobile applications (such as e.g. https://owasp.org/www-community/controls/Certificate_and_Public_Key_Pinning). Faced with that, we decided at that time not to make formal communications to customers, however reserving to do so during 2020 after a discussion with the security departments of our major banking and government customers. To date we have not yet published our position on pinning, but we intend to do so by the end of 2020: we plan to introduce clarifications at least in our CPS, which is currently under review, and perhaps also on our website. We will specify that pinning, in addition to being risky if not correctly implemented, is discouraged by us and that, while not forbidden, will not be considered a valid reason for delaying the revocation of certificates that must be revoked.

Comment 15

[Adriano Santoni]

We are on track with our revocation plan. To date, we have revoked approximately 98% of all certificates issued by our old ICAs, and approximately 99% of all affected certificates have been re-issued.

Comment 16

[Ryan Sleevi]

Thanks Adriano. I've also reached out to one of the authors for that page, to see if we can't get better industry guidance there.

Comment 17

[Jim Manico]

(In reply to Adriano Santoni from comment #14)

The OWASP page and other community pages on pinning are in GitHub here. https://github.com/OWASP/www-community/blob/master/pages/controls/Certificate_and_Public_Key_Pinning.md We are very open to PR's explaining the issues with pinning. I am sorry if this resource is causing problems in your world.

• Jim Manico, OWASP Volunteer

Comment 18

[Adriano Santoni]

The destruction of the affected ICA keys is scheduled for November 5 in the morning, and will be witnessed by our auditor.

Comment 19

[Adriano Santoni]

This morning, in the presence of our auditor, all the affected ICAs were destroyed and their certificates were revoked.

The related report from our auditor will be attached here as soon as it is released.

Shortly we will also provide a wrap-up including an update on further measures that we took.

Comment 20

[Ben Wilson]

Is there an auditor's attestation, report or letter concerning CA key destruction that can be uploaded here as an attachment?

Comment 21

[Adriano Santoni]

Yes, it's being prepared. It will be attached here as soon as it is released by our auditor.

Comment 22

[Adriano Santoni]

Attachment: Actalis_BVI-Statement-on-SECINC-807_Final-Statement-ANNEX-signed.pdf

Comment 23

[Adriano Santoni]

Find in attachment our auditor's attestation regarding the CA key destruction.

Comment 24

[Adriano Santoni]

It's also published on our auditor's website: https://www.bureauveritas.it/sites/g/files/zypfnx256/files/media/document/Actalis_BVI-Statement-on-SECINC-807_Final-Statement-ANNEX-signed.pdf

Comment 25

[Ben Wilson]

I believe this matter can be closed. I plan to close it this Friday, 18-December-2020, unless there are additional issues that need to be addressed.