Closed Bug 1572638 Opened 5 years ago Closed 5 years ago

Actalis: Failure to revoke certs within the BR required timeframe

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Giorgio.girelli, Assigned: Giorgio.girelli)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36

Type: defect → task
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

This bug is related to the process for dealing the incident https://bugzilla.mozilla.org/show_bug.cgi?id=1534295 , mainly about the timeframe for revoking part of the 249.627 certificates.
The initial plan https://bugzilla.mozilla.org/show_bug.cgi?id=1534295#c2 was: “We expect to revoke the majority of impacted certificates within approx. 30 days, barring unforeseen circumstances. We are preparing suitable procedures in order to achieve that with the minimum disruption to our customers. We anticipate that for a fraction of the impacted certificates it will take longer due to the limited agility of certain types of customers in handling certificate re-issuances at unexpected times”. The initial plan was to revoke all for the end of June… we did it by the end of July.
Following the template https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report, without some points as they are not relevant.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    We was aware from the beginning, in march, about the potential problem (Failure to revoke certs within the BR required timeframe ) because of the huge number of certificates and mainly considering the type of customers involved (big organizations, much of them were central public administrations). Anyway we was able to set a plan, considering all the rules in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation, and trying to avoid big service disruptions. Actalis at the beginning followed the defined goals (88% - 215.000 certificates revoked during the first month), even though we needed some preparation, but after that, starting mid of June, we understood that the goal to close the incident for the end of June was impossible to achieve, because of the communications, plans and complains we received back from the customers.
    We can consider the mid of June (see https://bugzilla.mozilla.org/show_bug.cgi?id=1534295#c12) as the milestone date of this incident.

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    All the details of the related incident are reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1534295.

The comments more relevant, representing the timeline of this incident, are:

Points from 3 to 5 of the incident template are not relevant in this incident.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    After our final analysis we have found 3 areas of improvement – two minors and one big, the third C), that can be considered the route cause of the problem.
    A) Slow handling of the incident at the beginning (but quick response in patching Ca’s in production) https://bugzilla.mozilla.org/show_bug.cgi?id=1534295#c4 – This was related on and strictly connected to following topics B) and C) – setting up and communicate a plan need clear information that in these case were not enough quick and detailed, mainly C) one.
    B) Process of issuing and revoking DVs certificates needed to be improved to achieve the expected rate of reissuing and revoking –– we discovered on the field that we was not 100% prepared to implement this process at the expected rate, so we needed to implement some ad hoc software procedures - The details are in https://bugzilla.mozilla.org/show_bug.cgi?id=1534295#c6
    C) Communication with our customers – the way we made communication to avoid disruption was not clear enough, and the way to dealing with customer’s complaints was inadequate to the circumstances. All the details are in https://bugzilla.mozilla.org/show_bug.cgi?id=1534295#c17 – Summarizing – We have met a lot of resistances and compliances from the enterprise customers, mainly public entities, to whom we have wrongly responded by giving more time.

  2. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
    Following there are the steps focused on the 3 areas of improvements we the final objective to avoid a similar situation and achieve at least these results:

  • performing timely impact analysis and a clear and committed plan for resolving matters of non-compliance faster than we did in that case
  • strictly follow the plan, dealing firmly with customer communications and complaints consistently with the milestones of the plan

List of steps:

  • Organization change:
    • SSL Infrastructure Manager has been replaced with another Manager already in the company – done the 22nd of July
    • Operation Manager, also responsible for customer communication has been moved to another role – the communication responsibility for enterprise customers is now in charge to the Enterprise Customer Service Director, for the online customers is now in charge to the on line Product Manager – We have indeed separated the responsibility of communication strategy defined for on-line and small customers from the one defined for enterprise and big customers (including public administration). All the communications need the approval by Mr. Andrea Sassetti, covering the role of SSL CA Director (exactly has it happens for EIDAS CA communication), done on 22nd of July
  • Internal procedures change:
    • revision of our Security Plan on handling critical circumstances like security issues or huge impact – key points of this new revision are:
      • immediate escalation directly to the Managing Director and CEO of the company
      • immediate constitution of a task force, lead directly by the Actalis Managing Director and SSL CA Director, with meeting on a daily bases with full responsibility on: undertake actions, approve communications (external and internal), reinforce the teams and activate, in case of huge impact on the enterprise customer, a specific team dedicated to follow all impacted customers day by day, with customer meetings, period phone calls and also technical support where needed. Updating the community is also a responsibility of this task force.
    • revision of our internal trainings aimed at raising the awareness of all internal stakeholders on the BRs and how we have to deal with incident involving revocations

Both these actions, including the execution of the trainings on all the involved internal organization, are planned to finish by the end of September

  • Communication and Contract change:
    • we need to raise the awareness of our customers on the BRs, on our CPS and consequently on the risks deriving from adopting bad practices. We have defined an internal plan that involves revision of our CPS, of the contract, of some communications done during issuing phase. The final versions of the documents and the necessary updates in production are planned by the 20th of September

Giorgio

Assignee: wthayer → Giorgio.girelli
Whiteboard: [ca-compliance]

Giorgio: please update this bug when each of the remaining actions described in comment #1 is completed.

Flags: needinfo?(Giorgio.girelli)

Sure. Tomorrow we have an internal review of the project and then I'll update this incident

Task update:

  • Internal procedures change:

    • We have updated our internal Incident management plan modifying how we handle specific situation about SSL security and compliance issues – This document is at final approval stage
    • We have developed, approved and published in our internal knowledge base a specific Playbook called PlayBook for SSL Server certificates mis issuance - Released
    • We have reviewed roles and responsibility of our internal Incident Response Team to better define roles and responsibilities in case of SSL Server certificates mis issuance - This document is at final approval stage
    • We have modified our internal procedure used to deliver SSL Server certificates adding references to the above mentioned Playbook - Released
    • Training – Yesterday has been done a specific internal training prepared to make people aware about how to handle this situations in the future. The training has been recorded to be added to our induction plan for new employees involved in SSL business
  • Communication and Contract change:

    • We have reviewed our General Terms and Conditions for SSl server certificates – this is under final review by our legal department
    • We have reviewed the Certification Practice Statement Certificati SSL Server - this is under final review by our legal department. Consider that before publishing it we have to ask an approval to AGID (Agency for Digitalization in Italy) because our SSL certificates are also eIDAS certified, and the rules requires this step.

Next update early next week.
Giorgio

Task update:

  • Internal procedures change (ALL DONE):

    • We have updated our internal Incident management plan modifying how we handle specific situation about SSL security and compliance issues – Released
    • We have developed, approved and published in our internal knowledge base a specific Playbook called PlayBook for SSL Server certificates mis issuance - Released
    • We have reviewed roles and responsibility of our internal Incident Response Team to better define roles and responsibilities in case of SSL Server certificates mis issuance - Released
    • We have modified our internal procedure used to deliver SSL Server certificates adding references to the above mentioned Playbook - Released
    • Training – The internal training and the final exam has been completed and the training is now part of our induction process for new employees involved in SSL business - Completed
  • Communication and Contract change:

    • We have reviewed our General Terms and Conditions for SSl server certificates – Released an internal change request has been opened to publish it on the wesite (english version probably will take a couple of day more)
    • We have reviewed the Certification Practice Statement Certificati SSL Server - this has been approved and signed and it is now under review by AGID (Agency for Digitalization in Italy) because our SSL certificates are also eIDAS certified, and the european rules requires this step.

With all the implemented changes we are sure to be able to manage next events, of any impact, in a way fully compliant with the rules in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
Waiting for any comments.

Next update when GTC and CPS will be finally updated on the web site
Giorgio

Last update.

We have received the final approval by AGID (Agency for Digitalization in Italy) and we will publish the new version of the CPS in a couple of day on our public site. All the planned actions have been completed.

We remain at disposal for farther informations.

Regards, Giorgio

Giorgio:

Could you indicate how to find this? When I navigate to https://www.actalis.it , and then click on the SSL and code signing, the CP/CPS linked from that page is dated 23 May 2019 (version 5.2). This matches if I follow the English version of the products page, which links to an English v5.2

Could you indicate where the new CPS is published?

Dear Sleevi,

this is my fault, I was sure I had added a new comment.

In fact AGID approved version 5.4 but the day after, on the 9th of October, they asked a small formal review. This morning we asked them, even though we still do not have the formal approval to publish the new 5.5 version. As only the final signature is still missing in their internal procedure, they give us an informal ok.

You can find the new versions exactly at the links you have mentioned.

Regards, Giorgio

Flags: needinfo?(Giorgio.girelli)

Giorgio: Thanks. I can confirm 5.5 is now loading.

I'm curious why the version I loaded in Comment #7 was version 5.2. It would appear versions 5.3 and 5.4 were not published?

I'm assuming these were part of the updates from Comment #1, which estimated 20 September, and took longer than expected (almost a month). I also wasn't sure what the other document updated (mentioned in Comment #4 - General Terms and Conditions for SSl server certificates) was, and if you could link to it.

Flags: needinfo?(Giorgio.girelli)

Dear Sleevi,

you are right, version 5.3 and 5.4 were not published:

  • Version 5.3 because of it was already under approval before our decision to rework it due to this incident review, so we discharged that version.
  • Version 5.4, has explained, because AGID requested a small formal change. And for procedure we have to request the approval for an upper version

The version 5.4, the one forecasted in our plan for this incident, was released the 1st of October as for my Comment #5 “We have reviewed the Certification Practice Statement Certificati SSL Server - this has been approved and signed and it is now under review by AGID (Agency for Digitalization in Italy) because our SSL certificates are also eIDAS certified, and the european rules requires this step.”

Finally the updated General Terms and Conditions have been released and published the 1st of October, or the days immediately after, as for my comment 5 “We have reviewed our General Terms and Conditions for SSl server certificates – Released an internal change request has been opened to publish it on the wesite (english version probably will take a couple of day more)”. The updated version is the 1.4 and you can find at these links (Italian and English versions)

https://www.actalis.it/documenti-it/sslserver_codesigning_condizionigenerali.pdf

https://www.actalis.it/documenti-en/sslserver_codesigning_termsconditions.aspx

Regards, Giorgio

Flags: needinfo?(Giorgio.girelli)

Thanks Giorgio.

Wayne, I think this goes to you.

The new Terms & Conditions include:

The Customer acknowledges and accepts that Actalis may:
...
b) revoke the Certificate within the maximum timescale indicated in the CPS
in the event that the Certificate demonstrates any non-compliance with CABF
Standards, regardless of the impact of such non-compliance on the security
or correct functioning of the Certificate; in such cases, Actalis shall notify the
Customer where possible, within the limits permitted by the maximum
timescales for revocation indicated in the CPS according to the circumstances.
In the event of revocation of certificates by Actalis, for any of the reasons
stated herein, Actalis accepts no liability for any inconvenience, disruption or
malfunctions suffered by the Customer as a result of the revocation.

The CP/CPS includes, in 4.9.1.1

Before revoking a certificate, the CA will make a reasonable effort to warn to the affected Subscriber of the
imminent revocation, compatibly with the maximum revocation times indicated above.

Comment #1 is a good example of a systemic and structured approach, in part related to Bug 1534295. We'll have to see how well it works in practice, although, ideally, we won't :)

There's other stuff that raises an eyebrow, but not to the level of this issue, and might be more relevant for future policy updates.

  • Providing versioned history for the CP/CPS (took me a bit to compare the diffs of the related documents)
  • Expectations, if any, around CP/CPS publication (e.g. the AGID process invokes possible challenges for future updates, such as was required for CA problem reporting)

But I think those are not specific to Actalis here and don't need to be dealt with on this issue.

Flags: needinfo?(wthayer)

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
Whiteboard: [ca-compliance] → [ca-compliance] [delayed-revocation-leaf]
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-leaf] → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.