1656882 - NetLock: Failure to revoke noncompliant ICA within 7 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Varga Viktor]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36

Steps to reproduce:

Mr. Ryan Sleevi reported in the ticket 1586795 on 2020-07-02, the Netlock failed with the timely revocation of the problematic ICAs. He requested a report regarding BR specific deadline breach.
(https://bugzilla.mozilla.org/show_bug.cgi?id=1586795)
The ticket 1586795 was started before 2020 january, under the than effective 2.6.1 policy.
When the 2.7 went info force (1st January 2020) in this case this requirement was missed unfortunately.
Those ICAs were under replacement, and later they were revoked.

Actual results:

• Receiving the original report on 2019-10-07 about the intermediate certificates with missing EKUs
• SSL ICAs were revoked on 2020-05-31
(see also relevant Ticket here)
• CA of digital signatures was revoked on 2020-06-30
(see also relevant Ticket here)
• Ryan Sleevi reported the missing incident reports on 2020. 07. 02

Expected results:

When the 2.7 went info force (1st January 2020) the rules were changed.
After this date the revocation is mandated in 7 days by the policy, but this was missed unfortunately.

Comment 1

[Varga Viktor]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Ryan Sleevi reported in the ticket 1586795 on 2020-07-02. He requested a report regarding BR specific deadline breach.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   Ticket 1586795 contains the timeline for the CA swap.

This Ticket is about the investigation of the missed revocation aiming to identify root causes.
Timeline of the Revocation topic:
• Receiving the original report on 2019-10-07 about the intermediate certificates with missing EKUs
• Interim SSL certificate revocation on 2020-05-31
• CA of digital signatures revocation on 2020-06-30
• Ryan Sleevi reported the missing incident reports on 2020. 07. 02

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

• Current incident report is only about the missed timely revocation. The revocations were not scheduled on 2020.01.07 as the new 2.7 policy mandated.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Number of certs: 3
First and last date: 2019-04-02; 2019-05-27
https://crt.sh/?id=1379548755
https://crt.sh/?id=1490728474
https://crt.sh/?id=1639470777

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
   Number of certs: 3
   First and last date: 2019-04-02; 2019-05-27
   https://crt.sh/?id=1379548755
   https://crt.sh/?id=1490728474
   https://crt.sh/?id=1639470777

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

• The v2.6.1 Root Policy was not implying the revocation unfortunately the policy update for 2.7 have been ignored in this case and followed up just after that went live.
• The incident was handled in a different workflow so the alignment with the new policy has been missed unfortunately.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• Risk management software renewal have been done to avoid misalignment with the - sometimes contradictory- external policies.
We have implemented into the ETSI EN and ISO 27000 requirements furthermore requirements of Microsoft ad Mozilla root program also added.
• Administrative control and monitoring have been raised with adding new colleagues from compliance side (recorded in CCADB). The task and the responsibility of tracking and applying changes, analyzing new requirements, checking deadlines, etc. officially added to the new role.
• We requested our auditor partner company to analyze the original problem since we are just running an audit now with them. They will check and supervise the -already implemented- changes in policy and processes also will refine these if its needed. The result certificate will be added to the ticket certainly.

Comment 2

[Ryan Sleevi]

Thanks. I'm not really sure I understand what you've changed, or how that addresses the systemic risks.

I think we've talked about, on Bug 1586795, why the original requirements were missed. What's not clear to me is, when the issue was first reported, you didn't immediately take steps to resolve this. We're now looking at further extended delays in revocation because of that initial failure and the disruption.

I think being detailed about how your policies and practices have changed to ensure prompt revocation is the goal of this incident report, while Bug 1586795 can continue to discuss the failure to incorporate and/or be aware of the Mozilla policy changes.

Comment 3

[Varga Viktor]

In November, we first changed the contact person and regarding that this new contact person works in our IT department so the handover, the performing and the controlling of the tasks have remained within the team.

After the second contact change, one of the contacts is provided by Compliance department so an independent unit reviews the IT processes.

Also it in progress to add supplement to the ruling policy that will handle with special attention the changes of Mozilla ROOT policy. This supplement will also ensure that similar cases are avoided in the future.

I will add a drawing to each ticket in a few days, to explain these case better, why they are similar, and to show what was wrong with the process previously.

Yours, Viktor

Comment 4

[Varga Viktor]

Attachment: Timeline and problems

The separated and long processes missed those red feedbacks, so the process shall changed too. Now the compliance is responsible and monitoring these activities as external to IT.

Comment 5

[Ben Wilson]

I will close this bug on or about 9-October-2020 unless there are additional questions or issues to discuss.