1849364 - SwissSign: Missed revocation and opening Bugzilla

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Mike Guenther]

This is a mis-issuance report for missing a revocation deadline and not opening a corresponding Bugzilla. This Bugzilla is linked to https://bugzilla.mozilla.org/show_bug.cgi?id=1848854
The report is created following the CCADB guidelines (https://www.ccadb.org/cas/incident-report#incident-reports)

1. How your CA first became aware of the problem (e.g., via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP or CCADB public mailing list, a Bugzilla bug, or internal self-audit), and the time and date

As part of the root cause analysis of Bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1848854 we recognized that we missed to revoke mis-issued certificates.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, a document changed, a bug was introduced, or an audit was performed.

all times are UTC (-2h for MESZ)

date / time (UTC)	Action
20221012	We are informed about mis-issued certificates
20221013 - 20221017	We solve the root cause of the mis-issuance
20230818 09:00	Finished root cause analysis of https://bugzilla.mozilla.org/show_bug.cgi?id=1848854. We realize that 'error 1' is a separate issue. From today's perspective we did not trigger our revocation process.
20230818 11:00	Auditor Body is informed
20230818 13:00	Preparation of this Bugzilla

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

The mis-issuance has been stopped as documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1848854

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g., OCSP failures, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help measure the severity of each problem.

The following numbers are included in https://bugzilla.mozilla.org/show_bug.cgi?id=1848854.
Summary: Total: 16; 11 valid, 5 revoked, 0 expired

These numbers are for the specific time period only:
First mis-issuance 20220808 15:22:56 UTC
Last mis-issuance 20221017 14:17:24 UTC

5. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate.

A complete list of the mis-issued certificates identified by SHA256 hash and serial ID is attached to Bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1848854.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As documented in the above mentioned Bugzilla ticket, we concentrated on finding and solving the root cause of the mis-issuance.
From today's perspective it seems that we did not do is to trigger the revocation process. This revocation process does also include the posting of a Bugzilla ticket to inform the community.

As of today we are still looking into the matter why the revocation process was not started. This investigation is still ongoing.

7. List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

The following tasks are ongoing:

• conducting the root cause analysis for this Bugzilla (starting for Monday 21 August 2023)
• Revocation of certificates: latest on Sunday, 20 August 2023, 08:00 UCT as part of Bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1848854

I will update this ticket latest on Wednesday, 23 August 2023

Comment 1

[Mike Guenther]

Update concerning point '6. Explanation …'

Our analysis concentrated on the events after our auditors informed us on 20221012 about the mis-issuances (as documented https://bugzilla.mozilla.org/show_bug.cgi?id=1848854#c2).

Based on the internal audit tickets from that time we are able to reconstruct the following:

• An audit ticket was created to check for the root cause of the mis-issuance.
• The same ticket mentioned the need for revocation.
• This ticket was then assigned to our tech staff who worked on the mitigation of the root cause.
• After this was successfully mitigated the ticket was then re-assigned to another person who checked that the mitigation is working.
• After that the ticket was closed as the involved persons reported an 'all green'

What went wrong?

1. We did not create independent tickets to address the 2 independent issues at hand
   • First ticket: revocation of mis-issuance by triggering the internal mis-issuance process
   • Second ticket: look and mitigate root cause
2. The people involved in the ticket all had an operative stake in the audit/analysis/mitigation: This resulted in split responsibilities. Which then lead to one issue being solved and the other wasn't.

Lessons learned from today's perspective

• Create multiple tickets where appropriate (even if it means more administrative tasks)
• To create the revocation ticket the process as documented in Bugzilla comment 7 (https://bugzilla.mozilla.org/show_bug.cgi?id=1671113#c7) and 9 (https://bugzilla.mozilla.org/show_bug.cgi?id=1671113#c9) will be used not only for external reporting of potential mis-issuances but also for internal and audit related reports of mis-issuance.
  This leads to a single revocation and reporting process for all potential mis-issuances independent of the reporter/reporting channel.

Update concerning point 7 - the certificate revocation:
Revocation is done as posted in https://bugzilla.mozilla.org/show_bug.cgi?id=1848854#c3

I will update this ticket latest Wednesday 30 August 2023

Comment 2

[Mike Guenther]

Update on point '7. List of steps ....'

To create the revocation ticket the process as documented in Bugzilla comment 7 (https://bugzilla.mozilla.org/show_bug.cgi?id=1671113#c7) and 9 (https://bugzilla.mozilla.org/show_bug.cgi?id=1671113#c9) will be used not only for external reporting of potential mis-issuances but also for internal and audit related reports of mis-issuance.
This leads to a single revocation and reporting process for all potential mis-issuances independent of the reporter/reporting channel.

We are currently updating our processes and procedures. We expect to have all finished latest by end of September 2023.

I will update this ticket on a weekly basis even if the next planned reportable action is by the end of September 2023.
Next update, Wednesday, 6 September 2023

Comment 3

[Roman Fischer]

Weekly update: We are still on course to update our processes and procedures.
Next update, Wednesday, 13 Sept. 2023

Comment 4

[Mike Guenther]

Weekly update: Work is progressing fine. Trial runs successful. Looking for further improvement in the process/procedure.
Expected implementation: latest first week of October 2023.
Next update: Wednesday, 13 September 2023

Comment 5

[Mike Guenther]

Next Update is of course next Wednesday 20. September 2023 and not tomorrow

Comment 6

[Roman Fischer]

We now also re-trained all involved parties on the unified revocation process and updated the process documentation.
If there are no open questions from the community, then we would ask for this Bugzilla to be closed.

Comment 7

[Ben Wilson]

I am looking to close this on Friday, 22-Sept-2023.