1822809 - NETLOCK: SSL certificates with OU field - revocation delay

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Tamás Horváth]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36

Steps to reproduce:

The certificates affected by the incident described in https://bugzilla.mozilla.org/show_bug.cgi?id=1820174 was not revoced within the time frame defined in the BR.

Actual results:

Based on a business decision Netlock has waited to revoke the certificates based on our customer's requests.

Expected results:

The affected certificates should have been revoked within 5 days.

Comment 1

[Tamás Horváth]

How your CA first became aware of the problem (e.g/via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

The bugzilla ticket for the original incident is available at the following link
https://bugzilla.mozilla.org/show_bug.cgi?id=1820174
Based on customer feedback, it became clear on 10/03/2023 that we would not be able to meet the 5 day revocation deadline.

A timeline of the actions your CA took in response/A timeline is a date-and-time-stamped sequence of all relevant events/This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was performed.

Date Action taken
16/02/2023. Based on customer feedback, the revocation date has been extended to 01/03/2023.
23/02/2023. One customer responded their system will be updated and configured on 16/03/2023/03:00 AM, therefor the last TLS certificate will be revoked on the same day.
01/03/2023. All but one TLS certificate was renewed with no OU field in them.
16/03/2023. Last certificate will be renewed.

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident/A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Netlock stopped certification issuance right on date the original notification was processed by our team.

In a case involving certificates, a summary of the problematic certificates/For each problem: the number of certificates, and the date the first and last certificates with that problem were issued/In other incidents that do not involve enumerating the affected certificates (e.g/OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified/This will help us measure the severity of each problem.

Only TLS server certificates were affected.

In a case involving TLS server certificates, the complete certificate data for the problematic certificates/The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem/It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise/When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate/In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

All affected certificates: 4
The attached CSV only contains the last 4 certificates related the to the same customer. Only one TLS certificate was not yet revoked.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The ticket cited earlier illustrates the original problem. After notifying customers, feedback indicated that there were several certificates that the business domain using them or the supporting IT could not support the earlier revocation, so after weighing the risks, the business decision was made to extend the revocation deadline to allow for the revocation of the defective certificates to benefit our customers.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future/The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

As described in the original ticket, we intend to shorten the detection time in order to ensure that customer communications and revocations related to certificates that may have been issued incorrectly can be done as soon as possible.

Comment 2

[Tamás Horváth]

Attachment: bugzilla_HPO_remaining.csv

The last certificate was revoked today.

Comment 3

[Chris Clements]

@Tamás

After notifying customers, feedback indicated that there were several certificates that the business domain using them or the supporting IT could not support the earlier revocation

Can you share some of the reason(s) this subscriber could not support the revocation timeline described in the BRs?

As described in the original ticket, we intend to shorten the detection time in order to ensure that customer communications and revocations related to certificates that may have been issued incorrectly can be done as soon as possible.

The root cause for this report, as stated, was subscribers not being able to support timely revocation. Are there other remediation actions that can be performed to avoid delayed revocation in the future?

Additionally, can you elaborate on your remediation action(s) to include the status of each action and the date each action will be completed?

Comment 4

[Tamás Horváth]

(In reply to Chris Clements from comment #3)

@Tamás
Hello Chris!

After notifying customers, feedback indicated that there were several certificates that the business domain using them or the supporting IT could not support the earlier revocation

Can you share some of the reason(s) this subscriber could not support the revocation timeline described in the BRs?

After sending the initial revocation notice, we received a request from the client that their system using the certificate is a high priority and their official maintenance window is March 16, when they can replace the certificate, so they request that the revocation only take place then. There has been no further technical consultation on this matter, we do not know which system or systems were affected by this request.

As described in the original ticket, we intend to shorten the detection time in order to ensure that customer communications and revocations related to certificates that may have been issued incorrectly can be done as soon as possible.

The root cause for this report, as stated, was subscribers not being able to support timely revocation. Are there other remediation actions that can be performed to avoid delayed revocation in the future?

In my view, there are cases of key withdrawal that are mandatory (e.g. key compromise), but there may be low-risk cases where concessions are necessary for business reasons.
The solution, as I see it, is rather to set up automatic control mechanisms for both critical and non-critical cases, which will flag malfunctions as soon as possible, thus avoiding situations where business interests may be harmed in the mandatory withdrawal procedure - speeding up both detection and response. This is what we are working on currently.

Additionally, can you elaborate on your remediation action(s) to include the status of each action and the date each action will be completed?

Action: Manual process implementation – weekly checking of statuses on crt.sh
State: Finished
Deadline: 16/03/2023

Action: Collection of business requirements
State: In progress
Deadline: 15/04/2023

Action: Technical specification
State: Not started
Deadline: 30/04/2023

Action: Implementation
State: Not started
Deadline: 31/05/2023 (planned)

Comment 5

[Ben Wilson]

Hello Tamás,
Please provide a status on your efforts.
Thanks,
Ben

Comment 6

[Tamás Horváth]

Hello All!

Tha manual, weekly controll process was implemented and it is still in progress.

The requirements for the automatic checks are defined, and most of the technical requirements are collected, but it is still under investigation if it will be implemented in netlock's current or the new backend software. We won't be implementing it by the end of May.

Best, Tamas

Comment 7

[Tamás Horváth]

Hello!

The technical requirements have been collected. Monitoring (C based linter) has been technically implemented in the backend software already.

The other automated checks available are still being investigated, on how we can run them on our current environment.

We are planning to integrate Go based linting at the earliest 31/08/2023.

Tamas

Comment 8

[Tamás Horváth]

Hello,

LINTING was implemented in our core software and the external monitoring tool was chosen and implemented (uptimerobot).

If there is no other questions or concenrns, I believe we can close this issue.

Tamas

Comment 9

[Ben Wilson]

I will close this bug on Friday, 29-Sept-2023.