1820174 - NETLOCK: SSL certificates with OU field

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Tamás Horváth]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36

Steps to reproduce:

We become aware of multiple SSL certificates created by NETLOCK after 01. 09. 2022. contains the OU field.

Actual results:

We created the steps necessary to modify our internal processes and systems to be able to comply with the requirements.

Expected results:

We have changed our policies to reflect the requirement.
We modified the SSL request form, so the customers won't be able to create certificate requests including the OU filed. .
Our internal CA software is under change so there won't be an option to include OU filed in SSL certificates.
We notified our customers who already have such a certificate that we need to revoke their certs and generate a new one without the field. The notification was sent out 09. 02. 2023 and also was published on our website.
The affected certificates will be revoked and a new one will be generated in multiple steps between 01. 03. 2023 and 16. 03. 2023.

Comment 1

[Claves Nostrum]

Few thoughts related to this:

• Will you be posting a full incident report?
• Why did you take a month between noting the issue and posting it on Bugzilla?
• Why did you not notice this for 5 months after the requirement took effect?
• Do you execute quarterly self-audits / reviews against the BR? Why was it not picked up during these?
• What about pre and post issuance linting? It should have been picked up that way as well
• Does the last sentence mean you are not revoking within the timelines set forth by the BR? If so, this requires a separate incident.

Comment 2

[Tamás Horváth]

Attachment: bugzilla_1820174_CRTSH_lst.csv

Comment 3

[Tamás Horváth]

Hello All!

Thanks for your feedback and pointing out the missing informations.

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

We have received an e-mail from Mr. Michael Lettona of DigiCert at 03. 02. 2022. about they discovered that we have issued TLS certificates with OU field present in them.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was performed.

Date Action taken
03. 02. 2023. E-mail notification received
03. 02. 2023. Last TLS certificate was issued with OU field in the Subject.
06. 02. 2023. E-mail was processed by our team, escalation to management regarding to the issue
06. 02. 2023. Check existing applications, immediately stop issuing certificates for all applications with the OU field.
06. 02. 2023. A priority request to the development team to disable the OU field in the application interface.
07. 02. 2023. E-mail response was sent to Mr. Michael Lettona
07. 02. 2023. Collection of already issued certificates containing the OU field.
07. 02. 2023. Notification was sent to the subscribers of the already issued certificates about the issue
07. 02. 2023. Consultations started with clients to mitigate impact.
09. 02. 2023. Planned revocation date (20. 02. 2023) was announced internally and to the customers.
16. 02. 2023. Based on customer feedback, the withdrawal date has been extended to 01. 03. 2023.
23. 02. 2023. One customer responded their system will be updated and configured on 16. 03. 2023. 03:00 AM, therefor the last TLS certificate will be revoked on the same day.
28. 02. 2023. Final software release was deployed to both frontend (registration interface) and to the internal systems (affecting later renewals).

1. 3. 2023. All but one TLS certificate was renewed with no OU field in them.
2. 3. 2023. Bugzilla ticket was published
3. 3. 2023. Last certificate will be renewed.

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Netlock stopped certification issuance right on date the notification was processed by our team.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Only TLS server certificates were affected.

In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

All affected certificates: 389
First issued TLS certificate: 01. 09. 2022.
Last issued TLS certificate: 03. 02. 2023.

CSV file attached.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

NETLOCK conducts regular self-audits, during which 3% of the certificates issued are verified on a quarterly basis, with a random selection of certificates to be verified. The last two audits were carried out in mid-September and mid-December. During the internal audit in mid-December, a random selection of certificates was made for the whole year, which unfortunately did not include any certificates issued in error. Our next internal audit is due at the end of March, which we think would have identified this discrepancy.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

We reviewed our processes and resources.
We have expanded the compliance team with one internal and one external staff member.
We amended the relevant policy to review 5% of certificates issued during quarterly internal audits. Our new policy came into force on 27 February 2023.
Complementing our internal processes, development and change requests from the compliance team are automatically upgraded one notch (from medium to high), avoiding the need to move these requests down in the development schedules.
We have modified our internal procedures to review only the last quarterly expenditure during the last quarterly internal audit instead of the full year.
We have revised our internal process, clarifying the deadline for incident reporting.
We have revised our internal process, to include linting on a regular basis.

Comment 4

[Tamás Horváth]

As of Mr. Claves Nostrum comments

Will you be posting a full incident report?
Yes, sorry about the delay.

Why did you take a month between noting the issue and posting it on Bugzilla?
For some of the clients concerned, the level of use of the certificate (national level, high criticality system) did not allow for an earlier withdrawal of the certificate. As the date of revocation was not defined, the decision was taken to add the bugzilla ticket after the final date had been defined.

Why did you not notice this for 5 months after the requirement took effect?
Hopefully this question was answered in the full incident report, as how we carried out our internal audits.

Do you execute quarterly self-audits / reviews against the BR? Why was it not picked up during these?
Yes, hopefully this question was answered in the full incident report in the self-audit section.

What about pre and post issuance linting? It should have been picked up that way as well
We are currently conducting quarterly internal audits to check the compliance of the certificates issued and we have not done linting on a regular basis. We are changing that also.

Does the last sentence mean you are not revoking within the timelines set forth by the BR? If so, this requires a separate incident.
Yes, indeed, we will create the incident report in the upcoming days.

Comment 5

[Tamás Horváth]

The last three of the timeline got messed up, the correct lines are

1. 3. 2023. All but one TLS certificate was renewed with no OU field in them.
2. 3. 2023. Bugzilla ticket was published
3. 3. 2023 Last certificate will be renewed

Comment 6

[Tamás Horváth]

Yet another try...

01/03/2023. All but one TLS certificate was renewed with no OU field in them.
03/03/2023. Bugzilla ticket was published
16/03/2023 Last certificate will be renewed

Comment 7

[Ryan Dickson]

@Tamás

What about pre and post issuance linting? It should have been picked up that way as well
We are currently conducting quarterly internal audits to check the compliance of the certificates issued and we have not done linting on a regular basis. We are changing that also.

Can you please share more detail regarding the planned linting implementation described in Comment 4?

• Provide a more specific set of remaining actions and timebound milestones related to the planned deployment.
• Describe whether the lints will be custom built, or whether Netlock is planning to use an open-source solution with wide community support and regular updates in response to changes to the BRs and Web PKI incident (e.g., ZLint).
• How do you intend on verifying the linting implementation is working as intended (i.e., do you intend on running the lints against the certificates disclosed in Netlock's previous bugs (like this one) to ensure known issues are detected?)
• Will Netlock commit to running the lints against the complete certificate corpus in effort to detect other potential issues not previously reported?
• How do you intend to maintain existing lints, or add new ones, in the future?

Comment 8

[Tamás Horváth]

Hello Ryan!

For now, we plan to use the *lint checks on crt.sh, checking the certificates in the chain on a monthly basis. We plan to do the checks manually, with the compliance team in charge. The first check will take place this week.

We do not want to develop our own custom solution, so in parallel with the above we are investigating how to automate the checks on e.g. crt.sh, or how to have a report of the weekly run that is displayed to the compliance team, checking the status of the certificates in addition to the success of the run. We plan to do this work in the next 60 days.

In addition to automated checks, we will introduce manual checks for certificates into the quarterly manual check cycle. This task is planned to be performed by the RO team, who will then have a comprehensive view of this aspect. The next audit is due this month.

Comment 9

[Ben Wilson]

Hello Tamás,
Please provide an update on this matter.
Thanks,
Ben

Comment 10

[Tamás Horváth]

Hello All!

All certificates were renewd by the defined deadline.
The internal policy was published, internal training of the changes were conducted.
Netlock's central software was updated and rolled out which does not allow TLS certificates to include OU filed.
Linting processes were establised, now on weekly and manual basis. The automatic checking is still under investigation, how it can be implemented in the software level to be able to check it on the fly and not needed to do it afterwards / manually.

Best, Tamas

Comment 11

[Tamás Horváth]

Hello!

Do you have any questions left regarding this issue that we can answer?

Tamas

Comment 12

[Tamás Horváth]

Hello, I believe if there is no other questions we can close the ticket.

Tamas

Comment 13

[Ben Wilson]

It doesn't appear there are any other questions, so I'll close this ticket.