Open Bug 1889570 Opened 3 months ago Updated 17 days ago

NETLOCK: Policy Qualifiers other than id-qt-cps is included in TLS certificates

Tracking

(Not tracked)

Status:

ASSIGNED

People

(Reporter: horvath.tamas2, Assigned: horvath.tamas2, NeedInfo)

References

(Blocks 1 open bug)

Details

(Whiteboard: [ca-compliance] [ev-misissuance])

Tamás Horváth

Assignee

Description

•

3 months ago

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36

Steps to reproduce:

NETLOCK was notified on 3 April 2024 that one of our TLS certificate (https://crt.sh/?id=11261465680), signed by our "NETLOCK Trust EV CA 3" (https://crt.sh/?id=11261465680 ) intermediate certificate gives an error on the zlint check (https://crt.sh/?id=11261465680&opt=zlint).

We started the investigation in due course, to identify the scope and extent of the incident.

This is a preliminary incident report. We will provide the detailed update as soon as possible.

Ben Wilson

Updated

•

3 months ago

Assignee: nobody → horvath.tamas

Status: UNCONFIRMED → ASSIGNED

Type: defect → task

Ever confirmed: true

Whiteboard: [ca-compliance] [ev-misissuance]

Ben Wilson

Updated

•

3 months ago

Summary: NETLOCK - Policy Qualifiers other than id-qt-cps is included in TLS certificates → NETLOCK: Policy Qualifiers other than id-qt-cps is included in TLS certificates

Tamás Horváth

Assignee

Comment 1

•

3 months ago

Incident Report

Summary

NETLOCK was notified on 3 April 2024 that one of our TLS certificate (https://crt.sh/?id=11261465680), signed by "NETLOCK Trust EV CA 3" (https://crt.sh/?id=11261465680 ) intermediate certificate gives an error on the zlint check (https://crt.sh/?id=11261465680&opt=zlint).
The investigation of the certificate above stated that the issued TLS certificates were created with the following User Notice field
Explicit Text: EV SSL certificate. Terms of service at: http://www.netlock.hu/html/dok.html value which is against BRG 2.0 since Sept. 15, 2023.

Affected intermediate issuers, from where the misissued end-user certificates were signed

NETLOCK Trust EV CA 3
NETLOCK Trust Qualified EV CA 3
NETLOCK DVSSL CA

After the initial investigation NETLOCK started the customer communication and planned the revocation of the affected certificates, as described in the Timeline section of the IR.

Impact

According to the investigation it became clear that all of the issued certificates contained the User notice field which is prohibited.
The investigation pointed out that this policy identifier does not affect nor the security, nor the usability of the issued certificates.

Timeline

All times are CEST

Action timestamp	Action taken
2024-04-03 10:52	E-mail arrived to our central address (compliance@netlock.hu)
2024-04-03 16:03	Initial investigation started
2024-04-03 17:11	Reply was sent to the reporter confirming the issue
2024-04-03 19:30	Most of the affected end-user certificates were identified
2024-04-04 01:18	Internal bug report was created on the technical investigation
2024-04-04 10:30	Planning of customer communication started
2024-04-04 11:30	New certificate profile configuration released, and test certificate was issued and checked on crt.sh
2024-04-05 10:00	Direct and undirect customer communication begun on about the error
2024-04-05 10:15	New certificate profile configurations were released on all production systems
2024-04-05 11:00	2024-04-07 22:30 – All new certificates were generated
2024-04-05 12:30	Customer responses arriving regarding the new certificates – either accepting the new cert and the expected change date, or objections (Second ticket is raised here:https://bugzilla.mozilla.org/show_bug.cgi?id=1891331)
2024-04-06 10:00	Revocations started
2024-04-09 13:21	All misissued certificates not requested to be revoked later by the customers (FIXME BUGZILLA ISSUE) were revoked
2024-04-30 23:59	Certificates requested to be revoked later will be revoked by 2024-04-09 13:21

Root Cause Analysis

The issue originated from the change of the BRG last year. NETLOCK’s staff investigated the changes in the profiles, but unfortunately the change regarding the policy identifier was not updated.

Lessons Learned

This issue pointed out that our internal systems monitoring should be improved to catch such issues more affectively. NETLOCK has worked hard in the past period on monitoring external and internal data regarding certificate issuance.

What went well

The changes in our internal processes let us handle this issue on a timely manner and we were able to meet the deadlines required by the BRG and Mozilla.

What didn't go well

As this Some of our customerslients used the relatedissued certificates in such systems, where the changing of the certificatese cannot be implemented done in such a short period of time.

Where we got lucky

The security of the certificates was not affected by the misissuance. Therefore NETLOCK decided to extend the revocation period for customers of major importance for the national economy.

Action Items

Action Item	Kind	Due Date
Fix of certification profiles	Repair	2024-04-09
Revoke affected certificates	Repair	2024-04-09
Revoke remaining affected certificates (based on customer requests)	Repair	2024-04-30
Extend monitoring capabilities (linting functionality and post checks)	Remediate	2024-05-31

Tamás Horváth

Assignee

Comment 2

•

3 months ago

List of certificates

amir

Comment 3

•

3 months ago

Why was there a 10 day delay between receiving the report and filing an incident?

Ryan Dickson

Comment 4

•

3 months ago

Bug 1875942 was filed on January 22, 2024, and demonstrated the same mis-issuance we observe disclosed in this report (i.e., certificates issued with Policy qualifiers other than id-qt-cps).

How is Netlock monitoring Bugzilla such that it is able to identify potential issues of its own and remediate them in a timely manner?
Can you explain what failures took place that resulted in Netlock failing to identify it was mis-issuing certificates in a manner that was consistent with Bug 1875942?
What action(s) does Netlock intend to take such that this same mistake is not repeated?

Flags: needinfo?(horvath.tamas2)

Tamás Horváth

Assignee

Comment 5

•

3 months ago

(In reply to amir from comment #3)

Why was there a 10 day delay between receiving the report and filing an incident?

Hello Amir!

The initial report was prepared according the requirements. In the meantime, Netlock identified the affected certificates, contacted the owners of them regarding this issue. The partner's feedback took a long time because we asked the partner to indicate in which system the certificate was integrated and how it can be replaced.
In light of this, we have decided to wait with the full incident report and also with the delayed revocation report until we settled all the steps and the final revocation date.

Tamas

Flags: needinfo?(horvath.tamas2)

Tamás Horváth

Assignee

Comment 6

•

3 months ago

(In reply to Ryan Dickson from comment #4)

Hello Ryan!

Bug 1875942 was filed on January 22, 2024, and demonstrated the same mis-issuance we observe disclosed in this report (i.e., certificates issued with Policy qualifiers other than id-qt-cps).

How is Netlock monitoring Bugzilla such that it is able to identify potential issues of its own and remediate them in a timely manner?

We monitor newsletters and requirement changes with an external consultant. The consultant shares the changes with the NETLOCK compliance team in an informative manner, who coordinate with operations and development.
The changes undergo verification, and if necessary, development initiates planning alongside compliance checks. Implementation of changes is handled by operations, while regulatory adjustments are made by compliance. Subsequently, requirement changes are introduced into the annual change tracking list.

Can you explain what failures took place that resulted in Netlock failing to identify it was mis-issuing certificates in a manner that was consistent with Bug 1875942?

Regrettably, due to human oversight, the defined issue in the ticket evaded our attention, thus preventing NETLOCK from identifying the specified error within its own systems.

What action(s) does Netlock intend to take such that this same mistake is not repeated?

Collaboration with the external consultant was reviewed, and the change tracking procedure was updated. Initiating change tracking and creating a JIRA ticket related to change tracking is on a tight deadline. This enhances the traceability and transparency of change management and the closure of tasks and processes completed. Additionally, the configuration of the applied linter will be reviewed.

Best, Tamas

Tamás Horváth

Assignee

Comment 7

•

2 months ago

Hello All!

All related certificates have been revoked.

Best, Tamas

Ben Wilson

Comment 8

•

2 months ago

I think this implies that the incident can be closed. Are there any other questions or requests? If not, I will close this on Friday, 3-May-2024.

Flags: needinfo?(bwilson)

Wayne

Comment 9

•

2 months ago

There's an outstanding Action Item before this incident can be closed:

Action Item Kind Due Date

Extend monitoring capabilities (linting functionality and post checks) Remediate 2024-05-31

Ben Wilson

Comment 10

•

2 months ago

Then I'll clear my "needinfo".

Flags: needinfo?(bwilson)

Ben Wilson

Updated

•

2 months ago

Blocks: 1891331

Wayne

Comment 11

•

1 month ago

(In reply to Wayne from comment #9)

There's an outstanding Action Item before this incident can be closed:

Action Item Kind Due Date

Extend monitoring capabilities (linting functionality and post checks) Remediate 2024-05-31

Has this action item been completed by 2024-05-31?

Flags: needinfo?(horvath.tamas2)

Tamás Horváth

Assignee

Updated

•

1 month ago

Flags: needinfo?(horvath.tamas2)

Wayne

Comment 12

•

23 days ago

Comment #11 hasn't been answered within 7 days.

Flags: needinfo?(horvath.tamas2)

dr. Nagy Nikolett

Comment 13

•

17 days ago

Hello Wayne,

Yes, our monitoring now covers the checking of TLS end-user certificates using zlint, x509lint and ca/b lint.

You need to log in before you can comment on or make changes to this bug.