1905509 - NETLOCK: CPR was not responded to in 24 hours

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne]

On 2024-06-28 19:45 UTC the below email was sent to '[e-mail]' - the contact listed in 1.5.2 of every one of NETLOCK's CP/S. No reply has been received within 24 hours. Below is a copy of the email for transparency:

Hello,

This is the email listed in the current CPS covering the 'NetLock Arany (Class Gold) Fotanúsítvány' root.

1: This is not listed on CCADB's Problem Reporting Mechanism's for NETLOCK which only provides:

• visszavonas[at]netlock[dot]hu
• https://netlock.hu
• compliance[at]netlock[dot]hu

Neither email address appears in any CPS I can see - this needs corrected.

2: This Certificate Problem Report has been filed due to a certificate failing both zlint and pkilint: >https://search.censys.io/certificates/51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496

$ lint_cabf_serverauth_cert lint -s ERROR -d 51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496.pem
SubscriberKeyUsageValidator @ certificate.tbsCertificate.extensions.1.extnValue.keyUsage
cabf.serverauth.subscriber_prohibited_ku_present (ERROR): Prohibited KU present: keyEncipherment
CertificatePolicyQualifierValidator @ certificate.tbsCertificate.extensions.5.extnValue.certificatePolicies.1.policyQualifiers.1
cabf.serverauth.prohibited_certificate_policy_qualifier_type (ERROR): Prohibited qualifier type: 1.3.6.1.5.5.7.2.2

$ zlint 51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496.pem
e_ecdsa_allowed_ku, e_mp_ecdsa_signature_encoding_correct e_policy_qualifiers_other_than_cps_not_permitted

Please note that zlint is reporting an additional error.

• Wayne

Now, finding out who to contact is an issue. The policy IDs listed in the certificate are generic, and the cpsuri gets redirected to a very generic page. Checking just the Root isn't helpful as every CP/S mentions it. Luckily every CP/S has the same contact method listed for issues:

1.5.2. Contact person of the document
The responsible contact person of the Policy Adopting Authority shall be the approver of the present document (see the cover page of the document).

Customers, End Users and the Relying Parties may submit their questions and comments related to the present document to the NETLOCK Policy Adopting Authority in e-mail to [e-mail].

As mentioned in the email... this isn't what is listed on CCADB, however the CP/S should be the authority relied on?

Now figuring out which CP/S is applicable was a headache. Which is to say that having checked CCADB I do not believe this intermediary is disclosed. It also does not appear in a single CP/S directly.
https://search.censys.io/certificates/d0eb908401f33242602634afd51991536b3ad7aa901586fdeb4955fe51b0e419
CN: NETLOCK TLS OV ECC CA
SHA256: d0eb908401f33242602634afd51991536b3ad7aa901586fdeb4955fe51b0e419

Looking further this is at least a repeat of #1889570 but for a freshly generated intermediary. Note that the undisclosed intermediary is different than the one stated in #1904041. Given the naming scheme I would not be shocked at more undisclosed intermediaries, but I have also not checked.

The lack of any obviously applicable CP/S for these intermediaries also raises a question I previously posed in #1891331 Comment 28. If no CP/S is covering these intermediaries and the CP/S isn't considered in-force until 30 days have past after the CP/S has been publicly published... what happens, and how do we tie a certificate to a CP/S?

As I see it this is a certificate published that isn't tied to any CP/S, and is from an undisclosed intermediary.

Please note this incident is purely for handling the lack of response from the CPR. Any other issue mentioned will need a separate incident raised by the CA.

Comment 1

[Aaron Gable]

The contact address provided in Section 1.5.2 is for communicating with the people who administer the CPS document itself. It is not necessarily the correct address for submitting a Certificate Problem Report, and since it is not an address disclosed for that purpose, it is not necessarily an incident for a CA to fail to respond to a report there. (For example, sending a message to the PMA physical mailing address that Let's Encrypt discloses in our Section 1.5.2 would not be a valid CPR.)

The address for submitting a Certificate Problem Report is disclosed in CCADB. I believe it should also be disclosed in Section 4.9.3 of the CPS, where the CA is supposed to describe the procedure for submitting a Certificate Problem Report.

Comment 2

[Corey Bonnell]

BR 4.9.3 requires that the CA’s problem reporting mechanism be disclosed in section 1.5.2 of the CPS.

If the only contact information in section 1.5.2 is the email address quoted above, then I agree with the reporter that they attempted to contact the CA using the correct reporting mechanism.

Comment 3

[Aaron Gable]

Ah you're right, I had forgotten that BRs 4.9.3 places requirements on CPS 1.5.2 (an anti-pattern I'd like to fix). It's not obvious to me that there's a failure-to-respond incident here, since it is clear from reading NetLock's Section 1.5.2 and their CCADB disclosures that this address is not intended for receiving CPRs (though I am not opposed to that interpretation of events either). But there definitely does appear to at least be a failure of their CPS to comply with the requirements of BR 4.9.3.

Comment 4

[amir]

I would consider the CCADB database, and the CPS of a CA both as sources of truth regarding CPR endpoints. So in regards to that, and the fact that this is the only email mentioned in section 1.5.2 of the CPS, there is a failure to respond here.

Furthermore, this incident is detailing that none of the documents take ownership over this intermediate - which might even mean no CPS applies to it?

I can confirm that CCADB does not have this entity listed https://ccadb.my.salesforce-sites.com/mozilla/PublicAllIntermediateCerts (nothing under "NETLOCK TLS OV ECC CA")

Beyond that, this CA has issued the following certificate: https://search.censys.io/certificates/51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496 that is failing 2 lints:

e_mp_ecdsa_signature_encoding_correct
n_ecdsa_ee_invalid_ku

There also seems to be a bunch more undisclosed intermediates that's showing up on Censys: parsed.issuer.common_name="NETLOCK*ECC*". There may even be more that's not indexed by censys/not found by that query.

For example: https://crt.sh/?id=13467134365

---

In a larger scale discussion, I do hope that root programs can consider making a requirement for CAs to explicitly use the terms "Certificate Policy" and "Certification Practice Statement" in their documents so we're not having to distinguish what word actually means what. In the case of Netlock here, the document is called "Service Practice Statement".

Comment 5

[Dimitris Zacharopoulos]

Sectigo has built a nice query that shows undisclosed intermediates in https://crt.sh/mozilla-disclosures#undisclosed.

Comment 6

[Rob Stradling]

There also seems to be a bunch more undisclosed intermediates

See bug 1904041.

Comment 7

[Nikolett]

Hello everyone!

Thank you for the ticket.

1. • We have created a new bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1906115 inconnection with the contact informations. We will update Section 1.5.2 accordingly.
2. • About the certification program:

User Notice: We are introducing new ECDSA certificates alongside the existing RSA certificates. The ECDSA certificates will operate on a brand new system, which requires new configuration. The new system is still under configuration. Due to an administrative error, an incorrect user notice was used in the configuration. We have identified the error, and the faulty certificates will be revoked in advance, with new certificates issued. Our RSA system does not use the faulty user notice, and we are making efforts to configure our certificates according to the existing regulations.

Key Usage: As we mentioned before, our previous system used the RSA algorithm. Due to an administrative error, we used 'Key Encipherment' instead of 'Key Agreement'. We investigated the problem and found that, unfortunately, all ECC TLS certificates were issued with this error. As a result, we revoke the faulty certificates and issue new ones.

Policy:
We used the 1.3.6.1.4.1.4146.1.1 GlobalSign policy for EV certificates instead of the general EVTLS 0.4.0.2042.1.4 policy. This configuration error was due to an administrative mistake, tracing back to the previous RSA system. We have corrected the configuration, we revoke the incorrect certificates, and will issue new ones.

The linters previously created in the RSA system were also adapted to the ECC system, as a result of which similar errors will not occur in the future.

As for the CN: NETLOCK TLS OV ECC CA
SHA256: d0eb908401f33242602634afd51991536b3ad7aa901586fdeb4955fe51b0e419
The CP/S change is in progress. You have not been able to see the CN in our current applicable policy version, because an update is in preparation, it will come into force on the 18th of July.

On our website we have a dedicated place for the drafts of the CP/S where you can see them in a 30 day period:
https://netlock.hu/drafts#others

Comment 8

[Ben Wilson]

Was this other bug mistakenly created - Bug #1906115?

Comment 9

[Wayne]

Confirming I have now received an email reply (14:59 UTC), and also a duplicate seems to have been created. Roughly 4 days, 19h turnaround period.

Please consider incidents for the other issues and a full list of impacted certificates. There is also the problem of whether any CP/S covered those even if it is in a 'drafts' section of netlock's website.

Comment 10

[Nikolett]

Incident report

Summary

On Friday 2024. 06. 28. 21:45 CET NETLOCK has received an e-mail about the non-compliant e-mail addresses in CCADB as well as in TSP’s CP/S.
The e-mail also disclosed a problem with a certificate failing both zlint and pkilint making the e-mail a certificate problem report. We were notified of the creation of the corresponding bug ticket at Saturday 2024. 06. 29. 21:46 CET.

Impact

NETLOCK’s lack of timely response did not meet the Baseline Requirements for the Issuance and Management of Publicly Trusted TLS Server Certificates, section 4.9.5, which states “within 24 hours after receiving a Certificate Problem Report, the CA SHALL investigate the facts and circumstances related to a Certificate Problem Report and provide a preliminary report on its findings to both the Subscriber and the entity who filed the Certificate Problem Report.”

Timeline

Action timestamp	Action taken
2024-june-28 21:45	From [e-mail] the e-mail arrived to our central address ([e-mail])
2024-june-29 21:46	Bugzilla ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1905509 was created by [e-mail] (Wayne) because of the lack of response.
2024-july-01 11:21	Compliance asked for support from the IT Department and an investigation started, assessing the impact of the indicated configuration changes on certificate issuance and infrastructure
2024-july-02 09:15	Following the operational meeting, we implemented a linter configuration refreshment. Testing of the systems continued.
2024-july-03 16:05	NETLOCK has created a new Bugzilla ticket #1906115. We see now that this is a duplicate and this action was not necessary.
2024-july-03 16:59	NETLOCK responded to the original email from [e-mail].
2024-july-04 09:15	Testing continued and cert configuration changed.
2024-july-05 14:11	We have received confirmation of the revocation of the certificates:
2024-july-05 14:36	NETLOCK’s CP/S 1.5.2 was updated with the following: In the event of suspected private key compromise, certificate misuse or other types of fraud, compromise, abuse, misconduct, or any other matter relating to Certificates, you may contact the Service Provider at [e-mail].

The revoked certificates are the following:

• tlseveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 11:57:28
• tlsoveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 14:05:11
• tlsdveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 13:35:57
• tlsqeveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 13:42:06
• Certificates with the "expired" domain name expired with a validity period of 1 day on 2024.06.21.

Root Cause Analysis

Due to human error the certificate configuration was not correct.

Lessons Learned

In the event of an algorithm change, the previous settings and the certificates issued with the new algorithm will be subject to a broader review.

What went well

The changes in the certificate configuration let us handle this issue on a timely manner and we were able to meet the deadlines for revocation required by the BRG and Mozilla.

What didn't go well

In the new system, the certificate configuration was not correct due to human error.

Where we got lucky

These wrongly configurated certificates completed the testing phase for the ECDSA TLS. No service was provided to customers therefore no external communication and other actions were needed.

Action Items

Action Item	Kind	Due Date
Fix of certification profiles	Repair	2024-07-04
Revoke affected certificates	Repair	2024-07-05
Contact information update in CP/S	Repair	2024-07-05

Comment 11

[Wayne]

Please read the CCADB Incident Response and Mozilla Responding to an incident page.

I cannot list how many things are wrong in this report, it is truly groundbreaking.

e: To be very kind at the absolute minimum:

• CET instead of UTC.
• The mixture of date/time formats
• The impact section
• Mentioning the email address of the reporter
• The report not being about the CPR at all, it is about different incidents that all need separate bugs raised here
• A certificate list with no sha256 hashes
• The Root Cause Analysis section
• The 'lessons learned', repeatedly leaning on human error
• The wrong category for action items and how it is also irrelevant on many layers

We haven't even addressed all of the intermediaries created and not disclosed, then certificates generated off of them failing multiple linters... so much has gone wrong here.

Comment 12

[Dorottya]

Thank you very much for the above feedback, we correct it.

Comment 13

[Dorottya]

Corrected Incident report

Summary
On Friday June 28th 2024. 19:45 UTC NETLOCK has received an e-mail about a problem with a certificate failing both zlint and pkilint making the e-mail a certificate problem report. We apologize for displaying the reporter's email address. We kindly ask the admin to remove it.

Impact
The incident affected the following certifications:

Certification SHA256 fingerprint Revoked/expired
tlseveccca.valid.ecc.netlock.hu 0E:6D:53:3B:CD:21:B6:D7:F4:D3:AD:19:99:18:D6:E9:73:07:E7:E1:A5:E9:0E:2E:15:71:39:55:8F:C4:58:C4 July 5 2024
tlsoveccca.valid.ecc.netlock.hu A2:7C:63:AD:A3:54:08:85:FF:15:E6:A5:ED:61:EB:DE:66:96:CF:0E:1C:FE:18:81:54:32:D5:03:49:52:CD:F8 July 5 2024
tlsdveccca.valid.ecc.netlock.hu 1F:79:86:F4:FB:C9:FB:7E:39:5D:7B:8E:DC:5F:C9:8B:D4:0F:C0:3E:1F:7F:B2:A3:02:0C:E6:1E:E3:B9:F0:32 July 5 2024
tlsqeveccca.valid.ecc.netlock.hu 9D:E0:B5:30:C4:9A:F1:D8:C7:15:1C:E3:3E:46:D5:D1:FB:C8:75:84:5E:11:6C:1D:FF:BF:E9:44:50:F2:4D:23 July 5 2024
tlseveccca.expired.ecc.netlock.hu 67:E4:0E:0C:AC:5D:55:A8:8F:71:D4:D2:02:E3:0B:EE:83:65:7A:17:73:7A:B6:22:39:D0:A4:10:C4:A0:A8:A6 June 21 2024
tlsoveccca.expired.ecc.netlock.hu F3:95:5A:62:62:55:61:D4:5E:37:1F:35:C6:2D:23:E4:02:23:5C:18:E8:A3:B4:95:23:93:AA:0F:9A:6A:CF:C4 June 21 2024
tlsdveccca.expired.ecc.netlock.hu 81:38:DF:47:DC:FB:A1:C4:32:8F:72:CD:69:71:8B:7D:2B:F7:9D:30:FC:86:35:BC:E7:52:A7:FF:71:E7:76:02 June 21 2024
tlsqeveccca.expired.ecc.netlock.hu 2E:B3:CD:38:B7:45:27:1E:74:42:9D:17:14:EA:21:D4:2C:8E:9E:CD:24:96:F2:40:A6:6D:6F:81:6A:26:79:0B June 21 2024

Out of the above, the last four certificates with 'expired' domain names expired within 24 hours of issuance. The provider should have filtered these certificates upon issuance and interrupted the issuance or filtered the erroneously issued certificates and revoked them ASAP, or within not later than 24 hours.

Timeline
Action timestamp Action taken

June 29th 2024 19:46 UTC Bugzilla ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1905509 was created by Wayne
July 1st 2024 09:21 UTC Compliance asked for support from the IT Department and an investigation started, assessing the impact of the indicated configuration changes on certificate issuance and infrastructure
July 2nd 2024 07:15 UTC Following the operational meeting, we implemented a linter configuration refreshment. Testing of the systems continued.
July 3rd 2024 14:59 UTC NETLOCK responded to the reporter
July 4th 2024 07:15 UTC Testing continued and cert configuration changed.
July 5th 2024 12:11 UTC NETLOCK IT confirmed the revocation of the certificates.
.
The revoked certificates are the following:
• tlseveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 09:57:28 UTC
• tlsoveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 12:05:11 UTC
• tlsdveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 11:35:57 UTC
• tlsqeveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 11:42:06 UTC
• Certificates with the "expired" domain name expired with a validity period of 1 day on June 21st 2024

Root Cause Analysis
The TSP intended to issue new TLS certificates with the ECDSA algorithm. The TSP had not previously issued ECDSA-based certificates, so their configuration took place prior to the issuance of the mentioned certificates. Certificates previously issued with the RSA algorithm were issued in a different system, so the configuration data was manually set by our staff. During configuration, due to human error in several cases, the 'user notice,' 'key usage,' and 'policy' fields were incorrectly configured and went unnoticed. The lint check in the ECDSA system at issuance built into the RSA system also not yet worked due to configuration errors, so the error was not detected before the certificate issuance. During the investigation of the error, we identified the above causes that led to the issuance of erroneous certificates and the lack of discovery and revocation.

Lessons Learned

What went well
Among the issued certificates, those with 'expired' domain names expired within 24 hours of issuance, thus losing their validity within 24 hours. The error contributed to the provider reconfiguring and correctly reissuing the new type of certificates.

What didn't go well
Unfortunately, the system configuration check was insufficient, and as a result, the necessary lint configuration built into the issuance was also insufficient. Provider had to refresh the configuration of the implemented linter. Consequently, the provider could issue certificates with multiple configuration errors.

Where we got lucky
The incorrectly configured certificates were issued during the ECDSA testing phase. Not a high number of erroneous certificates were issued. No customer service was associated with them, and external communication or other activities regarding the revocation were not necessary. The certificates took part in the TSP’s testing phase.

Action Items
TSP reviewed it’s processes and resources. To avoid human error, we have configured automated lint checks in the issuing system, which automatically identify incorrect certificate configurations during the issuance process. On July 15th 2024 and July 30th 2024 we are addressing the human-related issues through more frequent and detailed training on technical configurations.

Comment 14

[Wayne]

This is an incident about a failed 24h response to CPR. If you intend to raise incidents for the rest of the issues please raise them separately.

Please look at any other recent incident as a reference for your incident reports...

Comment 15

[Dorottya]

Dear Wayne, absolutely true, the subject of nr 1905509 ticket does not match the uploaded incident report. We are creating a separate bug ticket for this Certificate Problem Report. Thank you very much. In this case the nr 1906115 ticket would you please close or should we handle it in paralell as well?

Comment 16

[Nikolett]

Attachment: Incident report

Comment 17

[Nikolett]

Incident Report

Summary

Wayne e-mailed us on June 28th 2024 19:45 UTC in accordance to multiple issues (NETLOCK CP/S, certification problems). The e-mail was sent to [e-mail], e-mail was delivered to NETLOCK at the time mentioned before. But the e-mail was not responded within 24 hours.

Impact

The reported issue did not directly affect any certificate, although the email contained a certificate issue, for which the service provider will open a separate ticket. The service provider did not respond to the email notification within 24 hours. The service provider reviewed the emails received in the past quarter addressed to the email address specified by the reporter and determined that there were no other notifications that were not responded to in a timely manner.

Timeline

All times are UTC

Action item	Action taken
2024-06-28 19:45	Wayne e-mailed to NETLOCK and didn’t get answer within 24 hours
2024-06-29 19:45	Wayne opened new ticket for the lack of respond within 24 hours
2024-07-03 14:51	NETLOCK commented the ticket opened by Wayne
2024-07-03 14:59	NETLOCK answered Wayne’s e-mail

Root Cause Analysis

The service provider investigated the issue related to the received email notification and determined that in the past quarter, there had been no instance where inquiries sent to the email address specified in the ticket were not responded to in a timely manner. The email was received; however, due to human oversight, none of the relevant staff responded to it. The lack of response can be attributed to several factors. The relevant staff members reviewed the email but deemed that it did not require a response within 24 hours, as the [e-mail] email address is maintained by the service provider for CP/S-related questions and inquiries. Consequently, the necessary work to respond to the email did not begin until July 1st, 2024.

Lessons Learned

What went well

The issue does not affect the provider’s processes; no new processes need to be established nor any faulty processes corrected. The provider has the appropriate staff and expertise to address the issue, which is an isolated incident.

What didn’t go well

The provider's staff was not sufficiently attentive to the received email and did not escalate the issue because they did not recognize that the response time was limited to just 24 hours.

Where we got lucky

The provider reviewed recent inquiries sent to the email address and found no other requests that were not responded to in a timely manner. The provider has an adequate process for handling such inquiries.

Action Items

On July 8th, 2024, the provider trained the relevant staff on the response time requirements for inquiries received at the specified email address. The provider emphasized to staff the process for escalating inquiries related to faulty certificates to the supervisor and ensuring responses to the reporter are not delayed.

Comment 18

[Ben Wilson]

Netlock needs to address and/or explain what additional action items are being taken to improve response times when processing Certificate Problem Reports. The current Incident Report is insufficient in this regard.

Comment 19

[Nikolett]

Hello everyone,

Thank you for the latest comment, we will explain the additional action items.

Comment 20

[Nikolett]

Hello everyone!

Update

Action Item	Kind	Due Date - Status
Fix of certification profiles	Repair	2024-07-04 - Done
Revoke affected certificates	Repair	2024-07-05 - Done
Contact information update in CP/S	Repair	2024-07-05 - Done (Our updated CPS/S came into force on 2024.07.01.)

As per our previous comment: On July 8th, 2024, NETLOCK trained the relevant staff on the response time requirements for inquiries received at the specified email address.
To improve response times when processing Certificate Problem Reports we have done the following:

• We have created a dedicated email address where you can directly report any possible problems with the certificates.
• This new e-mai address has been added to the CP/S relevant contact chapter.
• For this mailing address, we have set up a dedicated monitoring channel, which notifies several relevant departments in parallel, in several different forums (e-mail and corporate information software), in order to ensure that the incoming alert is investigated immediately.
• Since the date mentioned above, our colleagues have been educated on the topic again - we are keeping this continuous, that is why we have included this topic in our annual education plan.

With regard to the above, we would like to ask for the ticket to be closed, as the measures described above will ensure that in the future reports will be dealt with in a timely manner.

Comment 21

[Ben Wilson]

If there are no other questions, I will ask Netlock to submit a Closure Summary, which includes a formal request for this bug to be closed.

Comment 22

[Wayne]

I do have a question: Why did it take Netlock nearly 6 months to respond to a question by Mozilla?

I'll be up front and say I presumed any discussion during that timeframe between Netlock and Mozilla was happening in private. This was due to how quiet the CA became for months across all of its incidents. Is this going to be the norm for CAs going forward?

Given this incident is for not responding to a CPR within 24h, and we have a CA who missed weekly updates for months, it seems rather relevant. Communication is key, but unfortunately lacking in Netlock's incidents to date.

Comment 23

[Ben Wilson]

I want to emphasize the importance of prompt and consistent communication by CA operators regarding incident reports. Mozilla expects CAs to engage actively in incident discussions and provide timely updates in Bugzilla. It is concerning that, over the past six months, Netlock has been incommunicative across multiple incidents.

To maintain trust in the Web PKI ecosystem, transparency and responsiveness are essential. Delays in communication hinder the resolution of compliance concerns and, if persistent, can lead to further scrutiny and potential distrust actions by root programs. Ensuring prompt and comprehensive responses will help mitigate these risks.

Please confirm that Netlock will improve its responsiveness to incident reports and outline any internal measures being implemented to prevent similar delays in the future.

Comment 24

[Nikolett]

Hello everyone, thank you for the comments, we will answer the following day, on 06.03.2025.

Comment 25

[Nikolett]

Hello Everyone!
As for comment 22:

Yes, NETLOCK understands that communication is key in solving future problems.
Because we had quite a few tickets open in the span of the last year, we wanted to improve our responses in a quality and transparent way and to set a good example for the community. Tickets have been closed since then, we are improving our processes and communicating continuously with our auditors. We acknowledge that our response time to the tickets was not adequate, therefore we created a summary manual to make it easier and quicker to respond altogether. It also highlights the usage of the whiteboard of this platform. If we are unable to give a detailed response we will use the whiteboard and indicate the nex update date and we will get back with the complete answer.
We know that time is the central issue: we created a prioritization matrix as well, stating and assigning responsibilities, with posibble labour sanctions for our employees.

Comment 26

[Zacharias]

(In reply to Nikolett from comment #25)

Hello Everyone!
As for comment 22:

Yes, NETLOCK understands that communication is key in solving future problems.
Because we had quite a few tickets open in the span of the last year, we wanted to improve our responses in a quality and transparent way and to set a good example for the community. Tickets have been closed since then, we are improving our processes and communicating continuously with our auditors. We acknowledge that our response time to the tickets was not adequate, therefore we created a summary manual to make it easier and quicker to respond altogether. It also highlights the usage of the whiteboard of this platform. If we are unable to give a detailed response we will use the whiteboard and indicate the nex update date and we will get back with the complete answer.

It’s great to hear that you are taking responsibility and are committed to improve.

We know that time is the central issue: we created a prioritization matrix as well, stating and assigning responsibilities, with posibble labour sanctions for our employees.

This however sounds terrible! Perhaps there is a misunderstanding in the communication, but I am unable to interpret “labour sanctions for our employees” in any manner that does not alarm me.

For an organization that is entrusted with so much responsibility as a CA I think it is critical to have a level of resilience and safety that a mistake of a single individual is not enough to cause an incident. The risk of punishment for mistakes also creates incentives for a cover up and that not only risks problems growing larger but it also prevents learning.

Responsibility should be shared for example, I found it very interesting to learn in a recent bug at another CA that they implemented a policy of using shared accounts (I think in this case it was CCADB) to ensure that any emails reach a group of people instead of any single individual. To me this was very interesting since from what I understand the prevailing consensus regarding accounts otherwise is that they should never be shared.

Working under the threat of “labour sanctions” must also create a terrible stress for your employees, how horrible it must be working under a sword of Damocles. Instead I believe you should be supporting your employees, making sure they have the resources they need (this might include more colleagues) while encouraging a culture of blamenessnes.

Comment 27

[Malcolm D]

There's an expectation of weekly status updates to all non-closed incidents.
There have not been recent updates to Bug #1904041, Bug# 1905509, Bug #1947691, Bug #1917046
In similar cases, the lack of updates is itself cause to file a separate incident report.

Comment 28

[Zacharias]

Ben,

As far as I can tell this is the last word we have from Netlock in over three weeks:

We know that time is the central issue: we created a prioritization matrix as well, stating and assigning responsibilities, with posibble labour sanctions for our employees.

As I initially responded to their comment, that sounds like a terrible practice. And I must say that I’m a bit worried about what could possibly be preventing Netlock from providing updates.

1947691 was created two months ago, no updates
1904041 promised an update on 2025-03-07
1917046 a non-conforming closure summary one month ago
1938167 set their own next update to 2025-03-15, but never did update.

Their webpage has a banner about a service disruption in 2025-03-19, you would expect something like that to be resolved or at least have an update after a week. Is there anyone awake over there?

Comment 29

[Ben Wilson]

Bug #1957474 has been opened.

Comment 30

[Dorottya]

Dear Zacharias,
we also believe that good communication is important for us and for our clients. Communicating with the community increases the perception that we are more transparent and compliant. NETLOCK operates in the European Economic Area. The activities of trust service providers are a highly regulated market, which requires regular independent audits and ongoing contact with the authorities. Compliance with the rules, not only the rules set by the community, but also the EU rules, puts a certain pressure on the service provider, which the employees of the service provider must be aware of. European and US labour law and labour law sanctions work differently and that is why the prospect of a labour law sanction may seem strange to you.
However, in our view, these sanctions should not be misinterpreted and portrayed as evil. TSPs are required to follow EU rules and at the same time the Community's rules on recurrent reporting. Which put the above mentioned pressure on TSPs and with it their employees. If the aim is to reduce the threat and pressure, it may also be necessary to relax administrative obligations, without compromising the stated objectives. It is also in the TSPs' vital interest to improve their service and to comply with the EU and national supervisory authorities.
The failure of a TSP to communicate does not mean that it is not trying to address the problem.
All TSPs are grateful for raising and reporting problems and it is beyond dispute that the obligations they have undertaken must be fulfilled, including communication obligations.
In Hungary, TSPs are obliged to report all errors and inform customers. Even those that may only affect the services customers receive. It should also be remembered that NETLOCK provides not only SSL certificates but also other signing certificates and time stamps. For this reason, any interpretation of events that NETLOCK's operation is inappropriate would lead both customers and market watchers to the wrong conclusion.

Thank you

Comment 31

[Zacharias]

(In reply to Dorottya from comment #30)

Dear Zacharias,
we also believe that good communication is important for us and for our clients. Communicating with the community increases the perception that we are more transparent and compliant. NETLOCK operates in the European Economic Area. The activities of trust service providers are a highly regulated market, which requires regular independent audits and ongoing contact with the authorities. Compliance with the rules, not only the rules set by the community, but also the EU rules, puts a certain pressure on the service provider, which the employees of the service provider must be aware of. European and US labour law and labour law sanctions work differently and that is why the prospect of a labour law sanction may seem strange to you.

I am also in the EU. What I believe is that the responsible thing to do is to structure CA operations in a way where the shortcomings of any individual is not enough to cause an incident. With redundancies and shared responsibility.

However, in our view, these sanctions should not be misinterpreted and portrayed as evil. TSPs are required to follow EU rules and at the same time the Community's rules on recurrent reporting. Which put the above mentioned pressure on TSPs and with it their employees. If the aim is to reduce the threat and pressure, it may also be necessary to relax administrative obligations, without compromising the stated objectives. It is also in the TSPs' vital interest to improve their service and to comply with the EU and national supervisory authorities.

I assume, that just like it is for me, English is not your native language. But the choice of this word: threat. That is what I object so strongly against. Being a TSP is about trust, it’s in the name. How trustworthy could you deem an organizations that resorts to ”threat and pressure” against their employees to be?

Is the administrative burden referenced above the requirements to participate in Bugzilla? To respond to comments within 7 days?

The failure of a TSP to communicate does not mean that it is not trying to address the problem.

But the failure of a TSP to communicate is in and of itself a problem. Does Netlock not agree?

All TSPs are grateful for raising and reporting problems and it is beyond dispute that the obligations they have undertaken must be fulfilled, including communication obligations.
In Hungary, TSPs are obliged to report all errors and inform customers. Even those that may only affect the services customers receive. It should also be remembered that NETLOCK provides not only SSL certificates but also other signing certificates and time stamps. For this reason, any interpretation of events that NETLOCK's operation is inappropriate would lead both customers and market watchers to the wrong conclusion.

Thank you

Netlock has not responded to comments or provided updates in several (all?) of their currently active incidents within the required timeframe. Thus it is beyond dispute that something is inappropriate with Netlock’s operations. For this to be an ”interpretation” it would be appropriate for Netlock to disregard the CCADB IRGs.

Distracted by the threat of labour sanctions I did not give this proper consideration:

we created a prioritization matrix as well, stating and assigning responsibilities

What did Netlock prioritize over its obligations to respond to comments and provide updates to incidents on Bugzilla?

Comment 32

[Nikolett]

Dear Zacharias,
NL always places the highest priority on correcting errors when they occur.
We do not use threats against our staff. We have made it known that we have assigned responsibilities and we have stressed this. We also drew attention to the labour consequences. Otherwise, the employer has a duty to provide information to employees.
NL acknowledges and has acknowledged its obligations, we have not denied our responsibility in any case. Community members were informed last week of the priority for improvements. We are trying to meet our obligations to provide updates.

Comment 33

[Nikolett]

Dear Community members, we are proceeding with our ticket update.
As for this ticket we would like to ask for its closure. The problem this ticket was raised for was that the CPR was not responded to in 24 hours.
This has been resolved and since then we have corrected the issue.
Please see Bug #1957474 for the for failure to comply with the weekly communication requirement.

Comment 34

[Ben Wilson]

Please review https://www.ccadb.org/cas/incident-report#how-are-reports-closed
Also, before filing an Incident Report Closure Summary, you must ensure that you have clearly identified and addressed all open questions.

Comment 35

[Nikolett]

Dear Community Members,
as indicated in the previous post, we will review the open issues and submit our closure summary.

Comment 36

[Nikolett]

Report Closure Summary

• Incident description:
  NETLOCK received an e-mailed on June 28th 2024 19:45 UTC in accordance to multiple issues (NETLOCK CP/S, certification problems). The e-mail was sent to [e-mail], it was delivered, but the e-mail was not responded to within 24 hours.
• Incident Root Cause(s):
  The root cause was a misalignment between the CPR contact information disclosed in the CCADB and that listed in section 1.5.2 of the CP/S. Additionally, there was an internal delay in distributing the CPR message to the appropriate personnel.
• Remediation description:
  The CA updated its CP/S to reflect the correct CPR contact address, implemented procedures to ensure faster triage of incoming reports. Furthermore, internal processes were revised to improve visibility and human resources have been made redundant
• Commitment summary:
  NETLOCK confirms that all action items described in the full incident report have been completed. The CPR contact is now accurately disclosed and monitored by multiple team members through internal escalation protocols.

Regarding Comment 26, we would like to clarify that no account sharing has taken place. Instead, to ensure that certificate problem reports reach the appropriate personnel regardless of the recipient, we have implemented internal information-sharing practices. Specifically, messages sent to individually assigned email addresses are internally redistributed among the compliance and operational teams. This ensures that all relevant team members are informed and can respond without delay, while maintaining individual account integrity.

In response to Comment 22, Regarding the question about the six-month response time, we would like to clarify that this period was necessary to properly prepare and implement internal training measures. Specifically, we developed comprehensive training materials, designed a structured training plan, and conducted the required training sessions for employees. During this time, staff were not only trained but also given the opportunity to practice and become familiar with the new procedures. This approach was essential to ensure effective understanding and long-term application of the compliance-related tasks across the organization.

All Action Items disclosed in this report have been completed as described, and we request that this incident now be closed.

Comment 37

[Nikolett]

Dear community members
Do you need any other data or information about the ticket?
Thank you

Comment 38

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, this bug will be closed on approximately 2025-05-08.