1904041 - NETLOCK: Intermediate CA Certificate not disclosed to CCADB

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Nikolett]

Incident Report

Summary of initial report

NETLOCK has failed to disclose a recently issued Subordinate CA Certificate (https://crt.sh/?id=13432786831) to the CCADB in accordance with the requirements of the Mozilla Root Store Policy - https://www.ccadb.org/policy#4-subordinate-ca-certificates:

"The CA operator with a certificate included in Mozilla’s root store MUST disclose such CA certificate in the CCADB within one week of certificate creation, and before any such CA is allowed to issue certificates"

The above-mentioned CA registration was started at the same time as this notification. The registration process will be completed within the next 5 working days.

We will upload the final report as soon as possible, max. in 2 weeks time.

Comment 1

[Rob Stradling]

The above-mentioned CA registration was started at the same time as this notification.

What is this supposed to mean? A CA either discloses a CA Certificate to CCADB, or they don't. There is no "started".

The registration process will be completed within the next 5 working days.

Non-compliance #1:
It has been more than 5 working days since you wrote that, but https://crt.sh/?id=13432786831 still has not been disclosed to CCADB.

Non-compliances #2, #3, and #4:
An additional three undisclosed Subordinate CA Certificates became known to crt.sh on the same day this bug was filed (see https://crt.sh/mozilla-disclosures#undisclosed). So there are now a total of FOUR CA certificates that Netlock has failed to disclose "in the CCADB within one week of certificate creation".

When will the necessary CCADB disclosures be completed?

Non-compliance #5:
It has been 10 days since your last comment, but https://www.ccadb.org/cas/incident-report#incident-reports requires that "Open incident reports should be updated...weekly, if a “Next update” date is not recorded".

Comment 2

[Ben Wilson]

Why haven't these intermediate CA certificates been uploaded to CCADB yet?

Comment 3

[Wayne]

It is also over 2 weeks after the preliminary incident was made...

Comment 4

[Rob Stradling]

The following 4 intermediate CA certificates have still not been disclosed to CCADB! Why?!?
https://crt.sh/?sha256=001279B949DD8670F1AE6DA407913B726F61AEF78A2FF19C2DA39522B31DC0C7
https://crt.sh/?sha256=F37E7AD92ECEA12F307501C126B3E2D6DE2C7417D3E1B72D26069C1370E78894
https://crt.sh/?sha256=D0EB908401F33242602634AFD51991536B3AD7AA901586FDEB4955FE51B0E419
https://crt.sh/?sha256=17771F6947FA3472786D3A44B5ADE2AACBA9ADA203BA31EBD4BD8CEBAFCE494A

Non-compliance #6:
Another intermediate CA certificate (https://crt.sh/?sha256=02B4AAF57AF0943FF1D7BAC4CE0FCC756E336F0F1C059AA8CAF84DFF9DD2E7A3) has been disclosed by NETLOCK to CCADB today. However, since this certificate has a notBefore date of May 13th 2024, we can conclude that NETLOCK failed to disclose it "in the CCADB within one week of certificate creation".

Comment 5

[Nikolett]

Hello Rob,

We have discloed the above mentioned 4 intermediate certificates to CCADB as of now.
We will open another ticket for non-compliance #6.

Comment 6

[Rob Stradling]

(In reply to Nikolett from comment #5)

Hello Rob,

We have discloed the above mentioned 4 intermediate certificates to CCADB as of now.

No, you haven't.

You've disclosed 4 different intermediate certificates, which were issued by your NETLOCK TLS ECC CA root CA that is present in CCADB but not (yet) trusted by any root programs. These 4 disclosed intermediate certificates share the same Subject DN and Public Key as the 4 intermediate certificates you still have not disclosed. The not-yet-disclosed intermediate certificates are still being reported by https://crt.sh/mozilla-disclosures#undisclosed.

Please try again.

We will open another ticket for non-compliance #6.

Please disregard my claim regarding "non-compliance #6". It's only today that I noticed that that certificate doesn't chain to a trusted root, which means that (AIUI) the requirement to disclose "in the CCADB within one week of certificate creation" does not apply in this case. Apologies for any confusion.

(In reply to Nikolett from comment #0)

We will upload the final report as soon as possible, max. in 2 weeks time.

You wrote that statement nearly 3 weeks ago. Where is the final report?

Comment 7

[Rob Stradling]

Another 7 days have passed.
NETLOCK has still not disclosed the 4 intermediate certs to the CCADB.
NETLOCK has still not posted the final incident report.
NETLOCK has still not answered questions that were asked over a week ago.
NETLOCK has still not been posting updates to this bug at least weekly.

What needs to happen for this incident to be taken seriously?

Comment 8

[Ryan Dickson]

Hi Nikolett,

Recent incident report handling by NetLock appears to be falling into a concerning pattern that conflicts with the factors described as significant by the Chrome Root Program Policy. We do not consider the handling of this report acceptable. Behavior observed in other open NetLock incident reports (e.g., 1905509) calls into question whether NetLock is actively observing and applying lessons learned from other reports disclosed to this forum.

Approximately 13-months ago, we (Chrome Root Program) - and other members of the community observed and communicated similar concerns (example) across multiple incident reports including:

• NetLock: Disclosed CRL is expired
• NetLock: Pre-certificates revoked with certificateHold reason
• NetLock: SSL certificates with OU field - revocation delay
• NetLock: SSL certificates with OU field

Beyond addressing the immediate concern described in this incident report, can you offer discussion on the specific set of time bound steps being taken by NetLock to better align incident reporting practices with community expectations (including those described on CCADB.org) - and why we’re seeing this regression in response to past concerns related to incident reporting?

Thanks,
Ryan

Comment 9

[Nikolett]

Dear Rob,

NETLOCK takes this incident seriously and we will do our utmost to ensure that any incidents that are detected are dealt with and resolved.

We wanted to provide complete information, so we first updated the CP and CPS and immediately after the above report, we started uploading the intermediate data to CCADB, which was done at 18:01 UTC on 09/07/2024.
The changed policies were published on our website at 17:21 UTC on 09/07/2024.

On 11/07/2024 14:01 UTC the following intermediate CAs were withdrawn:

SHA-256 EF7870B2D83AA14582714793B0D7D9EF1124FBDFCC88AE01C3779F9C31D6F532
https://crt.sh/?id=13673667537

SHA-256 80462B926AD10793A9036D78C054E7F31BD4E266020B33220ED21AD808DEEBD1
https://crt.sh/?id=13673667536

SHA-256 3DEC5FF836EB4B4A144F10CA02BEEDF065001F73F276D29B5C8535183FE43C78
https://crt.sh/?id=13673667535

SHA-256 02B4AAF57AF0943FF1D7BAC4CE0FCC756E336F0F1C059AA8CAF84DFF9DD2E7A3
https://crt.sh/?id=13671935339

Comment 10

[Nikolett]

Dear Ryan,

As we know, the Mozilla ROOT program was created to help Service Providers who provide the same services. The aim of the programme is to learn from our mistakes and from each other's. Previously recorded tickets have been issues that have been resolved, while we continue to improve our monitoring and certificate issuing systems to meet the expectations and to avoid further similar problems. The guidance of the community in solving these problems is very important to us and we thank them for their support in helping us to work with even better processes in the programme.

Comment 11

[Rob Stradling]

(In reply to Nikolett from comment #9)

we started uploading the intermediate data to CCADB, which was done at 18:01 UTC on 09/07/2024.

No, you did not, as I already pointed out in comment 6 and comment 7.

On 11/07/2024 14:01 UTC the following intermediate CAs were withdrawn:

Those 4 intermediate certificates are not relevant to the CCADB disclosure requirement, as I already pointed out in comment 6.

Comment 12

[Nikolett]

(In reply to Rob Stradling from comment #11)

(In reply to Nikolett from comment #9)

we started uploading the intermediate data to CCADB, which was done at 18:01 UTC on 09/07/2024.

No, you did not, as I already pointed out in comment 6 and comment 7.

Dear Rob,
In our CCADB the CAs are green, and the censys searh comes back as trusted, but in crt.sh we can see that it says it is not disclosed. We have updated the right CP/CPSs. Did we miss any steps we are not aware of?
https://search.censys.io/certificates/17771f6947fa3472786d3a44b5ade2aacba9ada203ba31ebd4bd8cebafce494a
https://search.censys.io/certificates/d0eb908401f33242602634afd51991536b3ad7aa901586fdeb4955fe51b0e419
https://search.censys.io/certificates/001279b949dd8670f1ae6da407913b726f61aef78a2ff19c2da39522b31dc0c7
https://crt.sh/?id=13432786831

Comment 13

[Wayne]

NETLOCK, what do you believe being a Trusted Intermediate means? How does this relate to CCADB?

Comment 14

[Ben Wilson]

Hi Nikolett,
All CA certificates (not just one certificate per CA) need to be uploaded into the CCADB. It looks like this still needs to be done for these CA certificates. This is a policy mentioned here: https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#532-publicly-disclosed-and-audited. Instructions can be found here: https://www.ccadb.org/cas/intermediates#adding-intermediate-certificate-data
Thanks,
Ben

Comment 15

[Nikolett]

Hello Ben,
Thank you for your helpful reply.

Comment 16

[Nikolett]

Incident Report

Summary

Bug 1904041 was opened on 21/06/2024 13:01 UTC for the following subject: NETLOCK: Intermediate CA certificate not published to CCADB

4 intermediary certificates not registered with CCADB within 7 days of their generation

Impact

this issue affected 4 intermediate CAs, which are cross-signed CAs

Timeline

All times are UTC.

Action timestamp	Action taken
2024-06-21 13:01 UTC	opening the incident ticket
2024-07-09 17:21 UTC	publication of CP/S on the netlock.hu website
2024-07-09 18:01 UTC	first registration of CCADB registration
2024-07-09 20:31 UTC	report to the supervisory authority
2024-07-10 12:22 UTC	Rob commented the incident
2024-07-17 13:59 UTC	undisclosed intermediate CAs checked
2024-07-18 06:30 UTC	all intermediate CAs marked as undisclosed were disclosed

Root Cause Analysis

We built our system step by step. The root was issued last year. New intermediate and cross-signed intermediate CAs were issued as well. After that, end user certificates were issued. We realized after the issuance that intermediate CAs were not disclosed within 7 days as required, and the regular CAs were revoked. We wanted to provide complete information on the intermediate CAs, which is why we issued end user certificates as well. This method allowed us to check the system. We checked all intermediate CAs marked as undisclosed and realized, that the CAs chain must be corrected on the disclosure form. Due to an administrative error the intermediate CAs were interchanged and referred to a wrong root and cert.sh showed these intermediate CAs undisclosed. After the correction all intermediate CAs showed as disclosed on cert.sh.

Lessons Learned

What went well

• We finished to build our system, which according to the tests, works well.

What didn't go well

• During the development of the new system, we encountered difficulties that needed to be addressed and resolved.

Where we got lucky

• We are fortunate because we have the opportunity to develop our systems and fix the issues that arise.

Action Items

Action Item	Kind	Due Date
The employees will be educated about the registration process of the CCADB within 3 days.	Prevent	2024-07-22

Details of affected certificates

Affected intermediate CAs related to trusted roots:

SHA256 17771F6947FA3472786D3A44B5ADE2AACBA9ADA203BA31EBD4BD8CEBAFCE494A
SHA 256 001279B949DD8670F1AE6DA407913B726F61AEF78A2FF19C2DA39522B31DC0C7
SHA 256 F37E7AD92ECEA12F307501C126B3E2D6DE2C7417D3E1B72D26069C1370E78894
SHA 256 D0EB908401F33242602634AFD51991536B3AD7AA901586FDEB4955FE51B0E419

Comment 17

[Chris Clements]

In comment #8, we asked that beyond addressing the immediate concern described in this incident report, you offer discussion on the specific set of time-bound steps being taken by NETLOCK to better align incident reporting practices with community expectations - and address why we’re seeing this regression in response to past concerns related to incident reporting.

Question #1: How will NETLOCK ensure timely updates and communication regarding incident reports in the future? What steps are being taken to improve transparency and responsiveness?

Question #2: Why are we seeing a regression in incident reporting when past improvement commitments were made?

Specific to your update in comment #16, this report should be updated further to better adhere to the criteria detailed at https://www.ccadb.org/cas/incident-report#incident-reports.

Question #3: Can you review the timeline to align to ensure alignment with the guidance on ccadb.org? Specifically,

“The timeline must include not just the actual discovery of the incident and subsequent events, but also relevant events occurring beforehand (e.g. something changed or was introduced).” and “In addition, it must indicate the following events:

• All policy, process, and software changes that contributed to the Root Cause [...]”

Question #4: Can you elaborate on the specific administrative error (or errors) that led to this incident? What processes failed or were not followed correctly, and why? Again, ccadb.org includes specifics for what needs to be included in the Root Cause Analysis. Assuming a more detailed Root Cause Analysis is performed, or additional conditions giving rise to the issue are disclosed, we suspect (1) additional Action Items are needed to address the root cause and prevent its recurrence and (2) more robust Lessons Learned can be conveyed for others to benefit from.

Question #5: Can you elaborate on “What didn’t go well”? What difficulties were encountered and resolved and how do they relate to this incident? In general, how can others in the community benefit from the Lessons Learned you documented?

Question #6: Have there been any changes to NETLOCK’s policies or procedures as a result of this incident? If so, what are they?

Comment 18

[Dorottya]

Hi Chris, thank you for the comments, we would like to inform you below:
Question #1: How will NETLOCK ensure timely updates and communication regarding incident reports in the future? What steps are being taken to improve transparency and responsiveness?

The TSP has revised its incident management process. All compliance staff have received training on incident management, including deadlines, content, and formal elements. As part of the training, staff reviewed previous incident tickets and analyzed the events with the help of their supervisor. We expect these measures to help us handle incidents more quickly and smoothly with the community’s support. All tickets have a decdicated ticket manager as well.

Question #2: Why are we seeing a regression in incident reporting when past improvement commitments were made?
Specific to your update in comment #16, this report should be updated further to better adhere to the criteria detailed at https://www.ccadb.org/cas/incident-report#incident-reports.

The TSP continues to make efforts to manage incidents properly from a technical standpoint and to administer and track the required incident tickets. We believe that our efforts outlined in the response to Question #1 will demonstrate our commitment to the community and provide assistance to its members by allowing them to learn from our mistakes and solutions. The implemented improvements ensure the excellent performance of the old system components, while the new developments contribute to the outstanding operation of the new system components.

Question #3: Can you review the timeline to align to ensure alignment with the guidance on ccadb.org? Specifically,
“The timeline must include not just the actual discovery of the incident and subsequent events, but also relevant events occurring beforehand (e.g. something changed or was introduced).” and “In addition, it must indicate the following events:
All policy, process, and software changes that contributed to the Root Cause [...]”

Yes, we will supplement the timeline.
All times are UTC.

Action timestamp Action taken
2024-06-21 13:01 UTC opening the incident ticket
2024-07-09 17:21 UTC publication of CP/S on the netlock.hu website
2024-07-09 18:01 UTC first registration of CCADB registration
2024-07-09 20:31 UTC report to the supervisory authority
2024-07-10 12:22 UTC Rob commented the incident
2024-07-17 13:59 UTC undisclosed intermediate CAs checked
2024-07-17 14:30 UTC IT operations involved in checking the datas for registrations, six-eyes principle review process started
2024-07-18 06:30 UTC all intermediate CAs marked as undisclosed were disclosed

Question #4: Can you elaborate on the specific administrative error (or errors) that led to this incident? What processes failed or were not followed correctly, and why? Again, ccadb.org includes specifics for what needs to be included in the Root Cause Analysis. Assuming a more detailed Root Cause Analysis is performed, or additional conditions giving rise to the issue are disclosed, we suspect (1) additional Action Items are needed to address the root cause and prevent its recurrence and (2) more robust Lessons Learned can be conveyed for others to benefit from.

We have prepared the necessary data for the intermediate CA disclosure. Among the intermediate CAs were cross-signed CAs, which data were confused by the registering staff and not correctly associated with the proper root CA. Unfortunately, the reviewing staff did not pay enough attention to the detailed verification, leading to the registration with mixed-up data.

Question #5: Can you elaborate on “What didn’t go well”? What difficulties were encountered and resolved and how do they relate to this incident? In general, how can others in the community benefit from the Lessons Learned you documented?

We emphasize the importance of accurately collecting data before registration, which has been done. However, under the "What didn’t go well?" section, we must mention that even the most meticulously compiled data will not yield the desired results if not registered according to the correct procedure, and if the verification does not uncover the error, it cannot be corrected during the process. We will also have the IT operations review the data to be uploaded to CCADB to ensure its accuracy and to prevent any issues when performing multiple steps in parallel.

Question #6: Have there been any changes to NETLOCK’s policies or procedures as a result of this incident? If so, what are they?

The previous in-house four-eyes principle-based review and pre-registration approval did not function correctly, necessitating process improvements. We have implemented a six-eyes principle for review and approval, where the third person must be a supervisor . The process must be documented electronically, allowing the data compilation, review, and approval steps to be traced back at any time. The relevant training has already been conducted.

Comment 19

[Wayne]

For the timeline please note when CCADB requirements were first put in place requiring the disclosure of newly generated intermediaries.

It is distressing that with the new six-eyes approach the timeline starts in late-June. These intermediaries were generated in May, and certificates were generated off of them well before a CP/S was published. There is the 30-day issue, but let us ignore it for now.

I hope NETLOCK appreciates the danger in how these intermediaries were generated and mis-handled. We are months into them being generated and still lack a full story of what happened and what went wrong. I am still awaiting an answer to Comment 13 as it is fundamental to NETLOCK's credibility. Please appreciate the implications of your original response, and why the answer to my question is so important.

There are other issues stemming from this incident already on bugzilla (#1905509, #1906115, #1907568). There was also prior discussion on more incidents being required to be raised, but nothing has occurred.

Would a meta incident be easier for everyone at this stage? I will note that NETLOCK have previously asked for guidance on handling these incidents. To be clear, I am a Relying Party and this is not a decision I should be making.

Comment 20

[Dorottya]

Dear All,
We greatly appreciate the helpful notifications and questions sent to us by community members. We have strived to address the issues and incidents that have arisen. Based on the feedback from the community, we have decided to review and renew our processes. We will further train our staff to fully meet the expectations of the community. Consequently, both we and the community members will be able to learn from the errors that occur, the immediate actions taken to resolve them, and our future action plans. Within this framework, we have defined our primary tasks in the following six points:
Adherence to Regulations and Standards:
TSP strictly follows industry standards and guidelines, such as the CA/Browser Forum principles and guidelines. These outline how certificates should be issued, renewed, and revoked. We will place significant emphasis on extraordinary and periodic training for our staff.
Strengthening Security Measures:
Maintaining robust security measures, including physical and digital security protocols, is of paramount importance to TSP. This includes access control, keeping encryption methods up-to-date, and regularly testing and fixing vulnerabilities. We will improve communication between IT Operations, compliance, and other departments. We have established communication channels within the company to enable quick and targeted communication. Additionally, we will hold weekly meetings involving supervisors from the relevant areas. Maintaining Transparency:
TSP is transparent in its activities, including decisions related to certificate issuance and revocation. Publishing certificates in a public registry and communicating with the community helps maintain trust. Incident Management and Rapid Response:
Quick and effective handling of security incidents, along with the proper revocation procedures, is essential. We will emphasize this commitment within the company to ensure all staff understand its importance. We will ensure that if a security issue arises, TSP takes immediate steps to resolve it and informs the affected parties. We aim to eliminate past administrative and human errors through periodic and extraordinary training and have introduced a six-eye principle in our review system, involving a third verifier who is an IT Operations supervisor.
Employing and Training Expert Staff:
Hiring well-trained and experienced professionals ensures that TSP has the necessary expertise to adhere to best practices and tackle new challenges. To ensure this, we have implemented periodic training sessions. In case of changes in regulations or lessons learned from an incident, we plan to hold workshop-style training sessions, involving external experts if necessary. Regular Audits and Compliance Checks:
Webroot auditing, conducted by independent auditor, helps identify potential problems and deficiencies before they cause serious issues. This also allows us to reinforce compliance with the aforementioned points. These measures collectively help maintain credibility and reliability, which is crucial for TSP to remain listed and retain user trust. They ensure that with renewed processes, a new and better operation can emerge in the near future. We trust that with improved communication, the community will be able to follow our efforts and that we will remain valuable members of the community.

Comment 21

[amir]

I hate that I have to ask this, but was comment 20 AI generated?

Anyway, that comment also doesn't really make sense in the context of this bug?

Comment 22

[Jan Schermer]

My thoughts exactly. But it might just be corpospeak. In any case, it has been clearly inspired by the huge success a certain CA achieved here with this type of communication recently.

Comment 23

[Ryan Dickson]

Hi Dorottya,

Comment 18 fails to directly address many of the elements requested in Comment 17.

The response to Question 1 lacks specificity and does not offer compelling evidence to suggest the changes described will meaningfully result in improved incident reporting procedures. It’s unclear what Netlock has changed related to its incident management process, or why previous processes and internal resources (e.g., training) were ineffective at preventing this incident.

As a general note, various incident reports in Bugzilla over the years have made it clear that training alone is insufficient as a response to compliance failures (e.g., 1, 2, 3, and 4). If an incident occurs due to human error, simply retraining the staff does not guarantee that the underlying process that allowed the error to happen is corrected (i.e., training addresses a symptom, not the root cause).

The response to Question 2 does not directly answer why we are observing a regression in incident reporting given Netlock’s past commitments. The response is only forward-looking. Demonstrating a clear understanding of why the processes and procedures in place at the time of this incident failed will support a more compelling narrative on whether the actions described in this report will sufficiently address the issue’s root cause(s).

The response to Question 3 remains incomplete. See Comment 19.

The response to Question 4, like the response to Question 2, does not provide the requested information. The response suggests human error was at play, and that the new certificates were incorrectly associated with the wrong roots in the CCADB.

The response to Question 5 includes “and if the verification does not uncover the error.” What specifically has changed within Netlock such that this failure does not repeat itself in the future? Can you describe the steps involved in the verification process?

The response to Question 6 suggests more participants involved in Netlock’s processes are intended to prevent future recurrence of the mistakes described in this incident report, however justification is not offered related to why an additional participant will meaningfully change the outcome observed here. Why, for example, are three participants more likely to detect and prevent missteps when compared to two?

Additionally, it appears as if many of Netlock’s open incident reports have gone stale, including this one. For example, requests or questions in Comment 19 and Comment 21 do not appear to have been responded to.

From CCADB.org:

“Once the report is posted, CA Owners should respond promptly to questions that are asked, and in no circumstances should a question linger without a response for more than one week, even if the response is only to acknowledge the question and provide a later date when an answer will be delivered.”

and

“Open incident reports should be updated:

• on or before the “Next update” date in the “Whiteboard” field of the bug;
• weekly, if a “Next update” date is not recorded; or
• when Action Items are changed, completed, or delayed.”

Thanks,
Ryan

Comment 24

[amir]

It has now been over 7 days for Ryan’s questions.

Netlock, can we assume that you no longer want to be a publicly trusted CA, as you’ve now ignored this bug for over 20 days?

Comment 25

[Dorottya]

Dear Amir, thank you very much for your comment. ( comment24).
Of course, NETLOCK would like to continue to be a publicy trusted CA. It is important for us to be compliant and to be part of the webroot. The system configuration changes and the changes in our processes commented and made in the previous period in the bug reports have been audited by an independent auditor in the past period. Therefore we can also have an independent audit to support compliance in addition to the selfsign audit. We have been working on this in the recent period. We would like a confirmation that the question indicated in comment 21 is real and needs to be answered. As a Trust Service Provider, we do not use AI solutions, but if you support them, we are happy to explore the possibilities. We will provide the information requested in Comment 23 shortly, and will also upload the independent auditor's report to the CCADB platform.

Comment 26

[Dorottya]

Dear Ryan, Comment 23:
The response to Question 1: According to Comment 16 in Question 6, the process is no longer monitored by four eyes, but by six with the involvement of our IT Department. The necessary trainings for this were done in several steps between 2024. May-July.

We expect improvements in the elimination of incidents with the improvement of our system configurations and the development of human resources.
We do think It is important to use the right linters and to regularly monitor the release processes. These checks will be carried out more frequently with the help and supervision of expert colleagues.

I agree with your point of view, but it is also necessary to educate colleagues. There were communication hole in the incident management process. We had to change the appropriate communication chain. Incidents are now processed by a rapid response group, which consists of managers and specialists, so that we can do the right thing quickly and reduce the response time. Prevention is really important, which is why we have set up several monitors to monitor the standards and our own systems as well.

Comment 27

[Ben Wilson]

Key Next Steps for Netlock:

1. Submit an updated incident report with a full timeline, updated root cause analysis, corrective actions, and supporting evidence (e.g. process improvements)
2. Answer community member questions about why the original four-eyes review process failed and what specific changes in the six-eyes review process will prevent recurrence
3. Ensure that all open Netlock incidents are properly updated and made as complete as possible

Comment 28

[Ben Wilson]

Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1905509#c23

Comment 29

[Nikolett]

Hello everyone. We will submit the updated incident report on 03.07.2025.
We will answer all remaining questions. Thank you.

Comment 30

[Malcolm D]

There's an expectation of weekly status updates to all non-closed incidents.
There have not been recent updates to Bug #1904041, Bug# 1905509, Bug #1947691, Bug #1917046
In similar cases, the lack of updates is itself cause to file a separate incident report.

Comment 31

[Ben Wilson]

Bug #1957474 has been opened.

Comment 32

[Nikolett]

Hi Ben,
we commented on the ticket you marked Bug #1957474. We will comment on the remaining questions soon.

Comment 33

[Nikolett]

Dear Community,
We are currently preparing a new and complete incident report for the end of next Friday, taking into account the new incident reporting guidelines, and will update the ticket next Wednesday.
The next update deadline is indicated on the whiteboard.

Comment 34

[Ben Wilson]

Hi Nikolett,
Accepted practice is to request a "Next Update" from the root programs, and then we will set it for you.
Thanks,
Ben

Comment 35

[Nikolett]

Hi Ben,

Thank you for the clarification, this is how we will act in the future. We will update the incident report tomorrow.

Comment 36

[Wayne]

When will the updated incident report be delivered?

Comment 37

[Nikolett]

Dear Community members,

We will update the report until this friday (05-09-25).

Comment 38

[Nikolett]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000039

• Incident description: This bug ticket was opened on 21/06/2024 13:01 UTC for the following subject: NETLOCK: Intermediate CA certificate not published to CCADB. Four intermediary certificates were registered in CCADB within 7 days of their generation

• Timeline summary:

  • Non-compliance start date: 2024-06-03 14:14 UTC
  • Non-compliance identified date: 2024-06-18 20:56 UTC
  • Non-compliance end date: 2024-07-18 06-30 UTC

• Relevant policies:
  Mozilla Root Store Policy v3.0, Section 5.3.2
  CCADB Policy v1.3.1, Section 5.1.2.2

• Source of incident disclosure:
  Third Party Reported, CA opened the ticket

Impact

TSP generated this 4 certificate but failed to disclose it into the CCADB within the required 7 days.
https://crt.sh/?sha256=001279B949DD8670F1AE6DA407913B726F61AEF78A2FF19C2DA39522B31DC0C7
https://crt.sh/?sha256=F37E7AD92ECEA12F307501C126B3E2D6DE2C7417D3E1B72D26069C1370E78894
https://crt.sh/?sha256=D0EB908401F33242602634AFD51991536B3AD7AA901586FDEB4955FE51B0E419
https://crt.sh/?sha256=17771F6947FA3472786D3A44B5ADE2AACBA9ADA203BA31EBD4BD8CEBAFCE494A

We have disclosed 4 intermediate certificates that share the same Subject DN and Public Key as the 4 intermediate certificates we failed to disclose. This happend due to employee oversight at the time.

• Total number of certificates: 4

• Total number of "remaining valid" certificates: 4

• Affected certificate types: TLS EV, TLS OV, TLS DV, TLS Quaified EV

• Incident heuristic: -

• Was issuance stopped in response to this incident, and why or why not?: No the issuance was not stopped. No user certificate has been issued other than the test certificates required for registration.

• Analysis: -

• Additional considerations: -

Timeline

Timestamp (UTC)	Action
2024-05-27 14:14	TSP began to generate the four affected Intermediate CA certificates.
2024-06-03 14:14	TSP missed the deadline to disclose the Certificates into CCADB
2024-06-18 20:56	Third party informed NETLOCK about the possible non-compliance
2024-06-19 18:26	NETLOCK confirmed the failure and responded to the report.
2024-06-21 13:01	Bugzilla incident ticked opened with a preliminary report by TSP
2024-06-24 14:00	Training for the six-eye verification process
2024-07-09 17:21	Publication of CP/S on the netlock.hu website because of the qualified intermediate CA
2024-07-09 18:01	Registration of the wrong - not needed - certificates to the CCADB under a root not yet included
2024-07-09 20:31	Report sent to the supervisory authority about the registration
2024-07-17 13:59	Undisclosed intermediate CAs checked on crt.sh - recognition of the misunderstanding of the same Subject DN and Public Key fields
2024-07-17 14:30	Another, six-eyes review process training
2024-07-18 06:30	All intermediates corrected and properly disclosed in CCADB
2024-07-19 14:42	Full Incident report filed for the bug ticket in accordance with the Incident Reporting Guidelines in force at the time
2024-07-22 13:00	Employee education about the registration process of the CCADB
2024-07-22 13:00	Employee education about the registration process of the CCADB

Related Incidents: N/A

Root Cause Analysis

We have built our new system step by step. Our new root was issued in 2023. New intermediate and cross-signed intermediate CAs were issued in may of 2024. After that, end user certificates were issued for the required testing.
We realized after the issuance that intermediate CAs were not disclosed within 7 days as required. We inspected on crt.sh all the intermediate CAs marked as undisclosed and realized, that the CAs chain and root must be corrected in the CCADB.
Due to an administrative error the intermediate CAs were reversed on paper and chained to a not yet included root, but the cross signed intermediate CAs were yet undisclosed. WE updated the CCADB, and after the correction all intermediate CAs showed up as disclosed on cert.sh as well.

Contributing Factor #: title

• Description:
  We have prepared the necessary data for the intermediate CA disclosure. Among the intermediate CAs were cross-signed CAs, which data were confused by the registering staff and not correctly associated with the proper root CA. Unfortunately, the reviewing staff did not pay enough attention to the detail verification, leading to the registration with mixed-up data.

• Timeline:

• Detection:

• Interaction with other factors:

• Root Cause Analysis methodology used:
  5 why

1. Why did our intermediate CAs fail to appear as disclosed within seven days?
   Because the entries in CCADB for those intermediate CAs were not created in time, within 7 days after issuance.
2. Why were the CCADB entries not created?
   We created the intermediate CAs under NETLOCK TLS EV ECC CA and the cross-signed intermediaries at the same time, and wanted to disclose them both, together with the root inclusion request to bugzilla. The former did not happen.
3. Why was the Intermediate CA information reversed and linked to the unregistered root?
   Because of an administrative mix‑up in the documentation and submission process: the new intermediate and cross‑signed intermediate certificates issued in May 2024 were mistakenly listed in reverse. We disclosed the CAs that were not needed insted of the cross-signed ones.
4. Why did this administrative mix‑up occur?
   Because the team lacked a clear checklist to confirm before submission, that which was the correct root CA present in CCADB. Even if the Subject DN and Public Key is the same - checking the Issuer of the cetitifcates would have prevented the disclousure of the not needed CAs insted of the cross-signed ones.
5. Why did we lack that verification step?
   Because our rollout process for the “new system” was built step-by-step without a formalized compliance‑check gate for CCADB submissions. In focusing on gradual issuance (root in 2023, intermediates in May 2024, then test end‑user certs), we didn’t embed a compliance review step that includes verifying the root’s presence in CCADB, confirming the Issuer of the CAs, and ensuring that the cross‑signed intermediates are also disclosed

Lessons Learned

• What went well:
  There are no customers impaced, the problem has not affected our service.
• What didn’t go well:
  We lacked a formal compliance‑check in our issuance process to verify CCADB registration and public disclosure.
  Administrative documentation mixed up the order of intermediate vs. cross‑signed intermediate and associated them with an unregistered root, leading to missed disclosure SLAs.
  There was no automated alert, to catch “undisclosed” intermediates, so the oversight wasn’t detected.
• Where we got lucky:
  There was no costumer involvment.
• Additional:

Action Items

Action Item	Kind	Corresponding Root Cause	Evaluation Criteria	Due Date	Status
Employees will be educated about the registration process of the CCADB	Prevent	Root Cause # 1	Training report	2024-07-22	Done
Published a step‑by‑step CCADB checklist for all future CA issuances	Prevent	Root Cause # 1	Stored checklist as evidence	2024-08-15	Done
Developing and testing a checker, what shows us upon generating if a new intermediate or cross‑signed intermediate cert is marked “disclosed, and indicates when it is not.	Prevent	Root Cause # 1	Development	2024-08-01-	Done
Training for the process in whole including how the checker is used	Prevent	Root Cause # 1	Training report	2025-08-15	Done

Appendix

Affected Certificates
https://crt.sh/?sha256=001279B949DD8670F1AE6DA407913B726F61AEF78A2FF19C2DA39522B31DC0C7
https://crt.sh/?sha256=F37E7AD92ECEA12F307501C126B3E2D6DE2C7417D3E1B72D26069C1370E78894
https://crt.sh/?sha256=D0EB908401F33242602634AFD51991536B3AD7AA901586FDEB4955FE51B0E419
https://crt.sh/?sha256=17771F6947FA3472786D3A44B5ADE2AACBA9ADA203BA31EBD4BD8CEBAFCE494A
crt.sh | 001279b949dd8670f1ae6da407913b726f61aef78a2ff19c2da39522b31dc0c7
Free CT Log Certificate Search Tool from Sectigo (formerly Comodo CA)

Comment 39

[Wayne]

(In reply to Nikolett from comment #38)

Timeline

Timestamp (UTC)	Action
2024-05-27 14:14	TSP began to generate the four affected Intermediate CA certificates.
2024-06-03 14:14	TSP missed the deadline to disclose the Certificates into CCADB
2024-06-18 20:56	Third party informed NETLOCK about the possible non-compliance
2024-06-19 18:26	NETLOCK confirmed the failure and responded to the report.
2024-06-21 13:01	Bugzilla incident ticked opened with a preliminary report by TSP
2024-06-24 14:00	Training for the six-eye verification process
2024-07-09 17:21	Publication of CP/S on the netlock.hu website because of the qualified intermediate CA
2024-07-09 18:01	Registration of the wrong - not needed - certificates to the CCADB under a root not yet included
2024-07-09 20:31	Report sent to the supervisory authority about the registration
2024-07-17 13:59	Undisclosed intermediate CAs checked on crt.sh - recognition of the misunderstanding of the same Subject DN and Public Key fields
2024-07-17 14:30	Another, six-eyes review process training
2024-07-18 06:30	All intermediates corrected and properly disclosed in CCADB

What caused the large gap in time between NETLOCK confirming the failure on 2024-06-19, and subsequently addressing it on 2024-07-18? I don't see this addressed in the incident report.

The report seems focused on the problems that happened at certificate generation, and why that did not lead to the certificates being disclosed on CCADB. What is missing is the lack of timeliness to correct the issue after it was specifically highlighted to NETLOCK. 2024-07-09 has the wrong certificates added to CCADB, and a further 9 days until any correction occurs despite a rather immediate correction in Comment 6 on 2024-07-10.

Can NETLOCK please explain their new timeline for generating a new intermediary, creating/updating a CP/S, disclosing it to CCADB, and generating subsequent certificates (Action Item 2)? If substantial changes have occurred to show that NETLOCK will not have a similar issue again then we need to understand your process.

Comment 40

[Nikolett]

Dear Community Members,

unfortunately, there was indeed a significant delay between the confirmation of the issue and its resolution. As indicated in the comments of the current ticket, on July 10, 2024, we believed that the intermediate certificates had been published.
This misunderstanding arose because the intermediate certificates were issued from a cross-signed root whose name was very similar to that of another cross-signed root. As a result, our colleague disclosed the certificates incorrectly, and we mistakenly thought the issue had already been resolved and communicated accordingly.

It is true that NETLOCK focused on the issue that occurred during the creation of the certificates, as this was the fundamental cause of the failure to publish. The publication could not proceed because the intermediate certificates were not issued from the root that NETLOCK had intended to use. Only after resolving this issue were we able to proceed with proper disclosion.

Due to this error, we had to initiate a comprehensive review of our processes. We were required to implement process control points that will ensure that such an issue cannot occur again in the future. However, to achieve this, we needed to broaden our focus beyond publication alone, and improve our practices around the creation of intermediate certificates as well. Based on the course of events, it became clear that we lacked sufficient experience in this area.

Comment 41

[Nikolett]

Dear Community Members,
do you have any further questions about the ticket?

Comment 42

[Wayne]

I did have some questions in Comment 39 that have not been addressed.

Comment 43

[Nikolett]

Between June 19 and July 10, we were testing our internal process adjustments to ensure that the issue would be handled effectively. During this testing phase, we did not detect any errors. Based on the results of these tests, we believed the problem had been successfully resolved.
However, following the feedback received on July 10, it became clear to us that although our internal validation did not reveal any remaining issues, the problem persisted. As we previously indicated in the comments on the current ticket:

“As indicated in the comments of the current ticket, on July 10, 2024, we believed that the intermediate certificates had been published.
This misunderstanding arose because the intermediate certificates were issued from a cross-signed root whose name was very similar to that of another cross-signed root. As a result, our colleague disclosed the certificates incorrectly, and we mistakenly thought the issue had already been resolved and communicated accordingly.”

This misunderstanding unfortunately delayed the actual resolution. In hindsight, it became evident that our internal processes were not sufficiently robust to detect and prevent this type of human error. As a result, we have initiated a comprehensive review and enhancement of these processes to ensure such issues cannot recur in the future.

Comment 44

[Wayne]

For clarity this is still outstanding from Comment 39:

Can NETLOCK please explain their new timeline for generating a new intermediary, creating/updating a CP/S, disclosing it to CCADB, and generating subsequent certificates (Action Item 2)? If substantial changes have occurred to show that NETLOCK will not have a similar issue again then we need to understand your process.

Comment 45

[Nikolett]

For comment 39:

We believed that the intermediate certificates had already been published. This misunderstanding arose because the intermediate certificates were issued from a cross-signed root whose name was very similar to that of another cross-signed root. As a result, our colleague disclosed the certificates under the wrong root, and we mistakenly assumed the disclosure had already been completed and communicated accordingly.
Recognizing that our previous process allowed for such a human error to occur, we have taken corrective actions to ensure this does not happen again. Specifically, we have implemented a step-by-step CCADB checklist for all future CA issuances. This checklist has been published and is now used as a mandatory part of the process. It guides the responsible team member through each stage of disclosure and verification, ensuring that no critical validation step is missed. This procedural safeguard addresses the root cause of the issue and mitigates the risk of recurrence.
As outlined in our corrective actions:

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Published a step-by-step CCADB checklist for all future CA issuances	Prevent	Root Cause #1	Stored checklist as evidence	2024-08-15	Done

A new intermediary will be generated only once the CP/CPS has been created or updated and disclosed to the CCADB in alignment with the updated checklist.
This includes internal approval of the CP/CPS, posting to the relevant repositories, and verification of publication through the CCADB interface before any certificates are issued.
Only once these steps have been validated internally and externally will we proceed with issuing certificates from the new intermediary.
This is the updated timeline:

Timeline

Date and Time (UTC)	Action
2024-05-27 14:14	TSP began to generate the four affected Intermediate CA certificates.
2024-06-03 14:14	TSP missed the deadline to disclose the Certificates into CCADB.
2024-06-18 20:56	Third party informed NETLOCK about the possible non-compliance
2024-06-19 18:26	NETLOCK confirmed the failure and responded to the report.
2024-06-21 13:01	Bugzilla incident ticket opened with a preliminary report by TSP.
2024-06-24 14:00	Training for the six-eye verification process.
2024-07-09 17:21	Publication of CP/S on the netlock.hu website for the qualified intermediate CA.
2024-07-09 18:01	Registration of the incorrect (unnecessary) certificates to the CCADB under a root not yet included.
2024-07-09 20:31	Report sent to the supervisory authority regarding the incorrect registration.
2024-07-17 13:59	Undisclosed intermediate CAs checked on crt.sh – misunderstanding identified due to same Subject DN and Public Key fields.
2024-07-17 14:30	Additional training on the six-eye review process.
2024-07-18 06:30	All intermediate certificates corrected and properly disclosed in CCADB.
2024-07-19 14:42	Full incident report filed for the bug ticket in accordance with the Incident Reporting Guidelines.
2024-07-22 13:00	Employee education on the CCADB registration process.
2024-08-01 12:00	Internal checker tool developed and tested – alerts on whether new intermediate or cross-signed certs are properly marked as “disclosed.”
2024-08-15 10:00	Step-by-step CCADB checklist published and implemented for all future CA issuances.
2025-08-15 14:00	Training completed for the entire issuance process, including use of the checker tool.

If you feel that our response to Comment 39 did not fully address your concerns, we kindly ask that you elaborate on the specific points that require further clarification. We are committed to transparency and are more than willing to provide additional detail as needed.

Comment 46

[Nikolett]

Dear community members, if you have no further questions, we will prepare the closure summary.

Comment 47

[Roland]

Attachment: ## Incident Closure

Comment 48

[Roland]

Incident Closure Summary

Bugzilla ID: 1904041 – NETLOCK: Intermediate CA Certificate not disclosed to CCADB

---

Summary

Summary Netlock failed to timely disclose some newly created intermediate CA certificates to the CCADB as required by section 4 of the Mozilla Root Store Policy and the CCADB Disclosure Policy.
The root cause analysis was performed, and remediation actions were completed to prevent recurrence.

---

Impact

• Undisclosed intermediates were technically capable of issuing end-entity certificates, and as such, should have been disclosed in the CCADB within one week of creation and prior to any use.
• No mis-issuance or certificate misuse was found; however, the late disclosure itself constitutes a material compliance failure.

---

Timeline

Date & Time (UTC)	Event Description
2024-05-27 14:14	Generation of the four affected Intermediate CA certificates was initiated by TSP.
2024-06-03 14:14	The deadline to disclose the certificates in CCADB was missed.
2024-06-18 20:56	Third party informed Netlock of possible non-compliance.
2024-06-19 18:26	Netlock confirmed the failure and responded to the notification.
2024-06-21 13:01	Preliminary incident report filed and Bugzilla ticket opened.
2024-06-24 14:00	Training conducted for the six-eye verification process.
2024-07-09 17:21	CP/S published on netlock.hu website for the qualified intermediate CA.
2024-07-09 18:01	First attempt at CCADB registration, including incorrect/unnecessary certificates under a root not yet included.
2024-07-09 20:31	Report submitted to the supervisory authority regarding registration activities.
2024-07-10 12:22	Community/Bugzilla (Rob) commented on the incident.
2024-07-17 13:59	Review of undisclosed intermediate CAs on crt.sh; misunderstanding identified concerning same Subject DN and Public Key.
2024-07-17 14:30	IT operations involved and additional six-eyes review process training conducted.
2024-07-18 06:30	All intermediate certificates previously marked as undisclosed were corrected and properly disclosed in CCADB.
2024-07-19 14:42	Full incident report filed for the Bugzilla ticket in accordance with Incident Reporting Guidelines.
2024-07-22 13:00	Employees educated on the CCADB registration process.
2024-08-01 12:00	Internal checker tool developed and tested to alert on disclosure status of new intermediates and cross-signed certificates.
2024-08-15 10:00	Step-by-step CCADB checklist published and implemented for all future CA issuances.
2024-08-15 14:00	Training completed for the entire certificate issuance process, including the use of the checker tool.

---

Root Cause Analysis

• Netlock’s internal CA generating process did not ensure that all newly created, technically capable intermediates were promptly disclosed in the CCADB prior to any certificate issuance.
• An internal misunderstanding existed regarding the timing and completeness of the registration process.
• Insufficient automation and lack of a systematic, detailed checklist contributed to human error and procedural gaps, resulting in late and incomplete disclosures.

---

Lessons Learned

• Actual, successful disclosure in the CCADB must be verified and completed within one week of creation and before any certificate issuance; merely initiating registration is insufficient.
• All technically capable intermediates must be tracked through an improved compliance workflow to prevent any delay or omission.
• Weekly updates and transparent, prompt reporting in case of future incidents are mandatory and critical for compliance.
• Training and regular staff education are essential to maintaining ongoing compliance with CCADB/Mozilla requirements.

---

Action Items

Action Item	Kind	Corresponding Root Cause	Evaluation Criteria	Due Date	Status
Employees educated about the registration process of the CCADB	Prevent	Root Cause #1	Training report	2024-07-22	Done
Step-by-step CCADB checklist published and implemented for all future CA issuances	Prevent	Root Cause #1	Stored checklist as evidence	2024-08-15	Done
Internal checker tool developed and tested to alert if new intermediate or cross-signed certs are not disclosed	Prevent	Root Cause #1	Development documentation	2024-08-01	Done
Training completed for the entire issuance process, including use of the checker tool	Prevent	Root Cause #1	Training report	2024-08-15	Done

---

Appendix

Affected intermediates (SHA-256 fingerprints):

• 001279B949DD8670F1AE6DA407913B726F61AEF78A2FF19C2DA39522B31DC0C7
• F37E7AD92ECEA12F307501C126B3E2D6DE2C7417D3E1B72D26069C1370E78894
• D0EB908401F33242602634AFD51991536B3AD7AA901586FDEB4955FE51B0E419
• 17771F6947FA3472786D3A44B5ADE2AACBA9ADA203BA31EBD4BD8CEBAFCE494A
• 02B4AAF57AF0943FF1D7BAC4CE0FCC756E336F0F1C059AA8CAF84DFF9DD2E7A3 (disclosed late)

Key policies referenced:

• Mozilla Root Store Policy, section 4
• CCADB Disclosure Policy
• CCADB Incident Reporting Guidelines

---

Incident Closure Summary

All missing intermediate CA certificates identified in this incident have now been properly disclosed in the CCADB. Netlock has updated its internal processes, published a step-by-step CCADB checklist, implemented automated internal controls, and completed additional training for all relevant staff. These changes are designed to prevent recurrence and ensure future compliance with all Mozilla and CCADB requirements. Netlock apologizes for the non-compliance and any resulting risk, and is committed to maintaining transparency, timely reporting, and full adherence to all applicable policies.

Comment 49

[incident-reporting]

Netlock should ensure that its CCADB checklist procedures include updating intermediate CA revocation information in the CCADB equally as they do for intermediate CA issuance.

With that final comment, this is the final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-01.

Comment 50

[Roland]

Dear Community Members,
thank you for the comment, we will manage our processes accordingly.