Open Bug 1680378 Opened 1 year ago Updated 27 days ago

Netlock: Replacement of enduser certificates after the EVGL 1.7.4 self-audit

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: varga.viktor, Assigned: banyai.anna)

Details

(Whiteboard: [ca-compliance])

Attachments

(2 files)

9.80 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
30.81 KB, application/x-zip-compressed
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36

Steps to reproduce:

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

During the self-audit on 20th November 2020 we have identified that we have to consider the EVGL 1.7.4 cleanups changes as significant changes which means that we should have put CAB Forum Organization ID in the certificates.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  • 2020-11-20 The problem was identified, the issuance was stopped.
  • 2020-11-23 The extension is set to mandatory for new certificates
  • 2020-11-30--12-01 Replacement certificates were issed.
  • 2020-12-07 Revocation of near al fo certificates
  • 2020-12-31 Revocation of the last certificate

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

On 2020-11-20 the problem was identified and the issuance was stopped until its review.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

19 valid certificates
(and also the corresponding CT certificates too)
List attached.

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

Attached to the ticket as zip.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

There were/are contradictions in the Section 9.8, which led to different understanding.

During the self-audit we have identified that the EVGL 1.7.4 cleanups changes are significant changes in meanings.
In the Relevant dates section the following line changed:
"If subject:organizationIdentifier is present, the Subject Organization
Identifier Extension MUST be present."
The new line:
If subject:organizationIdentifier is present, the CA/Browser Forum
Organization Identifier Extension MUST be present. "

This means that after this date it was mandatory to add this extension to EV certificates.
This extension was not set in our issued EV certificates, because the EVGL Section 9.8 mandates only the extension, which were indicated as "Required."

After we identified this change, we issued certificates (24) to our customers, and ask them to change the certificates on their servers.

The deadline of these changes for most certificates is December 7, 2020, but one customer (PSD2 aggregator) has asked longer time to change it in multiple banking systems.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Steps:

  • 2020-11-20 The problem was identified, the issuance was stopped.
  • 2020-11-23 The extension is set to mandatory for new certificates
  • 2020-12-01 Replacement certificates were issued.
  • 2020-12-07 Revocation of almost all certificates.
  • 2020-12-31 Revocation of the last certificate.
Assignee: bwilson → varga.viktor
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance]

I just wanna confirm, that the revocations were done in time as it was planed, so after 2020-13-31 all of the affected certificates were revoked.

Unfortunately mistyped the date in comment #2, following the corrected comment:

I just wanna confirm, that the revocations were done in time as it was planed, so after 2020-12-31 all of the affected certificates were revoked.

(In reply to Varga Viktor from comment #0)

During the self-audit on 20th November 2020 we have identified that we have to consider the EVGL 1.7.4 cleanups changes as significant changes which means that we should have put CAB Forum Organization ID in the certificates.

Can you clarify what you mean by this? What of the changes in 1.7.4 were applicable here?

From the description of the issue, it appears you're describing the changes from SC17, introduced in EVGL 1.7.0 in May 2019, with an effective requirement on 2020-01-31.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

There were/are contradictions in the Section 9.8, which led to different understanding.

During the self-audit we have identified that the EVGL 1.7.4 cleanups changes are significant changes in meanings.
In the Relevant dates section the following line changed:
"If subject:organizationIdentifier is present, the Subject Organization
Identifier Extension MUST be present."
The new line:
If subject:organizationIdentifier is present, the CA/Browser Forum
Organization Identifier Extension MUST be present. "

This means that after this date it was mandatory to add this extension to EV certificates.
This extension was not set in our issued EV certificates, because the EVGL Section 9.8 mandates only the extension, which were indicated as "Required."

I'm not sure I understand the contradiction you're highlighting. A subject field is not an extension, so there's no possibility of confusion here, in as much as the absence of an extension is the absence of an extension. Similarly, 9.8.2 mandates the presence with an effective date, so that seems independent - and unconflicted.

Can you explain more?

After we identified this change, we issued certificates (24) to our customers, and ask them to change the certificates on their servers.

The deadline of these changes for most certificates is December 7, 2020, but one customer (PSD2 aggregator) has asked longer time to change it in multiple banking systems.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Steps:

  • 2020-11-20 The problem was identified, the issuance was stopped.
  • 2020-11-23 The extension is set to mandatory for new certificates
  • 2020-12-01 Replacement certificates were issued.
  • 2020-12-07 Revocation of almost all certificates.
  • 2020-12-31 Revocation of the last certificate.
  1. Please file a separate issue for the delayed revocation, which is an independent violation of the BRs. Please use that new issue to discuss the steps you're taking to ensure that you do not delay future revocations, even for PSD2 aggregators, beyond the time permitted by the BRs, as well as the failure to adhere to https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
  2. Please address, in this issue, the steps Netlock is taking to address its failure to adhere to the EVGs for nearly a year (of non-compliance), for changes introduced 1.5 years ago. This incident report should provide full and complete details for how Netlock reviews each and every change to the relevant documents, how it identifies ambiguities that they may find, and how it confirms interpretations and understanding is correct.
Flags: needinfo?(varga.viktor)

Dear Ryan,
We are preparing the new ticket and the answer to your questions.
Yours, Viktor

(In reply to Ryan Sleevi from comment #4)

Can you clarify what you mean by this? What of the changes in 1.7.4 were applicable here?
From the description of the issue, it appears you're describing the changes from SC17, introduced in EVGL 1.7.0 in May 2019, with an effective requirement on 2020-01-31.


The SC35 (in 1.7.4) corrected the ambiguous sentence that was added to the document by SC17 (in 1.7.0).
The small change on page iv in the table of relevant dates.
in the first row of the table is actually a significant change to the previous content. According to its new meaning, if the organizationIdentifier field is present, then the CA/Browser Forum Subject Organization Identifier Extension field shall also present.



I'm not sure I understand the contradiction you're highlighting. A subject field is not an extension, so there's no possibility of confusion here, in as much as the absence of an extension is the absence of an extension. Similarly, 9.8.2 mandates the presence with an effective date, so that seems independent - and unconflicted.

Can you explain more?


In version 1.7.0, the Subject Organization
Identifier Extension was not found in Section 9.8.2, so no modification requirement was identified, so cabfOrganizationIdentifier was not set in the certificate profiles.

This position was confirmed by the fact that, according to Chapter 9.8, the content of the chapter is applicable only if the requirement has been marked "Required" and requirement 9.8.2 has not been so.

With the release of 1.7.4, the relevant date already refers to the cabfOrganizationIdentifier, so we took immediate action to replace the certificates.

In our opinion, "Optional (but see below)" still found in 1.7.4 is still not entirely clear, it would be better to use "Required (if conditions met)" instead.



Please address, in this issue, the steps Netlock is taking to address its failure to adhere to the EVGs for nearly a year (of non-compliance), for changes introduced 1.5 years ago. This incident report should provide full and complete details for how Netlock reviews each and every change to the relevant documents, how it identifies ambiguities that they may find, and how it confirms interpretations and understanding is correct.


Netlock electronically monitors the regulations governing its activities.
We are informed about changes in EU legislation concerning certificates via an RSS feed. For the monitoring of Hungarian legislation we have paid subscription.
For ETSI specifications, a subscription to the change notification provided on the ETSI website is in use. In the absence of a better opportunity for CAB Forum changes, we use a website change monitoring service, whixh alerts us about any change on EV or CAB forum documents page.

When such a change occurs, 2 designated persons (one is myself) examine the changes.
The changes are translated into hungarian, interpreted, discussed and evaluated, and then the necessary change steps are initiated.

In order to avoid a similar interpretation problem in the future, the interpretation of each change in the future will be performed by the entire Compliance team.

From a mathematical point of view, increasing the number of staff can reduce the risks, because there is a greater chance that we will encounter conflicting opinions and interpretations, if the wording is not clear.

Flags: needinfo?(varga.viktor)

Ben: Not sure how this dropped off.

It's not the most inspiring response, for sure. It does not seem like NetLock is actively monitoring the CA/Browser Forum in a way that would allow for them to both productively contribute as well as ensure they have the necessary understanding. Even simply passively monitoring the list discussions should have resolved any presumptive ambiguity.

While the work for CA/B Forum Profiles (currently, only focused on the BRs) will improve this, I do have deep reservations about Netlock's ability and skills for compliance (e.g. Bug 1676367 and Bug 1586795 both show Netlock's inability to understand existing requirements/update appropriately)

But since I have no further questions, sending this to you.

Flags: needinfo?(bwilson)

Hello Viktor,
Could you please respond to Ryan's Comment #7?
Thanks,
Ben

Flags: needinfo?(bwilson) → needinfo?(varga.viktor)
Flags: needinfo?(bwilson)

It appears that Viktor is no longer with Netlock. "In NETLOCK specific cases, please write to the compliance@netlock.hu "

Flags: needinfo?(bwilson)
Flags: needinfo?(varga.viktor) → needinfo?(bwilson)

Dear Ben,
Unfortunately we have to agree with you on the fact that there were several shortcomings in the past in regards to monitoring policy changes and implementing these in a timely fashion, error free. However we are committed to remedying the situation.
There have been several changes in the organization lately, we have a new CEO and the company parted ways with Viktor Varga, as you probably already know, who was one of the entities responsible for monitoring, and policy management. Our new leadership is very dedicated to getting things back on track and have appointed Zoltán Kővári-Szabó, our senior compliance manager to overview the related tasks and policies. We are hopeful that these processes are now in good hands.
We have also made some changes to the previous processes where we have added more weight to the two-man rule, to make sure that it not only applies to decision making but also for monitoring purposes. In addition we have implemented a new risk management software which we hope will help us track changes and warn us of potential threats also relating to policy changes, thereby improving the visibility of policy changes, implementation, testing, and monitoring.
We hope that these changes will help us regain lost confidence in Netlock and prove to you that we are committed to continuous improvement.
Best regards,
Zsófia Fehér

Hello Zoltán,
Could you please respond to Ryan's Comment #7?
Thanks,
Ben

Flags: needinfo?(bwilson) → needinfo?(kovari-szabo.zoltan)
Assignee: varga.viktor → kovari-szabo.zoltan

(In reply to Ben Wilson from comment #11)

Hello Zoltán,
Could you please respond to Ryan's Comment #7?
Thanks,
Ben

Dear Ben,
I would like to respond to the comment#7, but unfortunately I do not see any question in that comment. Moreover Ryan finished his comment that he does not have further questions.
If you could clarify what is the question I would answer that asap.

Thanks,
Zoltan

Flags: needinfo?(kovari-szabo.zoltan)

Hi Zoltan,
Does NetLock actively monitor discussions on CA/Browser Forum mailing lists and on the Mozilla Dev Security Policy mailing list?
Does Netlock have plans to productively contribute to such discussions, as appropriate or as needed, in the future?
Finally, in addition to Netlock taking the improvement measures mentioned in Comment #10 by Zsófia Fehér, what is Netlock doing to improve compliance based not only on policy changes, but also on the errors and shortcomings of other CAs? (i.e., in Bugzilla, see open and closed bugs in the Component "CA Certificate Compliance" under the Product "NSS".)
Thanks,
Ben

We are planning our new monitoring system that will allow us to avoid such errors.
As i have mentioned in another ticket Zsófia Fehér's employment has been terminated and we should review some of her plans, but I can confirm her claim that I am responsible for productively contributing to such discussions.
We would like to ask your patience until the fully comprehensive answer.

Ben: Comment #14 (and the lack of subsequent updates despite recent discussions) sort of reiterate the concerns in Comment #7. The lack of a concrete deliverable date in Comment #14 also suggests that, no, NetLock is not yet monitoring other CA incidents, or else they would know that it doesn't fit with expectations

That said, I'm not sure if you're expecting something more. I think we continue to see the pattern of concern, and that it's unlikely to be remediated anytime soon.

Flags: needinfo?(bwilson)

Hi Zoltán,
Will Netlock be providing any updated or additional information regarding "Steps your CA is taking to ensure that such situations or incidents will not be repeated in the future"? In Comment #14 you say that there will be a more comprehensive answer.
Thanks,
Ben

Flags: needinfo?(bwilson) → needinfo?(kovari-szabo.zoltan)

Dear Ben!
Thanks for your patience!
NETLOCK has setup a new taskforce to be able to handle the tasks and deadlines your comments mentioned and you are concerned about.
The taskforce has members from compliance, product management and the delivery team. Because of that we believe no update or discussion will go without checking the relevant information and requests.
The taskforce’s e-mail address is requirements_check@netlock.hu.
The former process was only dedicated to one individual inside the company, therefore misunderstanding certain topics was built into the system. To be able to eliminate such coded error all new requirements will be discussed in the team before starting the required processes to fulfill the changes in any area responsible for such change (compliance, or technical).
By registering to the mailing lists, we plan to start discussions on topics we have disagreement on internally, so before any implementation we will have understanding on the requirements. Also, being the member of such lists will allow us to help other CA’s on topics we are also working on.

Flags: needinfo?(kovari-szabo.zoltan)

Dear Ben,
do you need any more information on this?

By the way I would like to inform you, that my employment in NETLOCK is terminating on 12.11.21. From 12.11.21 you can discuss with Győző Drozdy (drozdy.gyozo@netlock.hu) and Anna Bányai (banyai.anna@netlock.hu) related to NETLOCK's cases.

Sincerely,
Zoltán

Flags: needinfo?(bwilson)
Assignee: kovari-szabo.zoltan → banyai.anna
Flags: needinfo?(bwilson)
You need to log in before you can comment on or make changes to this bug.