Bugzilla

Reporter

Updated

•

3 years ago

Assignee: bwilson → kovari-szabo.zoltan

Status: NEW → ASSIGNED

Comment 1

•

3 years ago

As I just recognised, the audit report contains a wrong SHA256 hash related to "NetLock OnlineSSL (Class Online) Tanúsítványkiadó" CA.
We are contacting to the Auditor asap to fix it.

Comment 2

•

3 years ago

I filed also an incident report: https://bugzilla.mozilla.org/show_bug.cgi?id=1718517

Reporter

Comment 3

•

3 years ago

Please follow the format provided in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report and provide the incident report in this bug. I will close the other bug as a duplicate.

Reporter

Updated

•

3 years ago

Flags: needinfo?(kovari-szabo.zoltan)

Comment 5

•

3 years ago

Note that, as part of providing the incident report (Comment #3), please be very clear about indicating what audit report(s) this intermediate has appeared in, from the moment the key was created until now.

Comment 6

•

3 years ago

(In reply to Ben Wilson from comment #3)

Please follow the format provided in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report and provide the incident report in this bug. I will close the other bug as a duplicate.

How we became aware first of the problem:
We first became aware of the problem via comment#1 of this current ticket. It was on 16.06.2021.
A timeline of the actions:

21.10.2020: Our latest audit report was issued.
Our compliance team checked the report on the next few days after the report was issued. A former member of our compliance team (Viktor Varga) was responsible for making a full check including the technical details.
27.10.2020: According to Viktor Varga's feedback, the report was acceptable. (It seems he did not recognised, that the hash of 'NetLock OnlineSSL (Class Online) Tanúsítványkiadó' CA was badly included in the report).
04.11.2020 10:17am: Viktor Varga uploaded the audit report to CCADB. See Case 00000678 on CCADB (https://ccadb.force.com/s/case/5001J00000xJM5FQAW/2020-audit-netlock-ltd)
05.11.2020 4:22pm: Case 00000678 was closed on CCADB.
16.06.2021: We first became aware of the problem via comment#1 of this current ticket.
25.06.2021: We checked the report and recognised that the audit report contains a wrong SHA256 hash related to "NetLock OnlineSSL (Class Online) Tanúsítványkiadó" CA.
28.06.2021: We contacted to the auditor via phone and e-mail asking them to fix the error.
29.06.2021: Auditor sent the improved audit letter, that was published on https://it.certop.com/wp-content/uploads/2021/07/Certop_cert_Netlock_with_annex_english_2020_final_3_sign.pdf

The process giving rise to the problem or incident and explanation on why we have not stopped the CA:
It was just an administration error in making the audit letter that was not recognised by my former coworker (Viktor Varga) who was responsible for the last check on audit letters before uploading to CCADB.
A summary of the problematic certificates:
A single intermediate CA certificate is involved: 'NetLock OnlineSSL (Class Online) Tanúsítványkiadó'.
The complete certificate data for the problematic certificates:
SHA256 hash: CC:9F:72:54:4F:F6:17:C1:4C:EB:E8:28:31:94:A7:0F:1E:89:81:CF:16:3D:5A:3F:D8:DC:CF:D8:46:D5:DC:1A

Certificate Data:
Version: 3 (0x2)
Serial Number:
49:41:2c:e4:00:27
Signature Algorithm: sha256WithRSAEncryption
Issuer:
countryName = HU
localityName = Budapest
organizationName = NetLock Kft.
organizationalUnitName = Tanúsítványkiadók (Certification Services)
commonName = NetLock Arany (Class Gold) Főtanúsítvány
Validity
Not Before: Aug 7 15:19:06 2012 GMT
Not After : Oct 23 15:19:06 2027 GMT
Subject:
countryName = HU
localityName = Budapest
organizationName = NetLock Kft.
organizationalUnitName = Tanúsítványkiadók (Certification Services)
commonName = NetLock OnlineSSL (Class Online) Tanúsítványkiadó
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:d8:40:44:3e:e8:33:62:9c:2d:b4:3b:67:38:9f:
8c:80:dd:2b:f9:1b:c9:50:6f:44:58:51:e8:11:f0:
8d:33:ac:ff:ec:2d:88:bf:dd:dd:ab:fc:05:8f:7f:
2c:a5:d8:c4:42:fa:88:65:74:a8:2a:33:7d:cc:fa:
f0:0e:82:ad:70:d4:e8:e3:f7:d4:f9:37:1b:43:db:
2e:56:a8:6f:7f:96:b9:ac:30:c5:9a:bb:2a:9c:ab:
52:e8:d1:ab:9f:78:8c:71:80:da:f0:23:74:24:e1:
29:65:a6:f6:1e:38:1e:fd:9f:98:84:28:a6:38:c5:
aa:60:a5:ae:9e:db:0b:aa:66:11:b0:fd:96:d3:07:
f6:e6:b8:48:01:1d:49:21:bc:73:52:f8:46:1e:16:
ff:51:23:6e:33:8b:1f:a8:31:69:a1:32:11:42:be:
39:d4:51:27:c5:4f:c1:6b:92:9f:b6:f4:1c:6a:74:
92:b8:8c:91:c7:6c:38:ad:93:56:c5:d4:90:e1:d8:
8b:8f:23:97:f0:1b:75:b0:8e:56:5a:4e:da:74:6d:
2d:e1:4c:f2:33:72:8b:1f:ff:e5:eb:ec:75:0d:e7:
56:cf:cf:0b:c5:00:99:1f:d3:50:32:cd:32:15:b9:
71:9a:52:1a:5c:16:e4:2a:d7:f2:8e:71:d6:20:f9:
18:65
Exponent: 37779 (0x9393)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:4
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Authority Key Identifier:
keyid:CC:FA:67:93:F0:B6:B8:D0:A5:C0:1E:F3:53:FD:8C:53:DF:83:D7:96

        X509v3 Subject Key Identifier: 
            9B:15:98:EA:B2:4C:F9:EA:CC:81:57:73:C2:DA:51:E9:79:8C:EE:FD
        Authority Information Access: 
            OCSP - URI:http://ocsp1.netlock.hu/gold.cgi
            OCSP - URI:http://ocsp2.netlock.hu/gold.cgi
            OCSP - URI:http://ocsp3.netlock.hu/gold.cgi
            CA Issuers - URI:http://aia1.netlock.hu/index.cgi?ca=gold
            CA Issuers - URI:http://aia2.netlock.hu/index.cgi?ca=gold
            CA Issuers - URI:http://aia3.netlock.hu/index.cgi?ca=gold

        X509v3 CRL Distribution Points: 

            Full Name:
              URI:http://crl1.netlock.hu/index.cgi?crl=gold

            Full Name:
              URI:http://crl2.netlock.hu/index.cgi?crl=gold

            Full Name:
              URI:http://crl3.netlock.hu/index.cgi?crl=gold

Signature Algorithm: sha256WithRSAEncryption
     13:92:53:60:ba:c1:8c:35:d8:71:8e:a2:8a:98:71:45:a5:b3:
     0c:16:ae:b3:fa:c0:c2:73:f7:e5:2e:f5:fa:e7:d6:7f:f8:95:
     e9:20:95:63:a5:fa:87:51:1f:9d:e3:82:f9:1e:06:8f:74:77:
     08:d1:d8:65:bc:eb:e5:21:42:b3:d8:cc:e1:54:d8:8a:6a:32:
     48:74:9d:56:77:d3:1f:e0:15:77:07:75:33:a6:11:6a:d6:c1:
     a4:84:bd:2e:4a:ab:10:f0:94:c4:5e:2e:1a:7a:4a:f4:0a:8e:
     15:4c:5d:8a:b4:46:ff:c2:8c:c8:19:68:a5:c9:f9:78:ae:b5:
     db:44:cb:a0:fc:0c:c4:8c:de:ea:72:64:5c:85:92:8f:ce:cb:
     b4:02:87:4c:10:ec:e2:45:8a:2e:fb:fb:1f:33:72:23:05:ad:
     ee:5e:3e:2e:21:42:8c:af:ef:76:08:8e:75:c8:c1:2d:ca:26:
     22:9c:7c:6c:3e:79:d1:0c:e5:d3:a5:e3:72:33:e1:5b:c3:1b:
     a3:8d:01:4e:d9:66:db:61:28:6b:47:7d:f3:3b:09:5d:3e:10:
     ec:28:7d:b6:6b:39:09:15:c3:8e:35:9c:b6:f6:08:29:69:b2:
     7e:45:fe:a8:a4:9c:f6:92:cd:7b:8c:92:d0:41:e1:01:63:bb:
     cf:37:2a:9b

Explanation about how and why the mistakes were made, and how we avoided detection until now:
It was an administration error that was made by the auditor in making the audit letter. We avoided detection the error until now, because our former coworker (Viktor Varga) who was responsible for checking all technical details in the audit letter did not recognised this error.
List of steps to resolve the situation and ensure that such situation or incident will not be repeated in the future:

Due to such errors, personnel changes have been made in NETLOCK. The responsible coworkors' employment were terminated, including the former head of compliance.
We asked and got an improved audit letter (see details in the point no.2)
Until the next audit we will review our processes on checking, accepting and uploading the audit letter.

Flags: needinfo?(kovari-szabo.zoltan)

Comment 7

•

3 years ago

(In reply to Ryan Sleevi from comment #5)

Note that, as part of providing the incident report (Comment #3), please be very clear about indicating what audit report(s) this intermediate has appeared in, from the moment the key was created until now.

Please notify me if you need any more details.

Comment 8

•

3 years ago

(In reply to Zoltán Kővári-Szabó from comment #7)

(In reply to Ryan Sleevi from comment #5)

Note that, as part of providing the incident report (Comment #3), please be very clear about indicating what audit report(s) this intermediate has appeared in, from the moment the key was created until now.

Please notify me if you need any more details.

I'm not seeing the requested details.

The goal here is to clearly identify whether this is a persistent issue across multiple audit reports, or an issue with the most recent audit report.

Please understand that the default expectation is immediate and prompt revocation of this intermediate. Netlock bears the burden to demonstrate why that's not necessary: why and how the community can objectively obtain assurance using the recognized means, rather than creating ad-hoc special exceptions to the BRs and policy when convenient.

So was this correct on previous reports? If so, please provide their details. Our goal is to have a better picture about the history for this CA. This is separate to/in addition to understanding what happened and the steps to prevent it in the future.

Flags: needinfo?(kovari-szabo.zoltan)

Reporter

Updated

•

3 years ago

Flags: needinfo?(bwilson)

Comment 9

•

3 years ago

It's been 11 days since Comment #8, which is inconsistent with https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

I would encourage you to review past discussions, to see if they provide greater clarity about the expectations here.

Comment 10

•

3 years ago

(In reply to Ryan Sleevi from comment #8)

(In reply to Zoltán Kővári-Szabó from comment #7)

(In reply to Ryan Sleevi from comment #5)

Note that, as part of providing the incident report (Comment #3), please be very clear about indicating what audit report(s) this intermediate has appeared in, from the moment the key was created until now.

Please notify me if you need any more details.

I'm not seeing the requested details.

The goal here is to clearly identify whether this is a persistent issue across multiple audit reports, or an issue with the most recent audit report.

Please understand that the default expectation is immediate and prompt revocation of this intermediate. Netlock bears the burden to demonstrate why that's not necessary: why and how the community can objectively obtain assurance using the recognized means, rather than creating ad-hoc special exceptions to the BRs and policy when convenient.

So was this correct on previous reports? If so, please provide their details. Our goal is to have a better picture about the history for this CA. This is separate to/in addition to understanding what happened and the steps to prevent it in the future.

Fair enough. The necessary details you have asked are really missing. I try to summarize these shortly.

First of all, unfortunately as we have checked this is a persistent issue across multiple audit reports.
Last two weeks we carried out an inner audit while we were investigating the cause of the issue.

As we realised, the source of this mistake was one of our web pages (https://www.netlock.hu/index.cgi?lang=HU&ca=olsslgca&tem=ANONYMOUS/kulcsjegyzok/adatok.tem). On this page the related hash was presented badly (it has been fixed). Unfortunately when our auditors made the report, they copied it to their reports without checking.

Related to the default expectation, in our opinion this issue is only an administrative mistake, that does not affect the trust in this intermediate CA. Every technical, security and policy requirements that are applicable related to this CA were met. This intermediate certificate and the end user certificates issued by this are compliant with the related technical, security and policy requirements that are applicable.

Considering the above we would like to avoid the revocation this CA immediately.

As an additional measure to the point no7 of the comment#6 we are going to draw the auditor's attention to the importance of the correctnes of the reports, and that they must not copy anything without checking to avoid such administrative issues.

Flags: needinfo?(kovari-szabo.zoltan)

Comment 11

•

3 years ago

(In reply to Zoltán Kővári-Szabó from comment #10)

Related to the default expectation, in our opinion this issue is only an administrative mistake, that does not affect the trust in this intermediate CA. Every technical, security and policy requirements that are applicable related to this CA were met. This intermediate certificate and the end user certificates issued by this are compliant with the related technical, security and policy requirements that are applicable.

By your own admission, however, you have no way to actually demonstrate this statement. The accepted form of demonstrating this was not, and cannot be, provided.

Considering the above we would like to avoid the revocation this CA immediately.

I have no doubt in this statement. But the issue is that you, as the Root CA, bear the responsibility - and the liability - for the mistake here.

You may feel it's unfair to assume the worst possible thing from a "simple" mistake, but that has always been upfront the expectation of what happens in these mistakes. It's part of treating CAs consistently, and it's part of ensuring users are secure: we cannot simply take Netlock's word at it. Administrative mistakes can have very real, and very profound, security consequences, and we treat them as such, just like if a CA incorrectly issued a basicConstraints: CA=true certificate as an administrative mistake (as has happened in the past).

This might seem incredibly inflexible. But Netlock agreed to the terms, and failed to deliver, in terms of having something unambiguously audited and in scope. You've suggested this was just a typo, but we should be explicit here: what's disclosed is a completely different CA. This is functionally no different than an unaudited CA, and as mentioned, the expectations here are clear, precisely to avoid situations that expose our users at risk.

I can appreciate the concern of "If it wasn't detected until now, is it really important" - and the answer is yes, it is. It's always been important and long been an expectation, and the tooling has been to make sure we don't miss these situations. We currently rely on the CA to do things correctly, and we expect that when the CA does things incorrectly, they'll revoke, just as they've promised to do. When they fail to do both of those things, it's a matter of significant concern.

Comment 12

•

3 years ago

This cert (https://crt.sh/?q=A4C463FAA16CC470A829B058058F1E3DFA8B839797B3861819F75807CAF1F02C) is really a totaly different one and has been expired in 2014.

In the audit letters the related other data, the "short name in structure", the "full name" and the "CA availability" are correct. I mean that these data present this CA: https://crt.sh/?q=CC9F72544FF617C14CEBE8283194A70F1E8981CF163D5A3FD8DCCFD846D5DC1A

We has started discussing the situation with our former auditors. We are asking them to issue a statement about that they audited the intermediate CA, that has the right hahs (CC:9F:72:54:4F:F6:17:C1:4C:EB:E8:28:31:94:A7:0F:1E:89:81:CF:16:3D:5A:3F:D8:DC:CF:D8:46:D5:DC:1A).

Győző Drozdy

Comment 13

•

3 years ago

Attached file NETLOCK_Cert_statement_20210720_signed.pdf — Details

Matrix auditor statement

Győző Drozdy

Comment 14

•

3 years ago

Dear Ryan,
we take your concerns very seriously, so our team has worked hard to untangle this issue and come up with a transparent solution.
First, we checked the related links on our website, also previous internal and audit documentations, and our communication with the auditors. We have also set up meetings with our auditor companies from previous years (Certop and Matrix), walked through the process step-by-step that resulted current situation, and thoroughly discussed possible ramification of this sensitive issue. Ultimately, we can could jointly establish beyond doubt: no users have been exposed to any security risk.

It is true that we had used the particular certificate we listed on our website (https://crt.sh/?q=A4C463FAA16CC470A829B058058F1E3DFA8B839797B3861819F75807CAF1F02C) before 2014. That certificate has been expired in 2014. After that time, nobody has ever used it so it is safe to say that no users were at risk.
Later, we started to use a new CA (https://crt.sh/?q=CC9F72544FF617C14CEBE8283194A70F1E8981CF163D5A3FD8DCCFD846D5DC1A), and set up said CA’s data on our website. Even though all the "short name in structure", the "full name", the "CA availability", and others were correctly updated, the hash we listed remained unchanged (i.e. from the previous certificate).
For the audits of the intermediate CA, we shared required data, documentation, and public links with our auditors. It seems that using above data as single point of truth for the description of the CA lead to an error of purely administrative nature in the documents.
Our auditors double checked our claims and that they erroneously listed the expired hash in the audit reports, but they were positive that this administrative mistake does not affect the legitimity and the end results of the audit reports in any way. For this reason, they accepted our request and issued statements for the correction of the particular (hash) data field.

Please find the signed statement of the above correction from our auditor Matrix uploaded. Additionally, you can find the correction of the CA on page 3 of Certop’s audit report "Annex to certificate 20-005783/19-00170" using the following link: https://it.certop.com/tanusitvanyok/

We hope that you will find all your previous concerns are addressed by our efforts, and accept the results of our investigation with the auditor companies, as well as the auditors official statements regarding the necessary administrative correction of the reports.

Thank you!

Comment 15

•

3 years ago

Unfortunately, this statement does not and cannot remediate the issue.

I realize NETLOCK may feel this is unjustified, but quite simply: We cannot be creating adhoc procedures like this, as stated in Comment #8. To be clear: NETLOCK had an obligation to deliver one thing, they failed to do so for a series of years. It does not seem like there's any disagreement here: NETLOCK failed to do what was expected of them, and every other CA.

The discussion in Comment #14 mostly is a request by NETLOCK to treat them special, rather than apply the same consistent standards and expectations of all CAs, on the basis of the reason why it happened. NETLOCK has attempted to create an ad-hoc process to amend reports here, but clearly, Comment #14 both demonstrates a lack of attentiveness by NETLOCK and a lack of attentiveness by the auditor - the latter undermining any faith in the correction. NETLOCK's own proposed path forward here fails to meet Mozilla's clearly stated guidance as well as Google's stated expectations

I hope we're in clear agreement: NETLOCK has failed to meet the BRs and failed to meet the requirements of multiple root programs. There are a number of options available to NETLOCK to get back into compliance, but it appears to have intentionally chosen not to follow them. The "why" is important to prevention of future incidents, but it does not alter the expected remediation.

Győző Drozdy

Comment 16

•

3 years ago

Dear Ryan,
while we believe that we have tried to objectively present the situation and close this issue, based on your answer we see no other remedy to this situation other than to revoke the intermediate. We are working on our plan to notify customers while issuing new certificates for them. We will get back to you as soon as possible.

Reporter

Updated

•

3 years ago

Flags: needinfo?(drozdy.gyozo)

Comment 17

•

3 years ago

Dear Ben,
We are scheduling the process that will include the following:

```
  informing the endusers
```

  revocation the related enduser certificates

  revocation the related intermediate CA

  reissuing the related enduser certificates from an other intermediate CA

We are going to share the detailed shcedule with you asap.

Comment 18

•

3 years ago

Comment #3 discussed the expectations of incident reports.

To reiterate, https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed sets out expectations for communication, including weekly progress reports. This wasn't provided in light of Comment #16, and Comment #17 bears the problematic signs of lacking a concrete date for follow-up, also an expectation. "ASAP" could mean 3 hours or 3 months, as it's a subjective view, and so it's important to transition to objective statements.

Flags: needinfo?(bwilson) → needinfo?(kovari-szabo.zoltan)

Comment 19

•

3 years ago

Dear Ryan and Ben,
the schedule is ready:

information in advance on expected certificate exchange for customers – on 23.08.21
creation and backup of new intermediate CA – on 25.08.21
disclosing the new intermediate CA in the CCADB and on NETLOCK’s webpage – on 26.08.21
publishing a new version of related CPS supplemented with the new intermediate CA – on 26.08.21
systems configuration and test for using the new intermediate CA – 25.08.21-03.09.21
detailed information on certificate exchange and specific things to do, for customers – on 06.09.21
the new version of CPS will enter into force – on 08.09.21
automatic reissuance of enduser certificates by the new intermediate CA, without subject data change – 08.09.21-09.09.21
revocation of the former enduser certificates – on 22.09.21
revocation of OnlineSSL intermediate CA – on 23.09.21

Flags: needinfo?(kovari-szabo.zoltan)

Comment 20

•

3 years ago

Update on schedulation:
We have carried out the steps 1-3.
The new intermediate CA cert's CCADB URL: https://ccadb.force.com/s/account/0014o00001nq4uQAAQ/netlock-dvssl-ca
URL on NETLOCK's website: https://www.netlock.hu/index.cgi?ca=DVCA&lang=HU&tem=ANONYMOUS/kulcsjegyzok/adatok.tem

Comment 21

•

3 years ago

Update on schedulation:
We have carried out the steps 1-8.
We are going to revoke of the the former end user certs on 22.09.21, then of the OnlineSSL CA on 23.09.21.

Comment 22

•

3 years ago

Update on schedulation:
As one of our clients asked us, unfortunately we have to postpone the revocation steps. This client has 42 certificate issued by OnlineSSL CA, and as they mentioned, it is impossible for them to change the certs on every server until 22.09.21.
So we have modified the schedulation above to the following:
9. revocation of the former enduser certificates – on 27.09.21
10. revocation of OnlineSSL intermediate CA – on 28.09.21

Comment 23

•

3 years ago

Dear Ryan,
we kindly inform you that all steps above have been done. Do you need any more information on this?
By the way I would like to inform you, that my employment in NETLOCK is terminating on 12.11.21. From 12.11.21 you can discuss with Győző Drozdy (drozdy.gyozo@netlock.hu) and Anna Bányai (banyai.anna@netlock.hu) related to NETLOCK's cases.

Sincerely,
Zoltán

Flags: needinfo?(bwilson)

Updated

•

3 years ago

Assignee: kovari-szabo.zoltan → banyai.anna

Reporter

Comment 24

•

2 years ago

It appears that this certificate was revoked on 9/28/2021. I will close this bug on or about Friday, 14-Jan-2022, unless there are other issues to discuss.

Nobody; OK to take it and work on it

Reporter

Updated

•

2 years ago

Status: ASSIGNED → RESOLVED

Closed: 2 years ago

Flags: needinfo?(bwilson)

Resolution: --- → FIXED

Updated

•

2 years ago

Product: NSS → CA Program

Chris Clements

Updated

•

1 year ago

Summary: Netlock: CA Certificate Missing from Audit Reports → NetLock: CA Certificate Missing from Audit Reports

David Lawrence [:dkl]

Updated

•

1 year ago

Whiteboard: [ca-compliance] → [ca-compliance] [audit-failure]