1905509 - NETLOCK: CPR was not responded to in 24 hours

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne]

On 2024-06-28 19:45 UTC the below email was sent to '[e-mail]' - the contact listed in 1.5.2 of every one of NETLOCK's CP/S. No reply has been received within 24 hours. Below is a copy of the email for transparency:

Hello,

This is the email listed in the current CPS covering the 'NetLock Arany (Class Gold) Fotanúsítvány' root.

1: This is not listed on CCADB's Problem Reporting Mechanism's for NETLOCK which only provides:

• visszavonas[at]netlock[dot]hu
• https://netlock.hu
• compliance[at]netlock[dot]hu

Neither email address appears in any CPS I can see - this needs corrected.

2: This Certificate Problem Report has been filed due to a certificate failing both zlint and pkilint: >https://search.censys.io/certificates/51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496

$ lint_cabf_serverauth_cert lint -s ERROR -d 51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496.pem
SubscriberKeyUsageValidator @ certificate.tbsCertificate.extensions.1.extnValue.keyUsage
cabf.serverauth.subscriber_prohibited_ku_present (ERROR): Prohibited KU present: keyEncipherment
CertificatePolicyQualifierValidator @ certificate.tbsCertificate.extensions.5.extnValue.certificatePolicies.1.policyQualifiers.1
cabf.serverauth.prohibited_certificate_policy_qualifier_type (ERROR): Prohibited qualifier type: 1.3.6.1.5.5.7.2.2

$ zlint 51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496.pem
e_ecdsa_allowed_ku, e_mp_ecdsa_signature_encoding_correct e_policy_qualifiers_other_than_cps_not_permitted

Please note that zlint is reporting an additional error.

• Wayne

Now, finding out who to contact is an issue. The policy IDs listed in the certificate are generic, and the cpsuri gets redirected to a very generic page. Checking just the Root isn't helpful as every CP/S mentions it. Luckily every CP/S has the same contact method listed for issues:

1.5.2. Contact person of the document
The responsible contact person of the Policy Adopting Authority shall be the approver of the present document (see the cover page of the document).

Customers, End Users and the Relying Parties may submit their questions and comments related to the present document to the NETLOCK Policy Adopting Authority in e-mail to [e-mail].

As mentioned in the email... this isn't what is listed on CCADB, however the CP/S should be the authority relied on?

Now figuring out which CP/S is applicable was a headache. Which is to say that having checked CCADB I do not believe this intermediary is disclosed. It also does not appear in a single CP/S directly.
https://search.censys.io/certificates/d0eb908401f33242602634afd51991536b3ad7aa901586fdeb4955fe51b0e419
CN: NETLOCK TLS OV ECC CA
SHA256: d0eb908401f33242602634afd51991536b3ad7aa901586fdeb4955fe51b0e419

Looking further this is at least a repeat of #1889570 but for a freshly generated intermediary. Note that the undisclosed intermediary is different than the one stated in #1904041. Given the naming scheme I would not be shocked at more undisclosed intermediaries, but I have also not checked.

The lack of any obviously applicable CP/S for these intermediaries also raises a question I previously posed in #1891331 Comment 28. If no CP/S is covering these intermediaries and the CP/S isn't considered in-force until 30 days have past after the CP/S has been publicly published... what happens, and how do we tie a certificate to a CP/S?

As I see it this is a certificate published that isn't tied to any CP/S, and is from an undisclosed intermediary.

Please note this incident is purely for handling the lack of response from the CPR. Any other issue mentioned will need a separate incident raised by the CA.

Comment 1

[Aaron Gable]

The contact address provided in Section 1.5.2 is for communicating with the people who administer the CPS document itself. It is not necessarily the correct address for submitting a Certificate Problem Report, and since it is not an address disclosed for that purpose, it is not necessarily an incident for a CA to fail to respond to a report there. (For example, sending a message to the PMA physical mailing address that Let's Encrypt discloses in our Section 1.5.2 would not be a valid CPR.)

The address for submitting a Certificate Problem Report is disclosed in CCADB. I believe it should also be disclosed in Section 4.9.3 of the CPS, where the CA is supposed to describe the procedure for submitting a Certificate Problem Report.

Comment 2

[Corey Bonnell]

BR 4.9.3 requires that the CA’s problem reporting mechanism be disclosed in section 1.5.2 of the CPS.

If the only contact information in section 1.5.2 is the email address quoted above, then I agree with the reporter that they attempted to contact the CA using the correct reporting mechanism.

Comment 3

[Aaron Gable]

Ah you're right, I had forgotten that BRs 4.9.3 places requirements on CPS 1.5.2 (an anti-pattern I'd like to fix). It's not obvious to me that there's a failure-to-respond incident here, since it is clear from reading NetLock's Section 1.5.2 and their CCADB disclosures that this address is not intended for receiving CPRs (though I am not opposed to that interpretation of events either). But there definitely does appear to at least be a failure of their CPS to comply with the requirements of BR 4.9.3.

Comment 4

[amir]

I would consider the CCADB database, and the CPS of a CA both as sources of truth regarding CPR endpoints. So in regards to that, and the fact that this is the only email mentioned in section 1.5.2 of the CPS, there is a failure to respond here.

Furthermore, this incident is detailing that none of the documents take ownership over this intermediate - which might even mean no CPS applies to it?

I can confirm that CCADB does not have this entity listed https://ccadb.my.salesforce-sites.com/mozilla/PublicAllIntermediateCerts (nothing under "NETLOCK TLS OV ECC CA")

Beyond that, this CA has issued the following certificate: https://search.censys.io/certificates/51c3bd8e8aff40028016bd452405042ead0d8f64c1a2221df088cce5338d1496 that is failing 2 lints:

e_mp_ecdsa_signature_encoding_correct
n_ecdsa_ee_invalid_ku

There also seems to be a bunch more undisclosed intermediates that's showing up on Censys: parsed.issuer.common_name="NETLOCK*ECC*". There may even be more that's not indexed by censys/not found by that query.

For example: https://crt.sh/?id=13467134365

---

In a larger scale discussion, I do hope that root programs can consider making a requirement for CAs to explicitly use the terms "Certificate Policy" and "Certification Practice Statement" in their documents so we're not having to distinguish what word actually means what. In the case of Netlock here, the document is called "Service Practice Statement".

Comment 5

[Dimitris Zacharopoulos]

Sectigo has built a nice query that shows undisclosed intermediates in https://crt.sh/mozilla-disclosures#undisclosed.

Comment 6

[Rob Stradling]

There also seems to be a bunch more undisclosed intermediates

See bug 1904041.

Comment 7

[Nikolett]

Hello everyone!

Thank you for the ticket.

1. • We have created a new bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1906115 inconnection with the contact informations. We will update Section 1.5.2 accordingly.
2. • About the certification program:

User Notice: We are introducing new ECDSA certificates alongside the existing RSA certificates. The ECDSA certificates will operate on a brand new system, which requires new configuration. The new system is still under configuration. Due to an administrative error, an incorrect user notice was used in the configuration. We have identified the error, and the faulty certificates will be revoked in advance, with new certificates issued. Our RSA system does not use the faulty user notice, and we are making efforts to configure our certificates according to the existing regulations.

Key Usage: As we mentioned before, our previous system used the RSA algorithm. Due to an administrative error, we used 'Key Encipherment' instead of 'Key Agreement'. We investigated the problem and found that, unfortunately, all ECC TLS certificates were issued with this error. As a result, we revoke the faulty certificates and issue new ones.

Policy:
We used the 1.3.6.1.4.1.4146.1.1 GlobalSign policy for EV certificates instead of the general EVTLS 0.4.0.2042.1.4 policy. This configuration error was due to an administrative mistake, tracing back to the previous RSA system. We have corrected the configuration, we revoke the incorrect certificates, and will issue new ones.

The linters previously created in the RSA system were also adapted to the ECC system, as a result of which similar errors will not occur in the future.

As for the CN: NETLOCK TLS OV ECC CA
SHA256: d0eb908401f33242602634afd51991536b3ad7aa901586fdeb4955fe51b0e419
The CP/S change is in progress. You have not been able to see the CN in our current applicable policy version, because an update is in preparation, it will come into force on the 18th of July.

On our website we have a dedicated place for the drafts of the CP/S where you can see them in a 30 day period:
https://netlock.hu/drafts#others

Comment 8

[Ben Wilson]

Was this other bug mistakenly created - Bug #1906115?

Comment 9

[Wayne]

Confirming I have now received an email reply (14:59 UTC), and also a duplicate seems to have been created. Roughly 4 days, 19h turnaround period.

Please consider incidents for the other issues and a full list of impacted certificates. There is also the problem of whether any CP/S covered those even if it is in a 'drafts' section of netlock's website.

Comment 10

[Nikolett]

Incident report

Summary

On Friday 2024. 06. 28. 21:45 CET NETLOCK has received an e-mail about the non-compliant e-mail addresses in CCADB as well as in TSP’s CP/S.
The e-mail also disclosed a problem with a certificate failing both zlint and pkilint making the e-mail a certificate problem report. We were notified of the creation of the corresponding bug ticket at Saturday 2024. 06. 29. 21:46 CET.

Impact

NETLOCK’s lack of timely response did not meet the Baseline Requirements for the Issuance and Management of Publicly Trusted TLS Server Certificates, section 4.9.5, which states “within 24 hours after receiving a Certificate Problem Report, the CA SHALL investigate the facts and circumstances related to a Certificate Problem Report and provide a preliminary report on its findings to both the Subscriber and the entity who filed the Certificate Problem Report.”

Timeline

Action timestamp	Action taken
2024-june-28 21:45	From [e-mail] the e-mail arrived to our central address ([e-mail])
2024-june-29 21:46	Bugzilla ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1905509 was created by [e-mail] (Wayne) because of the lack of response.
2024-july-01 11:21	Compliance asked for support from the IT Department and an investigation started, assessing the impact of the indicated configuration changes on certificate issuance and infrastructure
2024-july-02 09:15	Following the operational meeting, we implemented a linter configuration refreshment. Testing of the systems continued.
2024-july-03 16:05	NETLOCK has created a new Bugzilla ticket #1906115. We see now that this is a duplicate and this action was not necessary.
2024-july-03 16:59	NETLOCK responded to the original email from [e-mail].
2024-july-04 09:15	Testing continued and cert configuration changed.
2024-july-05 14:11	We have received confirmation of the revocation of the certificates:
2024-july-05 14:36	NETLOCK’s CP/S 1.5.2 was updated with the following: In the event of suspected private key compromise, certificate misuse or other types of fraud, compromise, abuse, misconduct, or any other matter relating to Certificates, you may contact the Service Provider at [e-mail].

The revoked certificates are the following:

• tlseveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 11:57:28
• tlsoveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 14:05:11
• tlsdveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 13:35:57
• tlsqeveccca.valid.ecc.netlock.hu - Revoked: 2024.07.05 13:42:06
• Certificates with the "expired" domain name expired with a validity period of 1 day on 2024.06.21.

Root Cause Analysis

Due to human error the certificate configuration was not correct.

Lessons Learned

In the event of an algorithm change, the previous settings and the certificates issued with the new algorithm will be subject to a broader review.

What went well

The changes in the certificate configuration let us handle this issue on a timely manner and we were able to meet the deadlines for revocation required by the BRG and Mozilla.

What didn't go well

In the new system, the certificate configuration was not correct due to human error.

Where we got lucky

These wrongly configurated certificates completed the testing phase for the ECDSA TLS. No service was provided to customers therefore no external communication and other actions were needed.

Action Items

Action Item	Kind	Due Date
Fix of certification profiles	Repair	2024-07-04
Revoke affected certificates	Repair	2024-07-05
Contact information update in CP/S	Repair	2024-07-05

Comment 11

[Wayne]

Please read the CCADB Incident Response and Mozilla Responding to an incident page.

I cannot list how many things are wrong in this report, it is truly groundbreaking.

e: To be very kind at the absolute minimum:

• CET instead of UTC.
• The mixture of date/time formats
• The impact section
• Mentioning the email address of the reporter
• The report not being about the CPR at all, it is about different incidents that all need separate bugs raised here
• A certificate list with no sha256 hashes
• The Root Cause Analysis section
• The 'lessons learned', repeatedly leaning on human error
• The wrong category for action items and how it is also irrelevant on many layers

We haven't even addressed all of the intermediaries created and not disclosed, then certificates generated off of them failing multiple linters... so much has gone wrong here.

Comment 12

[Dorottya]

Thank you very much for the above feedback, we correct it.

Comment 13

[Dorottya]

Corrected Incident report

Summary
On Friday June 28th 2024. 19:45 UTC NETLOCK has received an e-mail about a problem with a certificate failing both zlint and pkilint making the e-mail a certificate problem report. We apologize for displaying the reporter's email address. We kindly ask the admin to remove it.

Impact
The incident affected the following certifications:

Certification SHA256 fingerprint Revoked/expired
tlseveccca.valid.ecc.netlock.hu 0E:6D:53:3B:CD:21:B6:D7:F4:D3:AD:19:99:18:D6:E9:73:07:E7:E1:A5:E9:0E:2E:15:71:39:55:8F:C4:58:C4 July 5 2024
tlsoveccca.valid.ecc.netlock.hu A2:7C:63:AD:A3:54:08:85:FF:15:E6:A5:ED:61:EB:DE:66:96:CF:0E:1C:FE:18:81:54:32:D5:03:49:52:CD:F8 July 5 2024
tlsdveccca.valid.ecc.netlock.hu 1F:79:86:F4:FB:C9:FB:7E:39:5D:7B:8E:DC:5F:C9:8B:D4:0F:C0:3E:1F:7F:B2:A3:02:0C:E6:1E:E3:B9:F0:32 July 5 2024
tlsqeveccca.valid.ecc.netlock.hu 9D:E0:B5:30:C4:9A:F1:D8:C7:15:1C:E3:3E:46:D5:D1:FB:C8:75:84:5E:11:6C:1D:FF:BF:E9:44:50:F2:4D:23 July 5 2024
tlseveccca.expired.ecc.netlock.hu 67:E4:0E:0C:AC:5D:55:A8:8F:71:D4:D2:02:E3:0B:EE:83:65:7A:17:73:7A:B6:22:39:D0:A4:10:C4:A0:A8:A6 June 21 2024
tlsoveccca.expired.ecc.netlock.hu F3:95:5A:62:62:55:61:D4:5E:37:1F:35:C6:2D:23:E4:02:23:5C:18:E8:A3:B4:95:23:93:AA:0F:9A:6A:CF:C4 June 21 2024
tlsdveccca.expired.ecc.netlock.hu 81:38:DF:47:DC:FB:A1:C4:32:8F:72:CD:69:71:8B:7D:2B:F7:9D:30:FC:86:35:BC:E7:52:A7:FF:71:E7:76:02 June 21 2024
tlsqeveccca.expired.ecc.netlock.hu 2E:B3:CD:38:B7:45:27:1E:74:42:9D:17:14:EA:21:D4:2C:8E:9E:CD:24:96:F2:40:A6:6D:6F:81:6A:26:79:0B June 21 2024

Out of the above, the last four certificates with 'expired' domain names expired within 24 hours of issuance. The provider should have filtered these certificates upon issuance and interrupted the issuance or filtered the erroneously issued certificates and revoked them ASAP, or within not later than 24 hours.

Timeline
Action timestamp Action taken

June 29th 2024 19:46 UTC Bugzilla ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1905509 was created by Wayne
July 1st 2024 09:21 UTC Compliance asked for support from the IT Department and an investigation started, assessing the impact of the indicated configuration changes on certificate issuance and infrastructure
July 2nd 2024 07:15 UTC Following the operational meeting, we implemented a linter configuration refreshment. Testing of the systems continued.
July 3rd 2024 14:59 UTC NETLOCK responded to the reporter
July 4th 2024 07:15 UTC Testing continued and cert configuration changed.
July 5th 2024 12:11 UTC NETLOCK IT confirmed the revocation of the certificates.
.
The revoked certificates are the following:
• tlseveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 09:57:28 UTC
• tlsoveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 12:05:11 UTC
• tlsdveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 11:35:57 UTC
• tlsqeveccca.valid.ecc.netlock.hu - Revoked: July 5th 2024 11:42:06 UTC
• Certificates with the "expired" domain name expired with a validity period of 1 day on June 21st 2024

Root Cause Analysis
The TSP intended to issue new TLS certificates with the ECDSA algorithm. The TSP had not previously issued ECDSA-based certificates, so their configuration took place prior to the issuance of the mentioned certificates. Certificates previously issued with the RSA algorithm were issued in a different system, so the configuration data was manually set by our staff. During configuration, due to human error in several cases, the 'user notice,' 'key usage,' and 'policy' fields were incorrectly configured and went unnoticed. The lint check in the ECDSA system at issuance built into the RSA system also not yet worked due to configuration errors, so the error was not detected before the certificate issuance. During the investigation of the error, we identified the above causes that led to the issuance of erroneous certificates and the lack of discovery and revocation.

Lessons Learned

What went well
Among the issued certificates, those with 'expired' domain names expired within 24 hours of issuance, thus losing their validity within 24 hours. The error contributed to the provider reconfiguring and correctly reissuing the new type of certificates.

What didn't go well
Unfortunately, the system configuration check was insufficient, and as a result, the necessary lint configuration built into the issuance was also insufficient. Provider had to refresh the configuration of the implemented linter. Consequently, the provider could issue certificates with multiple configuration errors.

Where we got lucky
The incorrectly configured certificates were issued during the ECDSA testing phase. Not a high number of erroneous certificates were issued. No customer service was associated with them, and external communication or other activities regarding the revocation were not necessary. The certificates took part in the TSP’s testing phase.

Action Items
TSP reviewed it’s processes and resources. To avoid human error, we have configured automated lint checks in the issuing system, which automatically identify incorrect certificate configurations during the issuance process. On July 15th 2024 and July 30th 2024 we are addressing the human-related issues through more frequent and detailed training on technical configurations.

Comment 14

[Wayne]

This is an incident about a failed 24h response to CPR. If you intend to raise incidents for the rest of the issues please raise them separately.

Please look at any other recent incident as a reference for your incident reports...

Comment 15

[Dorottya]

Dear Wayne, absolutely true, the subject of nr 1905509 ticket does not match the uploaded incident report. We are creating a separate bug ticket for this Certificate Problem Report. Thank you very much. In this case the nr 1906115 ticket would you please close or should we handle it in paralell as well?

Comment 16

[Nikolett]

Attachment: Incident report

Comment 17

[Nikolett]

Incident Report

Summary

Wayne e-mailed us on June 28th 2024 19:45 UTC in accordance to multiple issues (NETLOCK CP/S, certification problems). The e-mail was sent to [e-mail], e-mail was delivered to NETLOCK at the time mentioned before. But the e-mail was not responded within 24 hours.

Impact

The reported issue did not directly affect any certificate, although the email contained a certificate issue, for which the service provider will open a separate ticket. The service provider did not respond to the email notification within 24 hours. The service provider reviewed the emails received in the past quarter addressed to the email address specified by the reporter and determined that there were no other notifications that were not responded to in a timely manner.

Timeline

All times are UTC

Action item	Action taken
2024-06-28 19:45	Wayne e-mailed to NETLOCK and didn’t get answer within 24 hours
2024-06-29 19:45	Wayne opened new ticket for the lack of respond within 24 hours
2024-07-03 14:51	NETLOCK commented the ticket opened by Wayne
2024-07-03 14:59	NETLOCK answered Wayne’s e-mail

Root Cause Analysis

The service provider investigated the issue related to the received email notification and determined that in the past quarter, there had been no instance where inquiries sent to the email address specified in the ticket were not responded to in a timely manner. The email was received; however, due to human oversight, none of the relevant staff responded to it. The lack of response can be attributed to several factors. The relevant staff members reviewed the email but deemed that it did not require a response within 24 hours, as the [e-mail] email address is maintained by the service provider for CP/S-related questions and inquiries. Consequently, the necessary work to respond to the email did not begin until July 1st, 2024.

Lessons Learned

What went well

The issue does not affect the provider’s processes; no new processes need to be established nor any faulty processes corrected. The provider has the appropriate staff and expertise to address the issue, which is an isolated incident.

What didn’t go well

The provider's staff was not sufficiently attentive to the received email and did not escalate the issue because they did not recognize that the response time was limited to just 24 hours.

Where we got lucky

The provider reviewed recent inquiries sent to the email address and found no other requests that were not responded to in a timely manner. The provider has an adequate process for handling such inquiries.

Action Items

On July 8th, 2024, the provider trained the relevant staff on the response time requirements for inquiries received at the specified email address. The provider emphasized to staff the process for escalating inquiries related to faulty certificates to the supervisor and ensuring responses to the reporter are not delayed.

Comment 18

[Ben Wilson]

Netlock needs to address and/or explain what additional action items are being taken to improve response times when processing Certificate Problem Reports. The current Incident Report is insufficient in this regard.

Comment 19

[Nikolett]

Hello everyone,

Thank you for the latest comment, we will explain the additional action items.