1962426 - NETLOCK: CA/Browser Forum TLS BR Non-compliance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Nikolett]

Preliminary Incident Report

Summary

• Incident description:
  A TLS certificate (https://crt.sh/?q=48e0eb3d7647f1ba001baf9751edea7e12daad575d79a0b506ad7a1f0209ea33) that is currently expired is being served on the domain https://valid.ev.tanusitvany.hu/.
  This domain is designated in the Common CA Database (CCADB) as the "Test Website - Valid" associated with the NetLock Arany (Class Gold) Root Certificate (CN=NetLock Arany (Class Gold) Főtanúsítvány; OU=Tanúsítványkiadók (Certification Services); O=NetLock Kft.; C=HU; L=Budapest).
  According to Section 2.2 of the CA/Browser Forum TLS Baseline Requirements, the "Test Website - Valid" must host a valid, unexpired, and unrevoked certificate at all times.

• Relevant policies:

  • CA/Browser Forum TLS Baseline Requirements v2.0.2
    • Section 2.2 - Publication of Information:

      "The CA SHALL host test Web pages that allow Application Software Suppliers to test their software with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber Certificates that are:

      • valid,
      • revoked, and
      • expired."

  • Mozilla Root Store Policy v3.0 (Effective March 15, 2025)
    • Section 2.4 - Incidents:

      "When a CA operator fails to comply with any requirement of this policy [...] the event is classified as an incident. CA operators MUST adhere to the CCADB's Incident Reporting Guidelines (IRGs)."

• Source of incident disclosure:
  Report submitted via email by community member, identifying the mismatch between the certificate status and its intended role in CCADB metadata.

Comment 1

[Zacharias]

The link to https://crt.sh/?q=48e0eb3d7647f1ba001baf9751edea7e12daad575d79a0b506ad7a1f0209ea33 has wrong url in the markdown and took me to bugzilla.com/url.
The URL in this comment is correct.

Comment 2

[Zacharias]

Apologizes for the double post. I wanted to check the cert to see when it expired and got distracted by the broken link. Then I remembered to check the expiration date which was almost 3 weeks ago!

How could this stay unnoticed by NETLOCK for so long?

Comment 3

[Nikolett]

Incident Report

Summary

• CA Owner CCADB unique ID: A000039
• Incident description:
  On 2025-04-21 20:30 UTC, a community member reported that the domain https://valid.ev.tanusitvany.hu/, which is designated in the Common CA Database (CCADB) as NETLOCK's "Test Website – Valid", was serving an expired TLS certificate. The affected certificate chained to the NetLock Arany (Class Gold) root certificate.
  This is a violation of Section 2.2 of the CA/Browser Forum Baseline Requirements, which requires that the designated “Valid” test website always present a valid, unexpired, and unrevoked certificate.
• Timeline summary:
  • Non-compliance start date: 2025-04-05
  • Non-compliance identified date: 2025-04-21
  • Non-compliance end date: 2025-04-28
• Relevant policies:
  • CA/Browser Forum TLS Baseline Requirements v2.0.2, Section 2.2
  • Mozilla Root Store Policy v3.0, Section 2.4
• Source of incident disclosure: Third Party Report (community member)

Impact

• Total number of certificates: 1
• Total number of "remaining valid" certificates: 0
• Affected certificate types: EV TLS
• Incident heuristic: Single certificate (SHA256: 48e0eb3d7647f1ba001baf9751edea7e12daad575d79a0b506ad7a1f0209ea33)
• Was issuance stopped in response to this incident, and why or why not?:
  No. This incident did not affect issuance systems or subscriber certificates.
• Analysis: N/A
• Additional considerations: The issue occurred during the transition to a newly implemented automated certificate issuance workflow.

Timeline

Date & Time (UTC)	Event
2025-04-05 07:46	Certificate for https://valid.ev.tanusitvany.hu/ expired
2025-04-21 20:30	Email received from community; NETLOCK became aware of the issue
2025-04-22 07:06	We replied to the community report
2025-04-24	We opened a preliminary incident report
2025-04-28 14:47	New certificate issued
2025-04-28 16:12	New certificate deployed to test website
2025-04-30	Full incident report

Related Incidents

N/A

Root Cause Analysis

Contributing Factor #1: The monitoring system was under configuration

• Description: No automated alerting or expiration monitoring system was in place for the test endpoint while the configuration was ongoing.
• Timeline: The certificate expired on 2025-04-05.
• Detection: External report via email; internal systems did not detect the issue.
• Interaction with other factors: The final tests of the adaptation of the meta linter were under.
• Root Cause Analysis methodology used: 5 Whys
  Why was the expired certificate not detected?
  Because the expiration monitoring system for the test endpoint did not generate any alerts.
  Why did the monitoring system not generate alerts?
  Because the relevant monitoring configuration was not yet active for test endpoints.
  Why was the configuration not yet active?
  Because the monitoring environment was undergoing integration testing with a new component (meta linter adaptation).
  Why was the integration testing still ongoing during certificate expiration?
  Because the rollout plan for the enhanced test infrastructure monitoring was scheduled for completion after the certificate’s expiration date.
  Why was the test site included in the scope before monitoring was finalized?
  Because the monitoring activation and test endpoint lifecycle processes were not yet fully synchronized during the transition phase to automated workflows.

Lessons Learned

• What went well:
  • Prompt action when community report came
  • Timely response and incident filing within Mozilla policy window
• What didn’t go well:
  • The configuration and testipng process was not implemented as quickly as we planned.
• Where we got lucky:
  • Only the test endpoint was affected; no impact to subscriber services or issued certificates
• Additional:
  • We need to take more time to implement the process and plan properly

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Complete deployment of test cert monitoring	Prevent	Monitoring was under configuration	Alerts working on all test certs	2025-04-28	Completed
New EV TLS certificate issued	Mitigate	Expired test certificate	https://crt.sh/?id=18109788462	2025-04-28	Completed
Assigned formal operational responsibility for test certificate renewal and maintenance.	Mitigate	No regular test validation	Quarterly checklist made	2025-04-29	Completed
Monitoring system configuration testing in progress	Mitigate	No regular test validation	N/A	2025-05-07	Ongoing

Appendix

Affected Certificate (Expired)

• SHA-256: 48e0eb3d7647f1ba001baf9751edea7e12daad575d79a0b506ad7a1f0209ea33
• Subject: CN=valid.ev.tanusitvany.hu
• Issuer: CN=NetLock Arany (Class Gold) Főtanúsítvány
• Not Before: 2024-04-05 07:46 UTC
• Not After: 2025-04-05 07:46 UTC
• Is revoked? N/A (expired)
• Revocation date: N/A
• Revocation reason: N/A

Replacement Certificate (Valid) - https://crt.sh/?id=18109788462

• SHA-256: b693f188620cbbc334b7b3681d7c174c034ceea8a73b720d93a5158e3806b189
• Issued: 2025-04-28 14:47 UTC
• Installed: 2025-04-28 16:12 UTC
• Subject: CN=valid.ev.tanusitvany.hu
• Issuer: CN=NetLock Arany (Class Gold) Főtanúsítvány

Response to (Comment 2)

We acknowledge that the expired certificate on our publicly listed “Valid” test endpoint went unnoticed for an unacceptable period. The incident occurred during our transition to a new automated certificate issuance workflow, which we were implementing to improve the overall reliability and efficiency of our infrastructure. During this migration, the manual renewal of the test certificate was inadvertently overlooked, and automated monitoring for this endpoint was not yet active.
We take full responsibility for this oversight.
While the test endpoint is not used by subscribers, it is an important part of our public compliance.
To prevent recurrence, we have taken the following steps:

• A new valid certificate was issued and deployed to the test endpoint on 2025-04-28.
• Automated issuance is now fully active for this test certificate.
• We have assigned formal operational responsibility for test certificate renewal and maintenance.
• We are implementing alerting systems to ensure proactive visibility into test infrastructure.

Comment 4

[walter.j.marks]

Netlock:

Are you aware that as of writing this message, valid.ev.tanusitvany.hu is still serving 48e0eb3d7647f1ba001baf9751edea7e12daad575d79a0b506ad7a1f0209ea33 when accessed via IPv6?

If this is a surprise to you, questions follow the quote:

We have assigned formal operational responsibility for test certificate renewal and maintenance.
We are implementing alerting systems to ensure proactive visibility into test infrastructure.

What operational responsibility / oversight is responsible for the certificate?
What alerting systems are involved in this test infrastructure, and were any alarms sounded by this system?
Was the proper certificate being served validated across both IPv6 and IPv4 prior to Comment 3?

Comment 5

[walter.j.marks]

As a follow-up to my above questions:

The Via header on your test site seems to point to a third party based on the certificates used by that Via. Is Netlock responsible for the entire test site (i.e. the entire server stack), or is it in collaboration with a third party?

Comment 6

[Aaron Gable]

Hi Netlock,

In my opinion, the Timeline in this incident report is incomplete. The CCADB Incident Reporting Guidelines say that the timeline is expected to include "All policy, process, and software changes that contributed to the Root Cause(s)". Your root cause analysis says that this incident occurred due to an in-progress migration, but the timeline does not address this migration at all. Please answer the following questions by adding the relevant entries to the Timeline:

• When did the infrastructure migration begin?
• When was monitoring for the test site removed?
• When did the testing of the new infrastructure begin?

Comment 7

[walter.j.marks]

(In reply to walter.j.marks from comment #4)

Netlock:

Are you aware that as of writing this message, valid.ev.tanusitvany.hu is still serving 48e0eb3d7647f1ba001baf9751edea7e12daad575d79a0b506ad7a1f0209ea33 when accessed via IPv6?

If this is a surprise to you, questions follow the quote:

We have assigned formal operational responsibility for test certificate renewal and maintenance.
We are implementing alerting systems to ensure proactive visibility into test infrastructure.

What operational responsibility / oversight is responsible for the certificate?
What alerting systems are involved in this test infrastructure, and were any alarms sounded by this system?
Was the proper certificate being served validated across both IPv6 and IPv4 prior to Comment 3?

While still unanswered, it seems that at some point between now and my previous post, the AAAA record resulting in 48e0eb3d7647f1ba001baf9751edea7e12daad575d79a0b506ad7a1f0209ea33 being served has been removed.

Please be sure to include the details of this in the timeline.

Comment 8

[Nikolett]

Dear Community members,

We will update the ticket and respond to all questions asked until this friday (05-09-25).

Comment 9

[Nikolett]

Dear Community members,
We acknowledge the community questions and the need to update the incident timeline.

We are currently preparing detailed responses and a revised timeline that reflects the latest findings.
Remediation activities are already in progress, and we will include a summary of the corrective measures being implemented in our next update.

We will provide this information as part of our next scheduled update, which will be posted within the next 7 days, accordingly, we kindly request that the Next update Whiteboard field be set to 2025-05-16.

Thank you for your patience and continued engagement.

Comment 10

[Nikolett]

Dear Community members,

As for comment 5 : NETLOCK is responsible for the entire stack, there is no collaboration with a third party.

For comment 6: The implementation process started on the 15th March, the monitoring of the test site was removed on the 1st of April. We finished the phase2 implementation of the linter on 17 April. Following this date we had a testing phase, when the monitoring was turned on again on the 22th of April.

For comment 7: Yes, it was repaired in the meantime.

Comment 11

[Nikolett]

would like to clarify comment 10: NETLOCK currently uses IPv4 for its services, for which we as the one-stop shop are responsible. For the test certificate and for our services, we used IPv6 prior to the system change, during which we used a 3rd party. We have replaced IPv6 serving with IPv4 serving for the simple reason that we can automate serving over IPv4.

Comment 12

[walter.j.marks]

Netlock:

Please define "meantime". (i.e. there should be a specific date and time known when the AAAA record was removed)

Q1: If monitoring was re-enabled on the 22nd of April, why was the v6 invalid cert not discovered until at least comment 4's timestamp (April 30)?

Q1a: Was your monitoring not configured to check IPv6, or was it sending warnings/alarms that were not actioned upon?

Comments 10 and 11 contradict each other with regard to your test site. Comment 10 says "we are responsible entirely", Comment 11 says "we used a third party prior to the system change". Just as some minor additional color: while I don't know the version of Apache used by your reverse proxy provider at the time of comment 4, I do know that the version of Varnish being used was 7.3. Varnish 7.3 was released in March 2023, with the last update to the 7.3 branch coming in 2024. While this isn't strictly speaking relevant, as a CA / security provider, I'd hope that you are keeping software up to date, which would include any third parties you use to run your testing infrastructure.

Q2: At the time I posted the relevant comment 4: was Netlock responsible for the whole test site stack, yes or no?

Q2a: Who was the responsible party for creating and removing the A/AAAA records for that domain? Was it Netlock, or the third party?

Q3: Was the fact that the invalid cert was still being served over v6 when I posted Comment 4 a surprise, yes or no?

Comment 13

[Nikolett]

Q1: If monitoring was re-enabled on the 22nd of April, why was the v6 invalid cert not discovered until at least comment 4's timestamp (April 30)?

The certificate expired on 05th April. Monitoring was re-enabled on 22th April, so no monitoring message was received for the certificate expiration. A new certificate was issued on 28th April and implemented in the system. Since the monitoring signals only before and upon expiration, the certificate issued on 28th April was valid, so no monitoring message was received.

Q1a: Was the monitoring system not configured to check for IPv6 or was it sending warnings/alerts that were not responded to?

The monitoring system was configured for certificate expiration.

Comments 10 and 11 contradict each other regarding your test location. Comment 10 says "we are entirely responsible", comment 11 says "we used a third party before the system change". Just some small additional information: although I don't know what version of Apache your reverse proxy provider was using at the time of comment 4, I do know that the version of Varnish used was 7.3. Varnish 7.3 was released in March 2023, the last update to the 7.3 branch was in 2024. While this is not strictly speaking relevant, as a CA/security provider I would hope that you would keep the software up to date, which includes the third parties used to run the testing infrastructure.
Q2: At the time I posted the relevant comment 4: was Netlock responsible for the whole test site stack, yes or no?
Q2a: Who was the responsible party for creating and removing the A/AAAA records for that domain? Was it Netlock, or the third party?
Q3: Was the fact that the invalid cert was still being served over v6 when I posted Comment 4 a surprise, yes or no?

Yes, apologies for the misunderstanding, we created comment 11 to complement comment 10.
In our opinion, it is not relevant information whether the test site is operated by a third party, because the test site belongs to NETLOCK and NETLOCK is responsible for it.

Comment 14

[Zacharias]

• (In reply to Nikolett from comment #13)

The certificate expired on 05th April. Monitoring was re-enabled on 22th April, so no monitoring message was received for the certificate expiration. A new certificate was issued on 28th April and implemented in the system. Since the monitoring signals only before and upon expiration, the certificate issued on 28th April was valid, so no monitoring message was received.

I’m not sure I understand the action items correctly, but are you or are you going to have monitoring that signals also after expiration? As I understand it the expiry went unnoticed because when the new monitoring was enabled the certificate was already expired. So I would expect there to be an action item to ensure this does not happen again in for example a situation of a national power outage like affected Spain and certificate expiration coinciding

Comment 15

[Nikolett]

Updated Timeline

Time (UTC)	Action Item
2025-03-15	Implementation process started for MetalLinter
2025-04-01	Test site monitoring removed
2025-04-05 07:46	Certificate for https://valid.ev.tanusitvany.hu/ expired
2025-04-17	Finished the phase2 implementation of the linter
2025-04-21 20:30	Email received from community; NETLOCK became aware of the issue
2025-04-22 07:06	We replied to the community report
2025-04-22 13:15	Monitoring was back in place
2025-04-24	We opened a preliminary incident report
2025-04-28 14:47	New certificate issued
2025-04-28 16:12	New certificate deployed to test website
2025-04-30	Full incident report

Updated Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Complete deployment of test cert monitoring	Prevent	Monitoring was under configuration	Alerts working on all test certs	2025-04-28	Completed
New EV TLS certificate issued	Mitigate	Expired test certificate	https://crt.sh/?id=18109788462	2025-04-28	Completed
Assigned formal operational responsibility for test certificate renewal and maintenance.	Mitigate	No regular test validation	Quarterly checklist made	2025-04-29	Completed
Monitoring system configuration testing in progress	Mitigate	No regular test validation	N/A	2025-05-07	Completed

Comment 16

[walter.j.marks]

In our opinion, it is not relevant information whether the test site is operated by a third party, because the test site belongs to NETLOCK and NETLOCK is responsible for it.

I would counter that it is relevant information. Per 2.2 of BRs

The CA SHALL host test Web pages that allow Application Software Suppliers to test their software with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber Certificates that are
valid,
revoked, and
expired.

Without getting into the weeds of whether or not the BRs require the CA to be responsible for the entirety of the test web page (which likely belongs on MDSP about the vagueness), there was a test page that should have been valid, and was not.

Additionally, your action items do not resolve the problem here, as your monitoring per your own admission only monitors for certificate expiry, not that the test sites are serving the certificates that are expected.

https://www.ccadb.org/cas/incident-report

Action Items: The Action Items section contains a list of remediation items that will be undertaken to ensure that similar incidents do not reoccur in the future. For example, if the Whiteboard field contains ‘revocation-delay’, the Action Items list MUST include steps reasonably calculated to prevent or reduce future revocation delays. Note that it is not sufficient for these action items to simply stop this incident, they MUST create additional protections to prevent future incidents.

Given that the issue here occurred via a failure to have ongoing monitoring of your test sites, a logical action item would be ongoing monitoring of said test site, not re-enabling the monitoring that did not detect the problem.

Comment 17

[Nikolett]

In Response to Comment 16
Thank you for your comment as well as the expectations regarding the ongoing monitoring of our test websites.
We acknowledge that we are responsible for test pages. NETLOCK removed the wrong records for the domain.
We also accept your point that our previous action items, do not sufficiently address the absence of end-to-end functional validation of the test website.

To properly address the issue and fulfill the expectations we have revised our remediation steps as follows:

Updated Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Implement active test site monitoring that verifies availability and that the expected certificate is served (valid, revoked, expired respectively)	Prevent	Inadequate test site validation	Monitoring alerts triggered within 24h of a discrepancy	2025-06-14	Ongoing
Conduct monthly manual verification of test page content and certificate consistency	Detect	Missing process control	Checklist logs reviewed monthly by compliance team	2025-06-30 (recurring)	Ongoing

These items aim to remediate this specific issue and ensure that similar failures do not occur in the future, in line with the frameworks.
We will report on the implementation of the tasks indicated in the action items in time

Comment 18

[Nikolett]

Dear community members,
do you have any further questions in the meantime?
The implementation is going well.

Comment 19

[Nikolett]

Dear Community members,
We have succesfully implemented the test site monitoring, here are the updated action items. We will follow up shortly.

Updated Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Implement active test site monitoring that verifies availability and that the expected certificate is served (valid, revoked, expired respectively)	Prevent	Inadequate test site validation	Monitoring alerts triggered within 24h of a discrepancy	2025-06-14	Completed on 2025-06-13
Conduct monthly manual verification of test page content and certificate consistency	Detect	Missing process control	Checklist logs reviewed monthly by compliance team	2025-06-30 (recurring)	Ongoing

Comment 20

[Roland]

Dear Community Members,
The remaining task: monthly manual checks of test page content and consistency of certificates, proceeding according to the terms and conditions agreed. We will follow up shortly.

Comment 21

[Roland]

Dear Community Members,
we are on schedule with the implementation of the task, we will provide feedback next week on the tasks completed.

Comment 22

[Roland]

Incident Closure Summary — Bug 1962426

Incident description

On 2025-04-21 20:30 UTC, a community member reported that the domain https://valid.ev.tanusitvany.hu/, designated in the CCADB as NETLOCK’s “Test Website – Valid,” was serving an expired EV TLS certificate. The expired certificate chained to the NetLock Arany (Class Gold) root. A designated “Valid” test website must present a valid, unexpired, and unrevoked certificate. The expired certificate was replaced and service was restored by 2025-04-28.

Incident Root Cause(s)

The expired certificate was not detected because the automated alerting and monitoring system for the test endpoint was under configuration at the time of expiration. The migration to a new automated issuance workflow was in progress, and manual oversight during this transition was inadequate. Specifically, the rollout of enhanced monitoring was scheduled to complete after the expiration date, and internal processes for synchronizing endpoint lifecycle management with monitoring activation were not yet mature.

Remediation description

We have remediated the issue by:

• Issuing and deploying a new valid certificate to the test endpoint on 2025-04-28.

• Completing the deployment of automated issuance and monitoring for the test certificate.

• Assigning clear operational responsibility for test certificate lifecycle management, with a quarterly checklist.

• Implementing alerting systems to proactively detect certificate expirations and endpoint status.

We also completed the following preventive and detective action items:

• Implemented active monitoring to verify test site availability and certificate validity, with alerts triggered within 24 hours.

• Established monthly manual verification of test page content and certificate consistency, logged and reviewed by the compliance team.

All immediate and ongoing action items listed in the full incident report have been completed.

Commitment summary

Beyond the specific action items, we reaffirm our commitment to:

• Continuous improvement of monitoring and lifecycle management for all test and production certificates.

• Regular reviews of our operational procedures to prevent recurrence.

• Transparent, timely reporting of any future incidents in compliance with CCADB and Mozilla Root Store Policy requirements.

We respectfully request closure of this incident report, as all action items have been completed, and no further questions or comments are outstanding.

On behalf of NETLOCK, we thank the Mozilla community for its vigilance and constructive feedback throughout this process.

Comment 23

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-14.