NETLOCK: Bug 1891331 replacement - delayed revocation -
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: nagy.nikolett, Assigned: nagy.nikolett)
References
(Blocks 1 open bug)
Details
(Whiteboard: [ca-compliance] [leaf-revocation-delay])
We opened a new ticket replacing bug 1891331 summarizing and aswering all the questions of the community. Here you can see the original ticket, the answers can be found in the Appendix.
Summary
Additional IR over https://bugzilla.mozilla.org/show_bug.cgi?id=1889570
NETLOCK was notified on 3 April 2024 that one of our TLS certificate (https://crt.sh/?id=11261465680), signed by our "NETLOCK Trust EV CA 3" (https://crt.sh/?id=11261465680 ) intermediate certificate gives an error on the zlint check (https://crt.sh/?id=11261465680&opt=zlint).
The investigation of the certificate above showed that the TLS certificates issued was created with User Notice field, with
Explicit Text: EV SSL certificate. Terms of service at: http://www.netlock.hu/html/dok.html value which is against BRG 2.0 since Sept. 15, 2023.
Affected intermediate issuers, from where the misissued end-user certificates were signed
• NETLOCK Trust EV CA 3
• NETLOCK Trust Qualified EV CA 3
• NETLOCK DVSSL CA
After the initial investigation NETLOCK started the customer communication and planning the revocation of the affected certificates, as described in the Timeline section of the IR.
Impact
after contacting some of our customers of major importance for the national economy regarding the misissuance of the certificates they requested the extension of the revocation deadline because of their internal procedures they are not able to change some of their certificates in their systems in such a short timeframe
Timeline
All times are CEST
Action timestamp | Action taken |
---|---|
2024-04-03 10:52 | E-mail arrived to our central address (compliance@netlock.hu) |
2024-04-05 10:00 | direct and undirect customer communication begung on the error |
2024-04-05 12:30 | Customer responses arriving regarding the new certificates – either accepting the new cert and the expected change date, or objections |
2024-04-06 10:00 | Revocations started |
2024-04-09 13:21 | All misissued certificates not requested to be revoked later by the customers (https://bugzilla.mozilla.org/show_bug.cgi?id=1889570) were revoked |
2024-04-30 23:59 | Certificates requested to be revoked later will be revoked by 2024-04-09 13:21 |
Root Cause Analysis
The issue originated from the change of the BRG last year. NETLOCK’s staff investigated the changes in the profiles, but unfortunately the change regarding the policy identifier was not updated.
Lessons Learned (Updated)
This issue pointed out that our internal systems monitoring should be improved to catch such issues more affectively. NETLOCK has worked hard in the past period on monitoring external and internal data regarding certificate issuance.
1 What went well
The changes in our internal processes let us handle this issue on a timely manner and we were able to meet the deadlines required by the BRG and Mozilla.
2 What didn't go well
Some of our customers used the related certificates in such systems, where the changing of the certificates cannot be implemented in such a short period of time.
3 Where we got lucky
The security of the certificates was not affected by the misissuance. Therefore NETLOCK decided to extend the revocation period for customers of major importance for the national economy.
Action Items
Action Item | Kind | Due Date |
---|---|---|
Fix of certification profiles | Repair | 2024-04-09 |
Revoke affected certificates | Repair | 2024-04-09 |
Revoke remaining affected certificates (based on customer requests) | Repair | 2024-04-30 |
Extend monitoring capabilities (linting functionality and post checks) | Remediate | 2024-05-31 |
Appendix
in the bug 1891331, many questions have been raised by the community and we would like to answer them below:
1 Comment 16
Can you clarify where in Netlock's Certificate Policy there is a requirement to revoke within five days?
A previous comment stated: „We strive to steer clear of errors to avoid the need for withdrawals within 5 days.”
We apologise, we made a typo here. The certificates have been revoked within 1 day. As you can see later in the comment, we have given ourselves the 5 day deadline to deliver the new certificates to the users, The 5 days in the first sentence did not refer to the withdrawal it was a mistake on our part. We understood the 24h mark for the revocation time then.
Who is actually responsible for handling revocation?
The Subscriber shall be responsible for: initiating the modification of the certificate, the replacement or revocation of his key in accordance with Sections 9.6.3 and 4.9.1 of the Service Policy and Sections 4.7 and 4.8 hereof;
This inserted part of our policy refers to the case when a user requests the revocation of their certificate.
This question also pointed out that in our Policies and Statements the responsibility of certificate revocation is not clear enough. Our regulatory documents only cover how we will deal with a customer's withdrawal request. In the case of certificate revocation due to a certificate configuration error/misissuance, the provisions are not clear despite the reference of section 4.9.1 of the TLS Baseline Requirements.
As a solution to this problem, it will be included in our Terms and Conditions that if TLS certificates are issued with incorrect content, the Trust Service Provider will be responsible for revoking them in accordance with the BRG Policy, this will be done within 24 hours without exception.
We would like to point out that since this comment was added, our regulatory documents in the CCADB have been corrected. We will continue to pay particular attention to the maintenance of the above documents in the future.
2 Comment 18
Netlock's Certificate Policy does not have 5-day revocation available. Section 4.9.1 only covers 24 hours, and 7 days. Due to this your requirement for revocation was 24 hours.
Yes, the policy does not include the 5 days because that was a typo on our part, apologies. We will take care of this in the future.
Netlock's General Terms and Conditions mention on page 24 that the Subscriber is responsible for revocation. Is this correct?
It is correct in the case of data change. The Subscribers must promptly notify us of any changes to their data that is in a certificate and request the suspension or revocation of their certificates. In case of misissuance the revocation is not the Subscibers responsibility.
Netlock's CCADB entry's for their certificate policy are out of date by years.
We updated the documents and will maintain them in the future as required.
CCADB and Chrome Root Program's have separate policies mandating an authoritative English language version. This needs to be addressed.
It has been adressed. We revised our Policies Ands Statements and get them translated again with help of professional translators. This is still an ongoing process.
I will note Section 2.1.2 as Netlock have created a big problem:
The Service Provider shall, at least 30 days before their entry into force, publish the new versions to be implemented of the Certificate Policy, the Practice Statement and the General Terms and Conditions pertaining to the services based on the present Certificate Policy and affected by the subject modification.
Certificate Policies are effectively contracts attached to certificates by reference, by including this language they are technically non-compliant until 30 days after the Certificate Policy has been published.
We agree with the above. We would like to apologise and clarify the situation:
Yes, our service policies and statemens have a 30-day waiting period. Service Providers here are required to publish the new policy on their website 30 days before this new policy comes into force. This 30 days is the period in which the current regulation document is in force. We see the problem with the above, and as a result of discussions with the supervisory authority, we have come to the conclusion that the 30-day period will continue to apply, but if we detect a discrepancy in the certificate profile or a problem that cannot wait for the 30-day lead time, we have the possibility to immediately amend the new regulatory document and make it available to the supervisory authority for registration and publication on our website and on the CCADB interface. Subject to these, it is possible to deviate from the general procedure, but prior consultation with the authorities is requested and needed. Following the consultation, the CP/S can be immediately modified.
3 Comment21
Hi, can you please provide some references of EU laws associated with Trust Services that support this statement?
Regulation (EU) No 910/2014 (eIDAS) on electronic identification and trust services for electronic transactions in the internal market, provides a regulatory environment for electronic identification of natural and legal persons and for trust services states in Article 24. 2. (a)
2. A qualified trust service provider providing qualified trust services shall:
(a) inform the supervisory body of any change in the provision of its qualified trust services and an intention to cease those activities;
This EU regulation is harmonised into the Hungarian legislation by the DÁP Act, which includes:
XV. FEJEZET A BIZALMI SZOLGÁLTATÁSOK NYÚJTÁSÁNAK ÁLTALÁNOS FELTÉTELEI 43. A bizalmi szolgáltatás nyújtásának megkezdése
83. § (6) A minősített bizalmi szolgáltatást nyújtó bizalmi szolgáltató a változás bevezetését legalább 30 nappal megelőzően értesíti a bizalmi felügyeletet a bizalmi szolgáltatás nyújtásában bekövetkező, tervezett változásokról.
Translation:
CHAPTER XV. GENERAL CONDITIONS FOR THE PROVISION OF TRUST SERVICES
43. Section 83§ (6) A trust service provider providing a qualified trust service shall notify the Supervisory Authority of the planned changes in the provision of the trust service at least 30 days before the change is implemented.
Sidenote: You may see Eüt in our previous ticket: The Eüt was repealed by the DÁP legislation over 2024 autumn.
Changes to our Service Policies and Statements fall within this scope. This "30-day rule" is only applicable for TSPs operated in Hungary because of the more specific rules described in "DÁP."
The changes we have made and the solutions we have came up with are the following:
- We have amended our Terms and Conditions, emphasising the 24-hour withdrawal period for TLS certificates. In case of misissuance the certificates will be revoked within 24 hours, no exception.
- Our policies have been revised with the help of a professional translator and the part that English translations do not prevail has been removed.
- NETLOCk has completed the full implementation of Zlinter prior to Certificate Release and we are also listed on GITHUB. - https://github.com/zmap/zlint
- Our regulatory documents have been updated in the CCADB and we are working on keeping them up-to-date.
Our deadlines for the above are as follows, we would like to include an updated Action Items table as the changes are currently in progress.
Action Item | Kind | Due Date |
---|---|---|
Zlint implementation | Prevent | Completed on 2024.10.04 |
Monitoring implementation on Certification issuance | Prevent | Completed on 2024-05-31 |
Monitoring phase 2 implementation on Checking after issuance | Prevent | Completed on 2024-07-31 |
Terms and condition update with the 24H revocation deadline | Remediate | 2025.03.10 |
Publication of the newly translated Policies | Remediate | 2025.03.10. |
CCADB documentation update | Remediate | 2025.03.10 |
Updated•8 months ago
|
There's an expectation of weekly status updates to all non-closed incidents.
There have not been recent updates to Bug #1904041, Bug# 1905509, Bug #1947691, Bug #1917046
In similar cases, the lack of updates is itself cause to file a separate incident report.
Updated•6 months ago
|
Comment 3•6 months ago
|
||
Bug #1957474 has been opened.
Hi Ben,
we commented on the ticket you marked Bug #1957474.
Dear Community members,
Our updated action items are as follows:
Our documents are under full review due to a change of seat of the company that is why we modified the committed deadline.
Action Item | Kind | Due Date |
---|---|---|
Zlint implementation | Prevent | Completed |
Monitoring implementation on Certification issuance | Prevent | Completed |
Monitoring phase 2 implementation on Checking after issuance | Prevent | Completed |
Terms and condition update with the 24H revocation deadline | Remediate | 2025.04.30. |
Publication of the newly translated Policies | Remediate | 2025.04.30. |
CCADB documentation update | Remediate | 2025.04.30. |
Dear Community Members,
we would like to inform you that we are continuing with the tasks, the completion date is still 30.04.2025.
Dear Community Members,
Please be informed, that our Terms and conditions have been updated with the 24H revocation and we are taking the necessary steps to put them into effect. The text will then be published.
Dear Community members,
Our next update date is 05-14-25.
Maybe I’m misunderstanding, but aren’t there actions items that should be completed by now. Other than the 24h revocation one I mean.
Comment 10•5 months ago
•
|
||
Because this is a delayed revocation incident, there is heightened scrutiny of your CA and its responses. Please note that according the CCADB Incident Reporting Guidelines (https://www.ccadb.org/cas/incident-report), reports "SHOULD be updated every 3 days and MUST be updated no less frequently than every 7 days to describe:
- the number of certificates that have been revoked;
- the number of certificates that have not yet been revoked;
- the number of certificates planned for revocation that have expired; and
- an estimate for when all remaining revocations will be completed."
The "Timeline" section must include the "time(s) at which the CA Owner is expected to complete revocation of affected certificates" and the "time(s) at which the CA Owner actually completed revocation of affected certificates".
The "Analysis" section must describe "the factors and rationales behind the decision to delay revocation (including detailed and substantiated explanations of how extensive harm would result to third parties–such as essential public services or widely relied-upon systems–and why the situation is exceptionally rare and unavoidable)."
"The Action Items list MUST include steps reasonably calculated to prevent or reduce future revocation delays. Note that it is not sufficient for these action items to simply stop this incident, they MUST create additional protections to prevent future incidents."
Assignee | ||
Comment 11•4 months ago
|
||
Dear Community Members,
the number of revoked certificates is 522, there are no certificates that have not already been revoked, no certificates that have not been revoked due to expiry. Given that there are no certificates that have not been revoked, there will be no further revocations.
As reported in Bug 1891331, revokation started on 2024-04-06 at 10:00 and ended on 2024-04-09 at 13:21. Certificates where clients requested later revocation were revoked at 2024-04-30 23:59.
As indicated in the Bug 1891331, for customers of high economic importance, the withdrawal was delayed to avoid the loss of their service causing further problems of national economic importance. We are aware that we have acted inappropriately by delaying withdrawals and have taken measures to ensure that this situation does not occur again.
We have implemented and are using Metalinter, which is recommended by community members, which we have configured and hope that it will prevent any certificates from being issued that suffer from any errors.
We will modify the Timeline, Analysis and Action Items sections as you request.
Assignee | ||
Comment 12•4 months ago
|
||
Impact
-
Total number of certificates: 522
-
Total number of "remaining valid" certificates: 0
-
Affected certificate types: OV TLS, EV TLS
-
Incident heuristic: 3 April 2024: NETLOCK was notified, that our TLS certificates issued were created with the User Notice field, with Explicit Text: EV SSL certificate. Terms of service at: http://www.netlock.hu/html/dok.html value - which is against BRG 2.0 since Sept. 15, 2023.
-
Was issuance stopped in response to this incident, and why or why not?: Yes, the issuance was stopped and we corrected the configuration.
-
Analysis:
NETLOCK delayed the revocation of some certificates because they belonged to customers of major importance for the national economy.
Customers of major importance for the national economy include: Government offices, Banks, insurance companies, Telecommunication service providers, National postal service provider. These providers must announce their system operations in advance and obtain approval from supervisory authorities. The national postal service provider also operates critical national infrastructure, the potential disruption of which could cause significant damage to the country's supply. -
Additional considerations:
We would like to emphasize that we revoked all other certificates and our company generated new certificates required for certificate renewals for our clients and delivered them.
Updated Action Items
Action Item | Kind | Due Date | Status |
---|---|---|---|
Fix of certification profiles | Repair | 2024.04.09 | Completed |
Revocation | Remediate | Started 2024.04.06. | Completed on 2024.04.09 |
Revocation of certificates of customers of high economic importance | Remediate | Completed | 2024.04.30. |
Zlint implementation | Prevent | Completed on 2024.10.04 | Completed |
Monitoring implementation on Certification issuance | Prevent | Completed on 2024-05-31 | Completed |
Monitoring phase 2 implementation on Checking after issuance | Prevent | Completed on 2024-07-31 | Completed |
Terms and condition update with the 24H revocation deadline | Remediate | 2025.03.10 | Completed |
Publication of the newly translated Policies | Remediate | 2025.03.10 | Due on - 2025-05-26 |
CCADB documentation update | Remediate | 2025.03.10 | Due on - 2025-05-26 |
Timeline
Item | Date and time |
---|---|
Zlint implementation | 2024.10.04 |
Monitoring implementation on Certification issuance | 2024-05-31 |
Monitoring phase 2 implementation on Checking after issuance | 2024.07.31 |
Terms and condition update with the 24H revocation deadline | 2025.03.10 |
Publication of the newly translated Policies | 2025.03.10 |
CCADB documentation update | 2025.03.10 |
Terms and condition update with the 24H revocation deadline | 2025.03.10 |
Publication of the newly translated Policies | 2025.05.26 - The necessary negotiations have taken place, the approval of the supervisory authority has been received and the documents will be available on Monday, according to comment 7. |
CCADB documentation update | 2025.05.26 - The necessary negotiations have taken place, the approval of the supervisory authority has been received and the documents will be available on Monday, according to comment 7. |
In light of the above, we will update the ticket on Monday 26.05.2025.
Comment 13•4 months ago
|
||
I will note and emphasize here that the broad justification currently offered — that revocation was delayed due to the certificate holders being “of major importance for the national economy” — is not acceptable. These types of broad arguments have been made over the past two years and have been consistently rejected by the community. Please acknowledge that the subscriber’s sector, size, and operational impact do not matter.
Neither the Baseline Requirements, nor the Mozilla Root Store Policy, nor the CCADB Incident Reporting Guidelines recognize or allow delayed revocation exceptions for government agencies, critical infrastructure providers, or high-profile institutions. All publicly trusted CAs are expected to comply with the 24-hour and 5-day revocation timelines in the event of certificate misissuance. This point has been reiterated numerous times.
Publicly trusted CAs should already have in place internal processes and external agreements with their subscribers to support compliance with the requirements for timely revocation. Delays due to a subscriber’s internal procedures or regulatory hurdles are not valid reasons to exceed the maximum allowable timeframe.
The “Analysis” section of your incident report remains wholly inadequate, and it must be revised substantially to meet the requirements of the CCADB Incident Reporting Guidelines and community expectations for delayed revocation incidents.
Again, it is insufficient to simply assert that certificates “belonged to customers of major importance for the national economy” or that revocation would disrupt services. These are vague, high-level assertions. As stated before, "there is heightened scrutiny of your CA and its responses". Detailed, evidence-based analyses of the specific factors that led to the decision to delay are required. These would include:
- Which specific customer/subscriber systems or services were at risk of disruption, how were those services relied upon by third parties or the public, and details of how revocation would have caused that disruption.
- What actual harm (quantified and clearly described) would have resulted to third parties if revocation had been performed within 24 hours, including how the harm was substantial, imminent and unavoidable, and that no other reasonable option existed.
- Why the situation could not have been mitigated in all other ways that could have been attempted by customers/subscribers, such as: rapid reissuance and replacement, working overtime, temporary failover, advance preparation, reversion to offline/paper-based operations, in-person service at physical locations, public advisories about service unavailability, private PKI, VPN tunneling, re-directs to other providers, selective shutdown of certain processes, emergency waivers from regulators, etc.
- Why each situation was exceptionally rare and unavoidable, why was this specific subscriber unlike any other, why was this situation unlike any other before and could not be anticipated, planned for, and mitigated in advance.
Until you provide this level of detail, your explanation cannot be properly evaluated by the community.
In addition, the current “Action Items” section is too shallow and reactive. While it lists technical corrections and minor documentation updates, it does not reflect the seriousness of this incident or the structural changes necessary to prevent recurrence.
A comprehensive plan is needed to address this situation. Netlock should first conduct a thorough post-mortem and incorporate its findings into policies, procedures, training, engineering, account management, plans, and planning.
Here are a few examples of Action Items to start with:
1. Revamp your customer agreements more thoroughly (I have not yet seen what the revised terms and conditions say).
Do they provide?
- Explicit subscriber acknowledgment of Netlock’s absolute right to revoke certificates as required by the Baseline Requirements and their obligation to support revocation/replacement within 24 hours of misissuance or compromise, regardless of internal policies, procedures, or rules.
- Contractual commitments that override internal change freezes, regulatory red tape, legal challenges, or coordination delays.
2. Automate Certificate Replacement and Revocation Workflows
- Ensure that critical subscribers have automated systems for certificate replacement and revocation, and create a mechanism to regularly test such preparedness.
- Advance preparation with automation and dedicated workflows: Set up ACME/ARI and similar automated processes for rapid reissuance to minimize deployment delays for critical subscribers and/or create a separate, streamlined path for high-priority certificate replacements.
- Adopt a rule that if Netlock suspects that a subscriber lacks the capability to have its certificate revoked within 24 hours, then Netlock will not issue publicly trusted certificates to them.
3. Develop and Implement a Mass Revocation Incident Preparation and Testing Plan – see https://wiki.mozilla.org/CA/Mass_Revocation_Events
Assignee | ||
Comment 14•4 months ago
|
||
We acknowledge and accept that neither the Baseline Requirements, the Mozilla Root Store Policy, nor the CCADB Incident Reporting Guidelines allow any exceptions to revocation timelines based on subscriber importance or sector affiliation, including government entities or critical infrastructure. As a publicly trusted CA, NetLock accepts that it must comply unconditionally with the 24-hour deadline, and that delays stemming from subscriber-related factors such as operational impact, legal constraints, or internal procedures are not valid justifications. In
We have previously acknowledged that we made a mistake when the event took place and, in response to questions from community members, we have explained what led to the incorrect decision in the past, which does not mean that we now believe that we have followed the community's rules and do not wish to correct our mistake. Our processes are designed to ensure that the incident does not happen again.
We deeply regret that our prior incident analysis included generalized statements about economic importance and potential disruption, without supplying adequate supporting detail. These high-level justifications were insufficient, and we are currently reworking the relevant sections of our report accordingly.
We recognize that trust in a CA is predicated not only on compliance but also on our ability to learn, correct, and improve.
We review our contracts and prepare changes that will allow us to operate properly and respond quickly. We contact our critical partners to jointly investigate the processes their systems have in place to achieve very fast certificate revocations.
We will develop a scenario for mass certificate revocation and issuance, drawing on our previous experience and the results of discussions with partners. The aim is, of course, to develop and document a compliant process.
Accordingly we update the Analysis and Action Items sections and continue to provide transparent progress reports every week until the community’s concerns are fully addressed.
Assignee | ||
Comment 15•4 months ago
|
||
Dear community members,
do you have any further questions?
We will update and post the updated action items and Analysis on June 11.
Assignee | ||
Comment 16•4 months ago
|
||
Updated Root Cause Analysis
Technical Root Cause
The immediate technical cause of this incident was the continued use of a deprecated explicitText
value in the User Notice
extension of certain EV and OV TLS certificate profiles. This explicit text became non-compliant with the CA/Browser Forum Baseline Requirements (BRG 2.0) as of 2023-09-15 but was not removed from certificate profiles due to oversight in the update process.
-
Remediation Action:
Fix of certification profiles (Completed: 2024.04.09) -
Preventive Actions:
Zlint implementation (Completed on 2024.10.04) and
Monitoring phase 1 and 2 (Completed on 2024.05 and 2024.07)
These controls are now in place to ensure early detection of misconfigurations before and after issuance.
Organizational Root Cause
At the time of the incident, internal documentation and subscriber contracts lacked explicit provisions to enforce unconditional compliance with the 24-hour revocation rule (BR 4.9.1). This led to hesitation and delayed decision-making in the case of subscribers perceived as critical to national infrastructure, where service disruption was feared.
-
Affected decisions were based on vague interpretations of risk, and no quantified or documented harm assessments were made.
-
The CA misjudged the scope of its authority and obligations, incorrectly allowing subscriber-side constraints to override mandatory policy requirements.
-
Remediation Actions:
Terms and condition update with the 24H revocation deadline
(Completed 2025.03.10)Internal review and revision of subscriber contracts to ensure timely revocation support regardless of operational constraints
(In progress, due 2025.06.30)
Compliance Process Weakness
Emergency escalation paths, revocation readiness checks, and structured incident response planning were either absent or insufficient. NetLock did not have a predefined or tested process for coordinating fast revocation with high-risk subscribers, nor any clearly delegated authority structure for executing emergency actions unconditionally within the BR timeframes.
- Planned Preventive Actions:
Initiate bilateral readiness reviews with critical infrastructure subscribers
(Due 2025.07.15)Document and validate emergency contact and escalation paths with high-risk subscribers
(Due 2025.07.31)Develop and simulate a full mass certificate revocation and reissuance scenario
(Due 2025.07.31)Publish internal and external guidelines for emergency revocation protocol and mass remediation readiness
(Due 2025.07.31)
These measures are designed to ensure that all future revocations can occur within policy limits, regardless of subscriber readiness or operational context.
Updated Action Items
Action Item | Kind | Due Date | Status |
---|---|---|---|
Fix of certification profiles | Repair | 2024.04.09 | Completed |
Revocation | Remediate | Started 2024.04.06. Completed on 2024.04.09 | Completed |
Revocation of certificates of customers of high economic importance | Remediate | Completed on 2024.04.30. | Completed |
Zlint implementation | Prevent | Completed on 2024.10.04 | Completed |
Monitoring implementation on Certification issuance | Prevent | Completed on 2024-05-31 | Completed |
Monitoring phase 2 implementation on Checking after issuance | Prevent | Completed on 2024-07-31 | Completed |
Terms and condition update with the 24H revocation deadline | Remediate | 2025.03.10 | Completed |
Publication of the newly translated Policies | Remediate | 2025-05-26 | Completed |
CCADB documentation update | Remediate | 2025-05-26 | Copleted |
Internal review and revision of subscriber contracts to ensure timely revocation support regardless of operational constraints | Remediate | 2025.06.30 | In progress |
Initiate bilateral readiness reviews with critical infrastructure subscribers to assess technical and procedural revocation agility | Prevent | 2025.07.15 | In progress |
Document and validate emergency contact and escalation paths with high-risk subscribers | Prevent | 2025.07.31 | Planned |
Develop and simulate a full mass certificate revocation and reissuance scenario in collaboration with key partners | Prevent | 2025.07.31 | Planned |
Publish internal and external guidelines for emergency revocation protocol and mass remediation readiness | Prevent | 2025.07.31 | Planned |
Assignee | ||
Comment 17•3 months ago
|
||
Dear Community members,
The revision process is going as planned, we will update the ticket next week with any upcoming information.
Comment 18•3 months ago
|
||
Dear Community Members,
we would like to inform you that we are proceeding with the implementation of the tasks according to the planned to-do list. We will update this ticket next week.
Comment 19•3 months ago
|
||
Dear Community members,
The bilateral readiness reviews are going as planned, we will update the ticket next week with any upcoming information.
Comment 20•3 months ago
|
||
Dear Community Members,
we are currently executing the task of initiating bilateral readiness reviews with critical infrastructure subscribers to assess technical and procedural revocation agility. The implementation is proceeding according to plan and is expected to be completed on schedule. Further updates regarding subsequent activities will be provided in due course.
Comment 21•2 months ago
|
||
Dear Community Members,
we have completed the task: Initiate bilateral readiness reviews with critical infrastructure subscribers to assess technical and procedural revocation agility.
All remaining tasks will be carried out in accordance with the committed timeline. You will be kept informed about the progress of task implementation.
Comment 22•2 months ago
|
||
Dear Community Members,
we have updated the Action Items table to reflect the tasks that have already been completed, as well as those currently in progress. The ongoing items are proceeding according to the planned schedule.
Action Item | Kind | Due Date | Status |
---|---|---|---|
Fix of certification profiles | Repair | 2024.04.09 | Completed |
Revocation | Remediate | Started 2024.04.06. Completed on 2024.04.09 | Completed |
Revocation of certificates of customers of high economic importance | Remediate | Completed on 2024.04.30. | Completed |
Zlint implementation | Prevent | Completed on 2024.10.04 | Completed |
Monitoring implementation on Certification issuance | Prevent | Completed on 2024-05-31 | Completed |
Monitoring phase 2 implementation on Checking after issuance | Prevent | Completed on 2024-07-31 | Completed |
Terms and condition update with the 24H revocation deadline | Remediate | 2025.03.10 | Completed |
Publication of the newly translated Policies | Remediate | 2025-05-26 | Completed |
CCADB documentation update | Remediate | 2025-05.26 | Completed |
Internal review and revision of subscriber contracts to ensure timely revocation support regardless of operational constraints | Remediate | 2025.06.30 | Completed |
Initiate bilateral readiness reviews with critical infrastructure subscribers to assess technical and procedural revocation agility | Prevent | 2025.07.15 | Completed |
Document and validate emergency contact and escalation paths with high-risk subscribers | Prevent | 2025.07.31 | In progress |
Develop and simulate a full mass certificate revocation and reissuance scenario in collaboration with key partners | Prevent | 2025.07.31 | In progress |
Publish internal and external guidelines for emergency revocation protocol and mass remediation readiness | Prevent | 2025.07.31 | In progress |
Assignee | ||
Comment 23•2 months ago
|
||
Dear Community Members,
we would like to inform you that we are proceeding with the implementation of the tasks according to the planned to-do list. We will update this ticket next week.
Assignee | ||
Comment 24•2 months ago
|
||
Report Closure Summary
- Incident description:
This bug was opened to replace Bug 1891331 and to provide a comprehensive response to the community regarding the delayed revocation of TLS certificates containing a deprecatedexplicitText
value in the User Notice extension. The issue affected certificates issued under several intermediate CAs and was in violation of BRG 2.0 requirements in effect since 2023-09-15.
Root Cause Analysis
Technical Root Cause
The explicitText
field value remained in use due to oversight when updating profiles for BRG 2.0 compliance. The misconfiguration was not detected prior to issuance.
Remediation Action:
- Certification profiles fixed (Completed: 2024-04-09)
- Zlint integrated and configured (Completed: 2024-10-04)
- Monitoring implemented pre- and post-issuance (Completed: 2024-05-31 and 2024-07-31)
Organizational Root Cause
Subscriber contracts and internal policies did not explicitly require compliance with the 24-hour revocation deadline. This resulted in delayed decisions in the case of high-priority economic entities. Decisions were made without documented impact assessment and misinterpreted compliance obligations.
Remediation Action:
- Terms and Conditions updated with 24H rule (Completed: 2025-03-10)
- Contracts reviewed and revised (Completed: 2025-06-30)
Compliance Process Weakness
The organization lacked emergency escalation paths and incident coordination protocols. No simulations or drills had been done for mass revocation scenarios.
Preventive Actions Implemented:
- Bilateral readiness reviews with critical subscribers (Completed: 2025-07-15)
- Emergency contact/escalation validation (Completed: 2025-07-31)
- Simulation of mass revocation and reissuance (Completed: 2025-07-31)
- Publication of internal/external revocation protocols (Completed: 2025-07-31)
Impact
- Total certificates affected: 522
- Types affected: OV TLS, EV TLS
- Remaining valid certificates: 0
- Issuance halted: Immediately after notification
- Revocation completed:
- Standard certs: 2024-04-09
- Delayed (upon request): 2024-04-30
Analysis
NETLOCK acknowledges that revocation delays based on subscriber importance do not comply with industry policy. Although some certificates were delayed due to perceived national importance, NETLOCK now fully recognizes that sector or size is not a valid justification. This understanding is now reflected in revised internal procedures, contracts, and emergency playbooks.
Final Status of Action Items
Action Item | Kind | Due Date | Status |
---|---|---|---|
Fix of certification profiles | Repair | 2024-04-09 | Completed |
Revocation | Remediate | 2024-04-09 | Completed |
Revocation of certificates for high-economic-importance subscribers | Remediate | 2024-04-30 | Completed |
Zlint implementation | Prevent | 2024-10-04 | Completed |
Monitoring Phase 1: On certification issuance | Prevent | 2024-05-31 | Completed |
Monitoring Phase 2: Post-issuance checking | Prevent | 2024-07-31 | Completed |
Terms and Conditions update with 24H revocation clause | Remediate | 2025-03-10 | Completed |
Publication of newly translated Policies | Remediate | 2025-05-26 | Completed |
CCADB documentation update | Remediate | 2025-05-26 | Completed |
Internal review of subscriber contracts for unconditional revocation support | Remediate | 2025-06-30 | Completed |
Bilateral readiness reviews with critical infrastructure subscribers | Prevent | 2025-07-15 | Completed |
Emergency contact and escalation path documentation | Prevent | 2025-07-31 | Completed |
Simulation of full mass certificate revocation and reissuance | Prevent | 2025-07-31 | Completed |
Publication of revocation protocols and remediation guidelines | Prevent | 2025-07-31 | Completed |
Lessons Learned
-
Detection: Zlint and monitoring upgrades catch future misconfigurations proactively
-
Governance: Regulatory documents and subscriber agreements now enforce BR-compliant revocation without exception
-
Preparedness: Emergency protocols and readiness plans are documented, tested, and integrated with subscribers
-
Transparency: Ongoing community dialogue and weekly reporting ensured accountability
-
Commitment summary:
NETLOCK regrets the incident and fully acknowledges its initial missteps in revocation decision-making. Through technical, organizational, and procedural changes, the CA is now better equipped to ensure full and timely compliance with all industry standards, regardless of customer profile or operational complexity.
All Action Items disclosed in this report have been completed as described, and we request its closure.
Updated•2 months ago
|
Comment 25•2 months ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, it will be closed on approximately 2025-08-18.
Updated•1 month ago
|
Description
•