Closed Bug 2007948 Opened 10 months ago Closed 6 months ago

NETLOCK: Full Incident Report was not published within 14 days of notification

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kaluha.roland, Assigned: kaluha.roland)

Details

(Whiteboard: [ca-compliance] [disclosure failure])

Preliminary Incident Report

Summary

  • Incident description:
    An external report indicates that NETLOCK has not provided a satisfactory incident report in Bugzilla within the required timeframe. Specifically, the Full Incident Report was not published within 14 days of notification, as required by CCADB policy. This concern has been raised in relation to Bugzilla bug 2004699.

  • Relevant policies:
    CCADB Incident Reporting Guidelines, including the requirement to publish a Full Incident Report within 14 days of incident notification.

  • Source of incident disclosure:
    Third-party report via Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=2004699, Comment 1).

Assignee: nobody → kaluha.roland
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [disclosure failure]

Full Incident Report

Summary

An external concern was raised via Bugzilla bug 2004699 regarding NETLOCK’s incident handling and control effectiveness, referencing prior commitments made in Bug 1904041 to improve the “six-eyes” review process. The reporter questioned why these controls did not detect that certain Authority Information Access (AIA) URIs referenced PEM-encoded certificates and whether the review process relied primarily on visual inspection of certificate contents rather than technical validation of URI behavior in accordance with applicable RFC requirements. The concern emphasized that the Web PKI community expects demonstrable technical blocking and verification controls, rather than reliance on procedural or human assurances.
In addition, the reporter requested specific audit workpapers or auditor statements for the 2023, 2024, and 2025 audit cycles, explaining how AIA URI validation was performed during audits. This report documents NETLOCK’s analysis of the incident, its root causes, and the corrective actions taken.

Impact

This incident is procedural and compliance-related in nature. No certificate mis-issuance, compromise of cryptographic material, or security breach occurred. Although the format of certain AIA resources was not compliant with expectations, the relevant certificate information remained accessible, and no customers or relying parties were exposed to increased security risk. The impact is therefore limited to process robustness and control completeness rather than relying-party security.

Timeline

  • 2025-12-27: External concern raised via Bugzilla bug 2004699 (Comment 1).
  • 2025-12-29: NETLOCK acknowledged the report and initiated internal investigation.
  • 2026-01-05: Root cause analysis completed.
  • 2026-01-06: Corrective and preventive actions defined.

Root Cause Analysis

The root cause of this incident was that, although the six-eyes review covers not only syntactic and visual checks of certificate fields but also the overall certificate issuance process, the specific format of AIA-referenced resources was not subject to a dedicated automated validation. Due to the introduction of linters in the issuance workflow, this aspect had not previously been verified separately.
Contributing factors included:

  1. Control scope gap:
    While the six-eyes principle was applied to the issuance process as a whole, automated validation of the content format returned by AIA URIs was not explicitly enforced.
  2. Absence of format-specific checks:
    No technical control existed to distinguish between DER-encoded and PEM-encoded resources referenced by AIA URIs.
  3. Audit methodology constraints:
    The auditor’s verification activities were based on sampling in accordance with the defined audit methodology. As a result, this specific issue was not identified during the audited samples.
  4. Reliance on established tooling assumptions:
    Existing linters ensured structural correctness but did not validate external resource encoding.

Lessons Learned

What went well

  • The identified format issue did not endanger customers or relying parties, as the relevant certificate information was accessible despite the encoding difference.
  • NETLOCK reacted promptly to the external report and conducted a focused internal investigation.
    What didn’t go well
  • The absence of automated, format-aware validation allowed the issue to pass existing controls.
  • The six-eyes review did not include explicit evidence from technical checks for this specific aspect.
    Where we got lucky
  • The issue did not result in security compromise or relying-party harm.

Action Items

Action Item Kind Due Date
Improve automated retrieval and content-type validation of all AIA URIs prior to certificate issuance Prevent 2026-02-12
Introduce hard blocking controls preventing issuance when AIA resources do not meet RFC expectations Prevent 2026-01-30
Update six-eyes review to explicitly require verification evidence from automated checks Detect 2026-01-09
Propose multi-level sampling approaches to the auditor for future audit procedures Mitigate 2026-03-09

Appendix

(In reply to Roland from comment #1)

Full Incident Report

...

Impact

This incident is procedural and compliance-related in nature. No certificate mis-issuance, compromise of cryptographic material, or security breach occurred. Although the format of certain AIA resources was not compliant with expectations, the relevant certificate information remained accessible, and no customers or relying parties were exposed to increased security risk. The impact is therefore limited to process robustness and control completeness rather than relying-party security.

Timeline

  • 2025-12-27: External concern raised via Bugzilla bug 2004699 (Comment 1).
  • 2025-12-29: NETLOCK acknowledged the report and initiated internal investigation.
  • 2026-01-05: Root cause analysis completed.
  • 2026-01-06: Corrective and preventive actions defined.
    ...
    What didn’t go well
  • The issue did not result in security compromise or relying-party harm.

As has been stated in previous incidents:

Please review the CCADB Incident Response Guidelines, particularly the sections on required templates and timelines.

In addition to not following the template:

Are there examples of “bad” practices?
Claims that are subjective, unqualified opinions, speculative, or impossible to substantiate:

  • Avoid making claims that are speculative or that cannot be corroborated (e.g. “there is no security impact due to this issue.”)

The above incident report is about the other incident's full incident report not being published within 14 days of notification. Most of the report does not deal with this issue at all, conflating it with the other incident's issues, causes and action items.

Dear Community Members,

thank you for your feedback; we are updating our previous report.

Full Incident Report

Summary

This incident report addresses a process and compliance issue: NETLOCK did not publish the Full Incident Report for a separate underlying incident within 14 days of notification, as required by the CCADB Incident Reporting Guidelines.

The notification regarding the underlying incident was received via Bugzilla bug 2004699 (Comment 1) on 2025-12-27. Based on this date, the deadline to publish the Full Incident Report was 2026-01-10. The Full Incident Report was not published by that deadline. This report documents the factors that led to the missed deadline and the corrective actions defined to prevent recurrence.

Impact

  • Total number of certificates:
    N/A — this incident relates solely to incident reporting timelines and does not involve certificate issuance.

  • Total number of "remaining valid" certificates:
    N/A — no certificates were within scope of this incident.

  • Affected certificate types:
    N/A — no certificate types were affected.

  • Incident heuristic:
    Process and compliance incident related to delayed publication of a Full Incident Report beyond the CCADB-required 14-day timeframe.

  • Was issuance stopped in response to this incident, and why or why not?:
    No. Certificate issuance processes were not altered in response to this incident, as the issue concerned reporting obligations rather than issuance activities.

  • Analysis:
    The observable outcome of this incident is that the Full Incident Report was published after the 14-day period specified in the CCADB Incident Reporting Guidelines. This report does not assess or infer security impact, as the scope of the incident is limited to compliance with reporting timelines.

  • Additional considerations:
    The incident highlights the need for clearly defined and enforceable internal controls governing reporting deadlines, independent of technical certificate issuance processes.

Timeline

  • 2025-12-27: Notification received via Bugzilla bug 2004699 (Comment 1).
  • 2026-01-10: Deadline to publish the Full Incident Report (14 days after notification).
  • 2026-01-19: Full Incident Report for this reporting-delay incident published.

Root Cause Analysis

The root cause of missing the 14-day publication deadline was the absence of a consistently enforced internal mechanism to track, escalate, and complete CCADB reporting deliverables within required timelines.

Contributing factors included:

  1. Deadline tracking gap:
    The incident handling process did not include a mandatory, centrally tracked milestone for the Full Incident Report publication deadline (T+14 days), supported by automated reminders or escalation.

  2. Ownership and escalation gap:
    End-to-end ownership for driving report completion and publication was not assigned in a way that ensured accountability under time constraints.

  3. Closure control gap:
    There was no control requiring confirmation that the Full Incident Report had been published, or that a justified delay had been communicated, before the reporting phase was considered complete.

Lessons Learned

What went well

  • Following community feedback, NETLOCK reviewed its incident reporting process and documented corrective actions to address the identified gap.

What didn’t go well

  • Internal processes did not reliably ensure adherence to the CCADB 14-day Full Incident Report publication requirement.
  • Reporting milestones lacked enforceable tracking and escalation mechanisms.

Where we got lucky

  • NETLOCK continuously reviews and seeks to improve its processes, and in this case the community explicitly highlighted a specific reporting non-compliance.
  • This feedback provided an opportunity to identify and address the gap in the reporting workflow before similar delays could recur.

Action Items

Action Item Kind Due Date
Implement a centrally tracked incident reporting checklist with explicit milestones for Preliminary (T+72h) and Full Report (T+14d) publication Prevent 2026-01-30
Assign a single accountable owner for each incident report and define an escalation path for at-risk milestones Prevent 2026-01-30
Introduce automated reminders and escalation notifications for reporting milestones (e.g. T+7, T+10, T+13) Detect 2026-02-12
Require documented evidence of report publication (or communicated extension) before closing the reporting phase Prevent 2026-02-12

Appendix

Dear Community Members,

the action items outlined above are being executed in accordance with the implementation plan and schedule.

We will provide formal updates to the Mozilla community regarding completed actions and relevant milestones, ensuring transparency and accountability throughout the process.

Dear Community Members,

here you can find the updated Action Items status

Action Items

Action Item Kind Due Date Status
Implement a centrally tracked incident reporting checklist with explicit milestones for Preliminary (T+72h) and Full Report (T+14d) publication Prevent 2026-01-30 Done
Assign a single accountable owner for each incident report and define an escalation path for at-risk milestones Prevent 2026-01-30 Done
Introduce automated reminders and escalation notifications for reporting milestones (e.g. T+7, T+10, T+13) Detect 2026-02-12 In progress
Require documented evidence of report publication (or communicated extension) before closing the reporting phase Prevent 2026-02-12 In progress

The action items outlined above are being executed in accordance with the implementation plan and schedule.

We will provide formal updates to the Mozilla community regarding completed actions and relevant milestones, ensuring transparency and accountability throughout the process.

(In reply to Roland from comment #5)

Action Items

Action Item Kind Due Date Status
Implement a centrally tracked incident reporting checklist with explicit milestones for Preliminary (T+72h) and Full Report (T+14d) publication Prevent 2026-01-30 Done
Assign a single accountable owner for each incident report and define an escalation path for at-risk milestones Prevent 2026-01-30 Done

In the interests of transparency what does the centrally tracked incident reporting checklist show for #2011314?

I will note this was allegedly implemented 2026-01-30 but as of 2026-02-02 has not highlighted ongoing non-compliance. If this Action Item was to Prevent anything then it is not doing it. The alternative is that it showing non-compliance but not being reported on in a timely manner.

Dear Community Members,

thank you for your feedback and observations. We appreciate the constructive and supportive intent of the questions and comments, and we hope that, based on these, we can further clarify and shed light on the underlying issue.
Although the question refers to another bug, which addresses a different root cause, the current issue originated from an error identified during the handling of that underlying problem. Therefore, we are, of course, happy to respond to questions related to the associated issue.
The centrally tracked incident reporting system is a follow-up tracking mechanism that monitors reported bugs from the time they are entered into the system. This tracking system is under continuous oversight, with ongoing review and verification performed collaboratively by a dedicated case handler, an additional colleague, and a supervisor.
It is important to note, however, that if incorrect data is initially entered into the system—for example, due to differing interpretations—the system will continue to process that data accordingly and may produce incorrect indications. The system is currently under configuration, and during this process, additional criteria are being introduced. These include validation and control questions designed to assist administrators in correctly interpreting and entering dates and other relevant information.
In the specific case referenced here, the system used the times of the preliminary report of the first bug as the reference point and calculated the submission deadline for the Full Incident Report based on that. The delayed bug reporting itself has been recorded separately under https://bugzilla.mozilla.org/show_bug.cgi?id=2011314.
The discrepancy in the recorded times was caused by the factors described above and is expected to be resolved as part of the ongoing system configuration.

(In reply to Roland from comment #7)

Dear Community Members,

thank you for your feedback and observations. We appreciate the constructive and supportive intent of the questions and comments, and we hope that, based on these, we can further clarify and shed light on the underlying issue.
Although the question refers to another bug, which addresses a different root cause, the current issue originated from an error identified during the handling of that underlying problem. Therefore, we are, of course, happy to respond to questions related to the associated issue.
The centrally tracked incident reporting system is a follow-up tracking mechanism that monitors reported bugs from the time they are entered into the system. This tracking system is under continuous oversight, with ongoing review and verification performed collaboratively by a dedicated case handler, an additional colleague, and a supervisor.
It is important to note, however, that if incorrect data is initially entered into the system—for example, due to differing interpretations—the system will continue to process that data accordingly and may produce incorrect indications. The system is currently under configuration, and during this process, additional criteria are being introduced. These include validation and control questions designed to assist administrators in correctly interpreting and entering dates and other relevant information.
In the specific case referenced here, the system used the times of the preliminary report of the first bug as the reference point and calculated the submission deadline for the Full Incident Report based on that. The delayed bug reporting itself has been recorded separately under https://bugzilla.mozilla.org/show_bug.cgi?id=2011314.
The discrepancy in the recorded times was caused by the factors described above and is expected to be resolved as part of the ongoing system configuration.

That is nice, however it is not what I asked.

In the interests of transparency what does the centrally tracked incident reporting checklist show for #2011314?

More specifically what did it show as of 2026-02-02?

Dear Community Members,
Please clarify the question a bit more precisely, as the checklist consists of multiple items, and we want to ensure we provide an accurate answer. A checklist includes multiple items, such as the start date, the occurrence time, policy references needed for the timing, the responsible parties, and the related approval process. Thank you in advance.

(In reply to Nikolett from comment #9)

Dear Community Members,
Please clarify the question a bit more precisely, as the checklist consists of multiple items, and we want to ensure we provide an accurate answer. A checklist includes multiple items, such as the start date, the occurrence time, policy references needed for the timing, the responsible parties, and the related approval process. Thank you in advance.

The start date and occurrence time, as recorded on 2026-02-02.

Dear Community Members,

for Bug #2011314, the start date recorded in the tracking system was 2026-01-19, and this date was entered as the occurrence date in the checklist.
In accordance with the comments and subsequent review, this has since been corrected to reflect the appropriate reference date.
We hope this clarifies the record.

Dear Community Members,

we have configured our tracking system and are continuously testing it in a separate test process within a test environment. The test results will be evaluated after a few weeks of testing. If the outcome of the evaluation confirms that the testing is satisfactory, we will provide an update to the Community accordingly.
We are also striving to ensure that whenever an external notification is received, the resolution of the issue begins immediately—as has been our practice so far—and that a bug ticket is created without delay.This ensures that we can meet our time-related obligations within the shortest possible timeframe.

Dear Community Members,
We have continued the testing of our tracking system during this week as well within the dedicated test environment. Based on our current plan, we intend to evaluate the collected test data next week and finalize the configuration of our tracking system afterwards.
Following the finalization, the system will be reviewed on a monthly basis to ensure its proper and effective operation.
We will keep the Community informed of any further developments.

Dear Community Members,

we continued the internal review of our tracking system in our dedicated test environment during this week. At this stage, no significant changes have been introduced compared to the previous update, and the testing process is still ongoing.
We will continue monitoring the results and will provide further updates once there are notable developments.

Dear Community Members,

during this week, we have completed the testing and configuration of our tracking system.
Based on the test results, the system has now been introduced into the production environment and is actively used in our incident handling processes.
We are confident that the implemented controls will support us in meeting our time-related obligations, and we expect that, going forward, community members will not encounter cases resulting from missed reporting deadlines.
We will continue to monitor the system and provide further updates if relevant.

Dear Community Members,

we would like to provide our weekly update regarding the status of this incident.

Following the introduction of our centrally tracked incident reporting system, it is now actively supporting our incident handling and reporting processes. During this week, we have focused on monitoring its operation in practice and verifying that the implemented controls function as intended.

So far, the system is operating as expected, and we continue to review its performance to ensure consistent compliance with CCADB reporting timelines. No issues have been identified that would impact its effectiveness.

We will continue to monitor the system and will provide further updates should any relevant developments arise.
In the meantime, should you have any further questions, we are happy to assist.

We will also prepare and share the closure report in the near future.

Dear Community Members,

we would like to provide this week’s update regarding the incident.

Since our last communication, no further changes have occurred. The centrally tracked incident reporting system remains in active use within our incident handling processes, and we continue to monitor its operation in practice. Based on our observations, the implemented controls are functioning as intended and support compliance with CCADB reporting timelines.

At this stage, we consider the corrective and preventive measures complete for this incident.

If the community has any remaining questions, feedback, or requests for clarification, we are happy to address them. Otherwise, we will proceed with preparing and publishing the closure report shortly.

We thank the community for the continued guidance and support throughout this process.

Report Closure Summary

  • Incident description:
    NETLOCK did not publish the Full Incident Report for an underlying incident within the 14-day deadline required by the CCADB Incident Reporting Guidelines. The notification was received on 2025-12-27, establishing a publication deadline of 2026-01-10, which was missed.

  • Incident Root Cause(s):
    The incident was caused by the absence of a reliably enforced internal control framework to track and ensure compliance with CCADB reporting deadlines. Specifically, there was no centrally governed mechanism to monitor the T+14-day milestone, no clearly enforced single-point ownership for timely report publication, and no mandatory verification step to confirm report publication prior to closure of the reporting phase. Additionally, early-stage deficiencies in deadline calculation logic (e.g., incorrect reference date selection) contributed to inaccurate tracking signals.

  • Remediation description:
    NETLOCK implemented a centrally tracked incident reporting system with explicit milestone management for Preliminary and Full Incident Reports. A single accountable owner is now assigned to each incident, supported by defined escalation paths for at-risk deadlines. Automated reminder and escalation mechanisms were introduced, and a mandatory control now requires documented evidence of report publication (or communicated delay) before the reporting phase can be closed. The system has been fully configured, tested in a dedicated environment, and deployed to production, where its operation is continuously monitored.

  • Commitment summary:
    NETLOCK commits to continuous monitoring and periodic review of the incident reporting system to ensure sustained compliance with CCADB timelines. The organization will maintain monthly reviews of the tracking system’s effectiveness, refine validation logic (including correct reference date handling), and incorporate lessons learned into ongoing process improvements. NETLOCK also commits to transparent communication with the community regarding any future deviations or identified weaknesses in reporting controls.

All Action Items disclosed in this report have been completed as described, and we request its closure.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-04-20.

Whiteboard: [ca-compliance] [disclosure failure] → [close on 2026-04-20] [ca-compliance] [disclosure failure]
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Whiteboard: [close on 2026-04-20] [ca-compliance] [disclosure failure] → [ca-compliance] [disclosure failure]
You need to log in before you can comment on or make changes to this bug.