2009612 - Telemetry: Settings, Models

Status: VERIFIED FIXED

(Core :: Machine Learning: Frontend, task, P1)

Description

[Chris Bellini (:cbellini)]

PRD sections

• https://docs.google.com/document/d/1dbbcvNhI7WRjsmWqHUlO_OZhu7xwYm3pG0KjimNGvbI/edit?tab=t.e148a5w2lful#heading=h.3ko6shcshpvt
• https://docs.google.com/document/d/1dbbcvNhI7WRjsmWqHUlO_OZhu7xwYm3pG0KjimNGvbI/edit?tab=t.e148a5w2lful#heading=h.i5oxx3num3vo

Comment 1

[:Gijs (he/him)]

Probes to implement:

Events:

• model_request
• model_response
• memories_optin
• memories_optout
• memories_panel_displayed
• memory_removed_panel
• memories_nuke

Metrics:

• smart_window.model

Comment 2

[Ed Lee :Mardak]

with bug 2021947 focusing on metering errors, this ticket is mostly settings related events for model selection and memory management

although i do see memories panel in comment 1 which is part of assistant ui probably handled in bug 2001527

Comment 3

[Ed Lee :Mardak]

giulia, bug 2023489 is updating MemoryStore.hardDeleteMemory to pass along what triggered deletion for both assistant and settings

Comment 4

[Giulia Cardieri (:gcardieri)]

Attachment: Bug 2009612 - include settings telemetry and its tests

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/3818f4491236
https://hg.mozilla.org/integration/autoland/rev/5f98a4b23117
include settings telemetry and its tests r=ai-frontend-reviewers,Gijs

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/b29464df6fee
https://hg.mozilla.org/integration/autoland/rev/0157fbc9d81f
Revert "Bug 2009612 - include settings telemetry and its tests r=ai-frontend-reviewers,Gijs" for causing bc failures @browser_aiFeatures_smart_window_telemetry.js.

Comment 7

[agoloman]

Backed out for causing bc failures @browser_aiFeatures_smart_window_telemetry.js.

• Backout link
• Push with failures
• Failure Log

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/2bf965317b8c
https://hg.mozilla.org/integration/autoland/rev/bd7318e26e46
include settings telemetry and its tests r=ai-frontend-reviewers,Gijs

Comment 9

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/bd7318e26e46

Comment 10

[:Gijs (he/him)]

Giulia, could you get this uplifted to 150 please, once this has been verified? Feel free to ping me here or on slack if you need help with this.

Rares, can we prioritize this for verification please? Thank you!

Comment 11

[Punam Dahiya [:pdahiya]]

[Tracking Requested - why for this release]:
Required for Smart Window Feature Fx150 MVP launch.

Comment 12

[Rares Doghi, Desktop QA]

Hi Gijs, Im having some issues with a few of those events:

memories_optin - missing
memories_optout - missing

memories_panel_displayed - will not show up if its displayed in Sidebar when a memory is used and opened by the user, as far as I cant tell it only showed up after I visited settings.

Comment 13

[Giulia Cardieri (:gcardieri)]

Hi @Rares,
The memories_optin and memories_optout are no longer part of this ticket, during development I talked with Vanessa from Data Science and we decided to change it. The updated metrics and events are listed in this google doc: https://docs.google.com/document/d/1EtlLGvLTSidFRoMMXGLLaNcxpYHjU-L5jYK7N88EGhw/edit?tab=t.0#heading=h.829h7k28fmpl

Here's an overall of what was implemented:

1. (event)memories_panel_displayed: User opens the memories preferences panel - this applies only to the settings, not to the sidebar
2. (event)settings_memories: User interacts with the checkboxes to change memories settings
3. (event)settings_model: User selects a different model from the settings page
   4.(event) memories_nuke: User clicks on “Delete all” button for memories
4. (metric) smart_window.model: Records the model selected by the user, or “custom-model”

Sorry this wasn't mentioned in the ticket, just in phabricator.

Comment 14

[Rares Doghi, Desktop QA]

Thank you for clearing that up. This issue is Verified as fixed in our latest Nightly 51.0a1 (2026-03-27)

Comment 15

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: This patch adds part of the telemetry required for the smart window launch, if this is declined an important chunk of telemetry data would be lost for smart window.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: This patch has a summary of changes: https://phabricator.services.mozilla.com/D288768
• Risk associated with taking this patch: low
• Explanation of risk level: Telemetry to the smart window settings for a feature behind a pref.
• String changes made/needed?: No
• Is Android affected?: no

Comment 16

[Giulia Cardieri (:gcardieri)]

Attachment: Bug 2009612 - include settings telemetry and its tests

Include telemetry for smart window preferences.

Events:

• memories nuke: when the delete all memories is clicked
• settings_memories: when the user checks/unchecks the memories checkboxes
• settings_model: when the user changes the model selection
• memories_panel_displayed: when the memories panel is visible

Metrics:

• model: when the model changes

Other updates:

• Using the MODELS constants for the model names
• Added a head_smart_window as a script to avoid repeating functions (over 40 lines of repetition)

Possible pain/reviewing points:

• head_smart_window is added as a script, should we do another approach? I could add the functions in the head.js but then they would be imported by all tests which seems unnecessary
• Global variable "previousAssistantModel" was needed to ensure when a custom model is selected we still know which model was previously set

Original Revision: https://phabricator.services.mozilla.com/D288768

Comment 17

[Dianna Smith [:diannaS]]

:gcardieri can you take a look at the intermittent regression before we land this on beta? (bug 2026069)

Comment 18

[Punam Dahiya [:pdahiya]]

Hi @diannaS Just commented in intermittent test regression bug closing it as it hasn't failed since merge and was only seen on autoland twice while merging and should be safe to unblock uplift thanks

Comment 19

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/88433fbe62c2
https://hg.mozilla.org/releases/mozilla-beta/rev/fb66bfab6ea9

Comment 20

[Rares Doghi, Desktop QA]

Verified as fixed in our latest Beta 150.0b3