2034541 - Send user feedback payload via Glean

Status: VERIFIED FIXED

(Core :: Machine Learning: Frontend, task)

Description

[Justin Levinsohn]

Instrument and send Smart Window user feedback data via Glean.

• Feedback data is collected when user:

  • clicks thumbs up
  • clicks thumbs down
  • submits feedback from modal

• Glean payload follows the structure defined in the feedback JSON spec

• Glean configuration is defined appropriately (e.g. metrics.yaml and/or pings.yaml)

• Data collection category is determined and documented

• Required data review and approvals are completed prior to landing (if applicable)

• Collection is only enabled when pref smartwindow.user_feedback_collection is enabled

• Tests are added to validate telemetry collection

• Instrumentation does not block feedback UI behavior

• Collected data is validated post-landing to confirm expected payloads are received

Reference

• User Feedback JSON Spec

Comment 1

[echa]

Attachment: Bug 2034541 - Add dedicated highly_sensitive metric for Smart Window feedback chat history r?pdahiya

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/103c7411199d
https://hg.mozilla.org/integration/autoland/rev/0c2b48498683
Add dedicated highly_sensitive metric for Smart Window feedback chat history r=pdahiya,omc-reviewers,data-stewards,aminomancer

Comment 3

[echa]

For QA, here are a few fields that require specific actions to populate:

toolUIData (requires a tab action)

1. Ask Smart Window to close a tab.

2. Submit feedback.

3. Verify the assistant message associated with the tab action contains:

   • toolUIData.uiType
   • toolUIData.toolCallId

Page content toggle

1. Ask Smart Window to summarize a page.
2. Submit feedback with "Include page content" checked.
3. Verify the recorded chat contains tool result messages (role: 3) with page content in content.body.result.

Repeat with "Include page content" unchecked:

1. Ask Smart Window to summarize a page.
2. Uncheck "Include page content" and submit feedback.
3. Verify tool result messages (role: 3) are absent from the recorded chat.

Comment 4

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/0c2b48498683

Comment 5

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: If declined, Smart Window feedback reports will continue to omit the conversation history that led to the user's feedback. This limits our ability to understand reported issues, reproduce user-reported problems, and evaluate feedback in the context of the interaction that generated it.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: Enable Smart Window and start a conversation.
  Ask Smart Window to summarize a webpage.
  Open the thumbs up or thumbs down feedback modal.
  Verify the chat preview is displayed.
  Submit feedback with Include page content checked.
  Verify the submitted telemetry contains the conversation history, including tool result messages (role: 3) containing page content.
  Repeat with Include page content unchecked.
  Verify tool result messages (role: 3) are excluded from the submitted conversation history.

Additional verification:
Ask Smart Window to perform a tab action (for example, close a tab).
Submit feedback.
Verify the associated assistant message contains toolUIData.uiType and toolUIData.toolCallId.

• Risk associated with taking this patch: medium
• Explanation of risk level: This patch adds a new telemetry metric and updates the Smart Window feedback flow to include conversation history in submitted feedback reports. It touches feedback UI, telemetry collection, and payload construction logic.
• String changes made/needed?: no
• Is Android affected?: unknown

Comment 6

[echa]

Attachment: Bug 2034541 - Add dedicated highly_sensitive metric for Smart Window feedback chat history

This patch adds a dedicated highly_sensitive microsurvey.smart_window.chat metric for the chat log and keeps the remaining feedback metadata in microsurvey.smart_window.user_feedback_data (technical). This separation allows the chat data to be handled independently and requires Sensitive Data Collection Review

Original Revision: https://phabricator.services.mozilla.com/D304565

Comment 7

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined/Reason for urgency: If declined, Smart Window feedback reports will continue to omit the conversation history that led to the user's feedback. This limits our ability to understand reported issues, reproduce user-reported problems, and evaluate feedback in the context of the interaction that generated it.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: Enable Smart Window and start a conversation.
  Ask Smart Window to summarize a webpage.
  Open the thumbs up or thumbs down feedback modal.
  Verify the chat preview is displayed.
  Submit feedback with Include page content checked.
  Verify the submitted telemetry contains the conversation history, including tool result messages (role: 3) containing page content.
  Repeat with Include page content unchecked.
  Verify tool result messages (role: 3) are excluded from the submitted conversation history.

Additional verification:
Ask Smart Window to perform a tab action (for example, close a tab).
Submit feedback.
Verify the associated assistant message contains toolUIData.uiType and toolUIData.toolCallId.

• Risk associated with taking this patch: medium
• Explanation of risk level: This patch adds a new telemetry metric and updates the Smart Window feedback flow to include conversation history in submitted feedback reports. It touches feedback UI, telemetry collection, and payload construction logic.
• String changes made/needed?: no
• Is Android affected?: unknown

Comment 8

[echa]

Attachment: Bug 2034541 - Add dedicated highly_sensitive metric for Smart Window feedback chat history

This patch adds a dedicated highly_sensitive microsurvey.smart_window.chat metric for the chat log and keeps the remaining feedback metadata in microsurvey.smart_window.user_feedback_data (technical). This separation allows the chat data to be handled independently and requires Sensitive Data Collection Review

Original Revision: https://phabricator.services.mozilla.com/D304565

Comment 9

[Rares Doghi, Desktop QA]

I tried to verify this fix but after reaching about:glean and submitting a Microsurvey type of ping Im getting the following issues:

I am getting this error multiple times :
JSON_VALIDATION_ERROR_DEBUG_VIEW [{"keyword": "additionalProperties", "dataPath": ".ping_info", "schemaPath": "#/properties/ping_info/additionalProperties", "params":{"additionalProperty": "server_knobs_config"},"message": "should NOT have additional properties"}]

and for the Microsurvey ping I get this error:
"labeled_counter": {
"glean.error.invalid_value": {
"microsurvey.smart_window.chat": 1
}
},

Please also note that microsurvey.smart_window.user_feedback_data is left blank.
You can find my events here:
https://debug-ping-preview.firebaseapp.com/pings/raresTel
https://debug-ping-preview.firebaseapp.com/pings/raresTel/6f9f04d5-0113-4770-b64d-e2d28cfb1abe

Comment 10

[echa]

Thanks Rares! Seems there was a schema mismatch, which will be fixed with Bug 2050195
Also I can see microsurvey.smart_window.user_feedback_data being populated in your ping here? https://debug-ping-preview.firebaseapp.com/pings/raresTel/6f9f04d5-0113-4770-b64d-e2d28cfb1abe

"microsurvey.smart_window.user_feedback_data": { "metadata": { "model": "qwen3-235b-a22b-instruct-2507-maas", "turn_count": 4, "prompt_version": 7 }

Comment 11

[Rares Doghi, Desktop QA]

Verified as fixed in our latest Nightly ID 20260625094708, Please note that we still get the following error every time we open or Close the Feedback modal:

messaging-System
JSON_VALIDATION_ERROR_DEBUG_VIEW [{"keyword": "additionalProperties", "dataPath": ".ping_info", "schemaPath": "#/properties/ping_info/additionalProperties", "params":{"additionalProperty": "server_knobs_config"},"message": "should NOT have additional properties"}]

Comment 12

[echa]

Attachment: Bug 2034541 - Add dedicated highly_sensitive metric for Smart Window feedback chat history

This patch adds a dedicated highly_sensitive microsurvey.smart_window.chat metric for the chat log and keeps the remaining feedback metadata in microsurvey.smart_window.user_feedback_data (technical). This separation allows the chat data to be handled independently and requires Sensitive Data Collection Review

Original Revision: https://phabricator.services.mozilla.com/D304565

Comment 13

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: If declined, Smart Window feedback reports will continue to omit the conversation history that led to the user's feedback. This limits our ability to understand reported issues, reproduce user-reported problems, and evaluate feedback in the context of the interaction that generated it.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: Enable Smart Window and start a conversation.
  Ask Smart Window to summarize a webpage.
  Open the thumbs up or thumbs down feedback modal.
  Verify the chat preview is displayed.
  Submit feedback with Include page content checked.
  Verify the submitted telemetry contains the conversation history, including tool result messages (role: 3) containing page content.
  Repeat with Include page content unchecked.
  Verify tool result messages (role: 3) are excluded from the submitted conversation history.

Additional verification:
Ask Smart Window to perform a tab action (for example, close a tab).
Submit feedback.
Verify the associated assistant message contains toolUIData.uiType and toolUIData.toolCallId.

• Risk associated with taking this patch: medium
• Explanation of risk level: This patch adds a new telemetry metric and updates the Smart Window feedback flow to include conversation history in submitted feedback reports. It touches feedback UI, telemetry collection, and payload construction logic.
• String changes made/needed?: no
• Is Android affected?: unknown

Comment 14

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined/Reason for urgency: If declined, Smart Window feedback reports will continue to omit the conversation history that led to the user's feedback. This limits our ability to understand reported issues, reproduce user-reported problems, and evaluate feedback in the context of the interaction that generated it.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: Enable Smart Window and start a conversation.
  Ask Smart Window to summarize a webpage.
  Open the thumbs up or thumbs down feedback modal.
  Verify the chat preview is displayed.
  Submit feedback with Include page content checked.
  Verify the submitted telemetry contains the conversation history, including tool result messages (role: 3) containing page content.
  Repeat with Include page content unchecked.
  Verify tool result messages (role: 3) are excluded from the submitted conversation history.

Additional verification:
Ask Smart Window to perform a tab action (for example, close a tab).
Submit feedback.
Verify the associated assistant message contains toolUIData.uiType and toolUIData.toolCallId.

• Risk associated with taking this patch: medium
• Explanation of risk level: This patch adds a new telemetry metric and updates the Smart Window feedback flow to include conversation history in submitted feedback reports. It touches feedback UI, telemetry collection, and payload construction logic.
• String changes made/needed?: no
• Is Android affected?: unknown

Comment 15

[echa]

Attachment: Bug 2034541 - Add dedicated highly_sensitive metric for Smart Window feedback chat history

This patch adds a dedicated highly_sensitive microsurvey.smart_window.chat metric for the chat log and keeps the remaining feedback metadata in microsurvey.smart_window.user_feedback_data (technical). This separation allows the chat data to be handled independently and requires Sensitive Data Collection Review

Original Revision: https://phabricator.services.mozilla.com/D304565

Comment 16

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/cf31792e2c0d
https://hg.mozilla.org/releases/mozilla-beta/rev/a0a929d0110a

Comment 17

[Rares Doghi, Desktop QA]

Verified as fixed in our latest Beta Verified as fixed in our latest Beta 153.0b5

Comment 18

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/4ef037f38f48
https://hg.mozilla.org/releases/mozilla-release/rev/ca5c9adc9fcc

Comment 19

[Rares Doghi, Desktop QA]

Verified as fixed in our latest Release 152.0.4