1869765 - Add telemetry to search

Status: VERIFIED FIXED

(Firefox :: Firefox View, task, P1)

Description

[Sarah Clements [:sclements]]

Per discussions with product and UX, they'd like to have probes for the following:

• For Search across all sections/pages:
  ** How often and in what sections do users use search
  ** How many searches do they make before clicking on a list item (similar to bug 1801290 for cumulative searches). This would be based on a debounced search "term", so not per keystroke. We also need to account for if a user makes multiple searches but doesn't click on any search result.
• For Recent Browsing specifically
  ** How often do users click on "Show all" links in each component to see more results.
• For History specifically
  ** How often do people use Sort by Date or Sort by Site (with and without search)

Comment 1

[soulfire]

I think that this is a bad idea and it should not be implemented. It seems like an invasion to users privacy.

Comment 2

[Sarah Clements [:sclements]]

(In reply to soulfire from comment #1)

I think that this is a bad idea and it should not be implemented. It seems like an invasion to users privacy.

It is very clear from the description that we are not tracking specific search terms or what specific item users click; we are only looking at how often users use the search functionality within Firefox View to determine its usefulness and if we need to add additional functionality.

Additionally, Mozilla has strict data collection requirements and any telemetry that is added will need to follow the required process and be approved by a data steward.

Comment 3

[Nikki Sharpley (:nikkis) (she/her)]

(In reply to Sarah Clements [:sclements] from comment #0)

• For History specifically
  ** How often do people use Sort by Date or Sort by Site (with and without search)

I think this may already exist here:
browser/components/firefoxview/history.mjs

async onChangeSortOption(e) {
   this.sortOption = e.target.value;
   Services.telemetry.recordEvent(
     "firefoxview_next",
     "sort_history",
     "tabs",
     null,
     {
       sort_type: this.sortOption,
     }
   );
   await this.updateHistoryData();
   await this.#updateSearchResults();
 }

Unless we intend to measure this once a search is initiated separately.

Comment 4

[Nikki Sharpley (:nikkis) (she/her)]

(In reply to Sarah Clements [:sclements] from comment #0)

• For Recent Browsing specifically
  ** How often do users click on "Show all" links in each component to see more results.

Would we also want the category recorded as an extra key for this event?

Comment 5

[Sarah Clements [:sclements]]

(In reply to Nikki Sharpley (:nikkis) (she/her) from comment #4)

(In reply to Sarah Clements [:sclements] from comment #0)

• For Recent Browsing specifically
  ** How often do users click on "Show all" links in each component to see more results.

Would we also want the category recorded as an extra key for this event?

If by category, you mean which section in recent browsing the "show all" link is clicked then yes. I suspect we do that for "view all" so the section/category names should be the same.(In reply to Nikki Sharpley (:nikkis) (she/her) from comment #3)

(In reply to Sarah Clements [:sclements] from comment #0)

• For History specifically
  ** How often do people use Sort by Date or Sort by Site (with and without search)

I think this may already exist here:
browser/components/firefoxview/history.mjs

async onChangeSortOption(e) {
   this.sortOption = e.target.value;
   Services.telemetry.recordEvent(
     "firefoxview_next",
     "sort_history",
     "tabs",
     null,
     {
       sort_type: this.sortOption,
     }
   );
   await this.updateHistoryData();
   await this.#updateSearchResults();
 }

Unless we intend to measure this once a search is initiated separately.

Yes, if it is straight-forward to modify this to include whether or not a search preceeded a sort by date or site then please do so.

Comment 6

[Nikki Sharpley (:nikkis) (she/her)]

Attachment: Bug 1869765 - Add telemetry to Firefox View search. r=#fxview-reviewers

• record frequency and page of search
• record how often show all search results is clicked on recent browsing and from which section
• record how often sort by date or site is clicked on history

TO DO:

• record how many searches are made before clicking on a search result

Comment 7

[Nikki Sharpley (:nikkis) (she/her)]

Attachment: Bug 1869765 - Add test coverage for View search telemetry. r=#fxview-reviewers

Follow up patch to add tests for telemetry added in parent patch.

Comment 8

[Nikki Sharpley (:nikkis) (she/her)]

Attachment: bug1869765 - data request.rtf

Need info-ing jhirsch for a data review please. Cheers!

Comment 9

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 9376347 [details]
bug1869765 - data request.rtf

data-review+

---

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the usual Firefox data collection controls.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, nsharpley will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, interaction data.

5. Is the data collection request for default-on or default-off?

Default on.

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes.

8. Does the data collection use a third-party collection tool?

No.

Comment 10

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

I should add, just to clarify for next time, that you'll want to put a data steward's username in the data-review? field, rather than your own.

Requesting data-review will automatically create a BMO notification for that person, which saves you having to also add a needinfo.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6de7e0b51444
Add telemetry to Firefox View search. r=fxview-reviewers,kcochrane

Comment 12

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6de7e0b51444

Comment 13

[Sarah Clements [:sclements]]

Comment on attachment 9376331 [details]
Bug 1869765 - Add test coverage for View search telemetry. r=#fxview-reviewers

Edit: Uplift request made in error on the WIP test coverage patch - resetting it.

Comment 14

[Sarah Clements [:sclements]]

Comment on attachment 9375872 [details]
Bug 1869765 - Add telemetry to Firefox View search. r=#fxview-reviewers

Beta/Release Uplift Approval Request

• User impact if declined: None, but it will delay product/UX getting data on the search features being released in 123 and we'd miss having QA test this along with those search features.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: (It needs beta testing but QA already has a ticket tracking this.)
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This telemetry is only for the search features in Firefox View. Although this patch is lacking tests, there is a WIP patch for test coverage that I expect to land early this week. Additionally, we are landing this early so as to allow QA to do manual testing before the Wed deadline.
• String changes made/needed: No
• Is Android affected?: No

Comment 15

[Pascal Chevrel:pascalc]

Comment on attachment 9375872 [details]
Bug 1869765 - Add telemetry to Firefox View search. r=#fxview-reviewers

Approved for 123 beta 5, thanks.

Comment 16

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/eeea8c7a4b92

Comment 17

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-beta/rev/eeea8c7a4b92

Comment 18

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Verified that the telemetry for search is working as expected, data is collected and shown in about:telemetry (in keyed scalars, histograms as well as all the events) from each section of Firefox View. Tests were performed on macOS 13.6.4, Ubuntu 22.04 and Windows 10 with Firefox 123.0b6 / 124.0a1.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1db3eb95cc3b
Add test coverage for View search telemetry. r=fxview-reviewers,kcochrane

Comment 20

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/1db3eb95cc3b

Comment 21

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/1db3eb95cc3b