1654680 - Rethink telemetry for filter-style one-offs

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P2)

Description

[Harry Twyford]

From Marco's review of bug 1647888:

I was just thinking this will largely skew our one-offs telemetry, currently we measure searches started from one-off buttons, those will disappear but we'll also have no indication of a one-off button being picked. Do we already have a bug on file to fix telemetry? If not please file one.
We should ensure we measure when one-off buttons are picked (and likely which kind, engine or other), and probably update other telemetry so we don't count a search-mode search as a normal default engine search, also engagement telemetry wants some extra property to tell if search mode was enabled and which restriction was picked.

Comment 1

[Harry Twyford]

Teon, it would be great to get your input on how you'd like telemetry to work here. This bug refers to the filter-style one-offs introduced with the update2 prefs. Currently, when a one-off is selected we execute a search for the current search string. With the update2 prefs applied, one-offs instead enter a "search mode", showing suggestions for the selected engine. You can flip all the prefs prefixed browser.urlbar.update2 on Nightly to try it yourself. There's also a design spec here.

I'm wondering what level of detail you'd like about search mode in the Urlbar's telemetry probes. Some probes I have questions about:

1. FX_URLBAR_SELECTED_RESULT_TYPE_2
   Many kinds of results can appear in search mode, including bookmarks, history, searchengine, searchsuggestion, switchtab, remotetab, and formhistory. Do we need data about whether the result type was selected in search mode? We could register a bunch of new result types like bookmarks-searchmode, searchsuggestion-searchmode, etc., but that would really crowd the histogram.

2. Event telemetry
   I'm thinking of just adding a searchMode property to the event extra. It would be a string with the engine name/restriction mode (e.g. "Google", "Bookmarks", "DuckDuckGo", "History", etc.). Is that enough coverage?

3. SEARCH_COUNTS
   Do we want to treat search mode as a different SAP from urlbar?

4. browser.engagement.navigation.*
   Again, do we want to treat search mode as its own SAP? Treating it as a recorded action may not be enough since nearly any kind of result can appear in search mode.

Like Marco's quote in comment 0 points out, even with all these changes we will lose coverage of how often one-offs are selected, since selecting a one-off will no longer immediately execute a search. We also can't equate new search mode telemetry with one-off usage since in the future there will be other ways of entering search mode. Would you say this warrants an entirely new probe? Maybe a keyed scalar to count the number of times one-offs are clicked, keyed to the search engine/restriction mode.

Any input you can offer would be greatly appreciated!

Comment 2

[Harry Twyford]

I believe Teon is on PTO right now so I'm also tagging Ben. Ben, mikedeboer said he'd schedule a meeting for Data and Engineering to talk through this, but I'm tagging you here so you can take a look at this ahead of time.

Comment 3

[Teon Brooks [:teon]]

sorry I missed this before PTO. just updated my status

Comment 4

[Harry Twyford]

I met with Ben about this. We're going to add searchmode as a new SAP. It should eventually replace alias and oneoff as SAPs. After this feature launches, we expect alias and oneoff to go to zero.

We're also going to introduce some new parameters on event telemetry to record when search mode is entered. We're interested in recording how search mode was entered (one-off selected; typed alias; token alias engine picked; tab-to-search; etc) and what search mode was entered (engine name/source).

Engineering will meet with DS next week to discuss our patch. We should have one ready by then.

Comment 5

[Harry Twyford]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ca563cb294c9d96afa4f48b50be8f70b24997845

Comment 6

[Harry Twyford]

Attachment: Bug 1654680 - Add telemetry for search mode. r?adw!

Summary of changes:

1. Adds an entry property to the searchMode object to tag how search mode was entered.
2. Adds searchModeEntry and searchModeSrc to the event telemetry extra object. The former is entry, the latter is an engine name or a local source. This is collected on both engagement and abandonment when search mode is active.
3. Adds a urlbar-searchmode SAP to SEARCH_COUNTS.
4. Adds a browser.engagement.navigation.urlbar_searchmode probe.
5. Adds a urlbar_searchmode SAP to the navigation.search event.

Comment 7

[Harry Twyford]

This patch also fixes bug 1655849 on my machine.

Comment 8

[Harry Twyford]

Teon and I just met about this bug. We're not going to include the searchModeEntry/searchModeSrc fields in the event telemetry extra object. Instead, we are going to introduce a number of new keyed scalars, named searchmode.<entry point> (e.g.. searchmode.oneoff). The keys will be the name of the search mode source. Since keyed scalars can have a maximum of 100 keys, we will only specify built-in engines and the local modes ("bookmark", "history", "tabs"). Engines added by the user will just be grouped together as "other".

This approach will significantly increase the number of scalars: the patch will add seven, one for each search mode entry point plus one for "other". This is still preferable to using event telemetry, since scalars are collected for all users and are easier to analyze.

Comment 9

[Harry Twyford]

Comment on attachment 9170801 [details]
Bug 1654680 - Add telemetry for search mode. r?adw!

1. What questions will you answer with this data?

How the new search mode feature in the Urlbar is used, including how users find the feature and what parts of it they use the most.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

This is a new feature and we don't currently have any telemetry on it. We want to know how the feature is used, if at all, to know how to invest in it.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We plan on also doing user studies with this feature, but we want aggregate data to understand how it is used more broadly.

4. Can current instrumentation answer these questions?

No, this feature is not currently instrumented.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurements

SEARCH_COUNTS

Category 2: Adding a urlbar_searchmode SAP to SEARCH_COUNTS. This allows us to see how many searches are made in search mode and with what engine.

browser.engagement.navigation.urlbar_searchmode

Category 2: Adding a browser.engagement.navigation.urlbar_searchmode probe. This allows us to see how searches are confirmed when in search mode (pressing Enter on the first result, clicking a search suggestion, etc.)

navigation.search

Category 2: Adding a urlbar_searchmode SAP to the navigation.search event. navigation.search is legacy telemetry and is not currently collected. I'm just keeping it up to date. It's equivalent to browser.engagement.navigation.urlbar_searchmode, but can also report the picked search engine.

urlbar.searchmode.*

Category 2: Adding several urlbar.searchmode.* scalars. These scalars are suffix with an entry point into search mode, for example urlbar.searchmode.oneoff, or urlbar.searchmode.topsites_urlbar. That entry points tells us how search mode is entered most often. The keys for these scalars are strings describing what kind of search mode was entered. In most cases, this will be the name of a search engine, like "Google", or "DuckDuckGo". It may also be one of "history", "bookmarks", or "tabs". We only collect the names of engines that are bundled with Firefox. If the user enters search mode with an engine they installed themselves, we record "other" as the key.

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

The documentation on Urlbar telemetry will be updated with this patch to describe the new probes.

7. How long will this data be collected? Choose one of the following:

The search team will permanently monitor this data. Teon will look after it.

8. What populations will you measure?

All channels, countries, and locales.

9. If this data collection is default on, what is the opt-out mechanism for users?

Standard telemetry opt-out.

10. Please provide a general description of how you will analyze this data.

ni?teon

11. Where do you intend to share the results of your analysis?

Internally, with the search team and Firefox Desktop leadership.

12. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?

No.

Comment 10

[Harry Twyford]

Teon, could you please provide an answer for question 10?

Comment 11

[Harry Twyford]

(In reply to Harry Twyford [:harry] from comment #10)

answer for question 10?

From Teon: "we will use the measurement dashboard and STMO".

Comment 12

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9170801 [details]
Bug 1654680 - Add telemetry for search mode. r?adw!

Data Review Form

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, here: https://firefox-source-docs.mozilla.org/browser/urlbar/telemetry.html

2. Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Users can disable by turning off Telemetry in Preferences.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

:harry/:teon will monitor this data.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2: Interaction Data

5. Is the data collection request for default-on or default-off?

default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

No.

9. Does the data collection use a third-party collection tool? If yes, escalate to legal.

No.

data-review: r+

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/66bb2eab21ce
Add telemetry for search mode. r=adw

Comment 14

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/66bb2eab21ce

Comment 15

[Cristian Comorasu, QA [:ccomorasu], Releae Desktop QA - Please contact [e-mail]]

I verified this issue using 82.0a1 (2020-09-14) on macOS 10.13 and Windows 10 x64.
Adrian could you help me with Ubuntu verification?

Comment 16

[Adrian Florinescu [:aflorinescu]]

Let's keep this issue un-verified a bit more since this also implies updating the test coverage updates for the telemetry area and I'd like to make sure nothing gets missed. Leaving the NI as a reminder.

Comment 17

[Cristian Comorasu, QA [:ccomorasu], Releae Desktop QA - Please contact [e-mail]]

It is safe to close this issue now. We covered an extra telemetry run on Fx 82 and added telemetry test cases for the pre-release sign off(search shortcuts).