1674874 - Record telemetry for results picked in search mode

Status: VERIFIED FIXED

(Firefox :: Address Bar, enhancement, P1)

Description

[Harry Twyford]

Data Science would like a count of results picked when in search mode, divided by search mode entry point. This will help them determine the conversion rate of search mode entry to a result being picked.

I propose that we use a similar model to the urlbar.picked.* keyed scalars introduced in bug 1671508. Those scalars suffixed the scalar name with the picked result type. The keys are the index of the picked result. In this bug, we could introduce scalars named urlbar.picked.searchmode.* where the scalar name is suffixed with the search mode entry point. The keys would be the result type. For example, if a user clicks a one-off to enter search mode then picks a search suggestion result, we would record

urlbar.picked.searchmode.oneoff: {
  searchsuggestion: 1,
}

The keys for urlbar.picked.searchmode.* would be the same as the scalar suffixes for urlbar.picked, since we'd record all the same result types.

Comment 1

[Harry Twyford]

I spoke with Teon about this. Instead of the keys being result type, they will be result indices 0-9, to match urlbar.picked.*. E.g. we will record

urlbar.picked.searchmode.oneoff: {
  4: 1,
}

when the user selects a one-off then selects the result in the 5th position.

Comment 2

[Harry Twyford]

Attachment: Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

Comment 3

[Harry Twyford]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a8527b22d3bdccd5d794c38a64384fdd0b9d2534

Comment 4

[Harry Twyford]

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

Request for data collection review form

1. What questions will you answer with this data?

How many results are picked when in Search Mode. Since we already have data on the number of times Search Mode is entered, this will allow us to calculate the number of completed engagements.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

Preliminary Search Mode data on how often it is entered are unexpectedly high. Calculating the number of completed engagements will tell us if these data represent real or accidental engagements.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We can look at the number of Search Mode entries with current telemetry, but without telemetry on the number of impressions, we can't determine if these are real or accidental engagements.

4. Can current instrumentation answer these questions?

No, we only instrument how many times Search Mode is entered.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Category 2: Adding urlbar.picked.searchmode.* scalars. These keyed scalars are suffixed with the name of a search mode entry point. Their keys are the index of a picked result. For example, if a user selects a one-off search button to enter Search Mode, then selects the result at the third index, we would record:

urlbar.picked.searchmode.oneoff: {
  3: 1
}

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

On the address bar Telemetry documentation page.

7. How long will this data be collected? Choose one of the following:

I want to permanently monitor this data. Teon will look after it.

8. What populations will you measure?

All release channels, countries, and locales.

9. If this data collection is default on, what is the opt-out mechanism for users?

Standard Telemetry opt-out.

10. Please provide a general description of how you will analyze this data.
    Calculating the number of completed Search Mode engagements by comparing the number of Search More entries with the number of picked results This will only be calculated as an aggregate.

11. Where do you intend to share the results of your analysis?
    Internally, with the Search Team and Desktop leadership.

12. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?
    No.

Comment 5

[Harry Twyford]

[Tracking Requested - why for this release]:

Data Science asked for this Telemetry to make important observations about the new Search Mode feature in 83. While the patch is large-ish, it only adds a couple of lines to product code: recording a telemetry probe from a codepath where we already record telemetry. The rest is telemetry definitions, documentation, and tests.

Comment 6

[Megan McCorquodale]

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   Yes, this will be documented in the probe dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Teon Brooks will monitor over time.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, Interaction data.

5. Is the data collection request for default-on or default-off?
   Default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data?
   No.

9. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 7

[Harry Twyford]

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

Beta/Release Uplift Approval Request

• User impact if declined: Data Science will be lacking Telemetry they requested to monitor the success of the new search mode feature.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: 1. Open the address bar and select the Google one-off search button.

2. Observe that we enter Google search mode (a blue "Google" indicator is shown in the address bar). Click on the first result.
3. Open about:telemetry and search for urlbar.picked.searchmode.oneoff.
4. Observe that the key 0 has a value of 1.

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch just records new telemetry from one place where we already record telemetry. The vast majority of the code is just Telemetry definitions, tests, and documentation.

I requested QE verification, but the worst case with this patch is that no/bogus data is recorded in a new Telemetry probe I'm adding (so we'd just ignore it), so this could be uplifted before it's verified.

• String changes made/needed:

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1c37a165ba43
Record telemetry for results picked in search mode. r=adw

Comment 9

[Narcis Beleuzu [:NarcisB]]

Backed out for bc failure on browser_UsageTelemetry_urlbar_searchmode.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/9d28f7aac6577dfc238f02d08a6e0af2ef1916ac
Log link: https://treeherder.mozilla.org/logviewer?job_id=320876498&repo=autoland&lineNumber=9808

Comment 10

[Harry Twyford]

New try for new revision on Phabricator: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1c40c437cb5d0f3124f7091a00320f0b20350a5

Comment 11

[Harry Twyford]

I needed to run a non-artifact try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2176bab9917484a83294f9fca30b6a4e17008cde

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3311fce1b2f4
Record telemetry for results picked in search mode. r=adw

Comment 13

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/3311fce1b2f4

Comment 14

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

Approved for the beta branch before the merge, thanks.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

This needs a rebased patch for Beta83.

Comment 16

[Harry Twyford]

Attachment: beta-patch.diff

This patch is based on the latest mozilla-beta.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/c7c5be78c65b2651c1373975ce7596729d2dbcc5

Comment 18

[Cristian Baica [:cbaica], Release Desktop QA]

The issue is verified fixed using Fx84.0a1 and Fx83.0b10 on Windows 10, Ubuntu 18.04 and macOS 10.13. The "urlbar.picked.searchmode.oneoff" telemetry is correctly recorded when clicking a result from the address bar search drop-down. This has been tested using multiple one-offs with the same correct results.