Closed Bug 1674874 Opened 4 years ago Closed 4 years ago

Record telemetry for results picked in search mode


(Firefox :: Address Bar, enhancement, P1)




84 Branch
84.2 - Nov 2 - Nov 15
Tracking Status
firefox83 + verified
firefox84 --- verified


(Reporter: bugzilla, Assigned: bugzilla)




(2 files)

Data Science would like a count of results picked when in search mode, divided by search mode entry point. This will help them determine the conversion rate of search mode entry to a result being picked.

I propose that we use a similar model to the urlbar.picked.* keyed scalars introduced in bug 1671508. Those scalars suffixed the scalar name with the picked result type. The keys are the index of the picked result. In this bug, we could introduce scalars named urlbar.picked.searchmode.* where the scalar name is suffixed with the search mode entry point. The keys would be the result type. For example, if a user clicks a one-off to enter search mode then picks a search suggestion result, we would record

urlbar.picked.searchmode.oneoff: {
  searchsuggestion: 1,

The keys for urlbar.picked.searchmode.* would be the same as the scalar suffixes for urlbar.picked, since we'd record all the same result types.

Severity: -- → N/A

I spoke with Teon about this. Instead of the keys being result type, they will be result indices 0-9, to match urlbar.picked.*. E.g. we will record

urlbar.picked.searchmode.oneoff: {
  4: 1,

when the user selects a one-off then selects the result in the 5th position.

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

Request for data collection review form

  1. What questions will you answer with this data?

How many results are picked when in Search Mode. Since we already have data on the number of times Search Mode is entered, this will allow us to calculate the number of completed engagements.

  1. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

Preliminary Search Mode data on how often it is entered are unexpectedly high. Calculating the number of completed engagements will tell us if these data represent real or accidental engagements.

  1. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We can look at the number of Search Mode entries with current telemetry, but without telemetry on the number of impressions, we can't determine if these are real or accidental engagements.

  1. Can current instrumentation answer these questions?

No, we only instrument how many times Search Mode is entered.

  1. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Category 2: Adding urlbar.picked.searchmode.* scalars. These keyed scalars are suffixed with the name of a search mode entry point. Their keys are the index of a picked result. For example, if a user selects a one-off search button to enter Search Mode, then selects the result at the third index, we would record:

urlbar.picked.searchmode.oneoff: {
  3: 1
  1. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

On the address bar Telemetry documentation page.

  1. How long will this data be collected? Choose one of the following:

I want to permanently monitor this data. Teon will look after it.

  1. What populations will you measure?

All release channels, countries, and locales.

  1. If this data collection is default on, what is the opt-out mechanism for users?

Standard Telemetry opt-out.

  1. Please provide a general description of how you will analyze this data.
    Calculating the number of completed Search Mode engagements by comparing the number of Search More entries with the number of picked results This will only be calculated as an aggregate.

  2. Where do you intend to share the results of your analysis?
    Internally, with the Search Team and Desktop leadership.

  3. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?

Attachment #9185597 - Flags: data-review?(mmccorquodale)

[Tracking Requested - why for this release]:

Data Science asked for this Telemetry to make important observations about the new Search Mode feature in 83. While the patch is large-ish, it only adds a couple of lines to product code: recording a telemetry probe from a codepath where we already record telemetry. The rest is telemetry definitions, documentation, and tests.

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
    Yes, this will be documented in the probe dictionary.

  2. Is there a control mechanism that allows the user to turn the data collection on and off?
    Yes, users can opt out of telemetry.

  3. If the request is for permanent data collection, is there someone who will monitor the data over time?
    Teon Brooks will monitor over time.

  4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
    Category 2, Interaction data.

  5. Is the data collection request for default-on or default-off?
    Default on.

  6. Does the instrumentation include the addition of any new identifiers?
    No new identifiers.

  7. Is the data collection covered by the existing Firefox privacy notice?

  8. Does there need to be a check-in in the future to determine whether to renew the data?

  9. Does the data collection use a third-party collection tool?

data-review +

Attachment #9185597 - Flags: data-review?(mmccorquodale) → data-review+

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

Beta/Release Uplift Approval Request

  • User impact if declined: Data Science will be lacking Telemetry they requested to monitor the success of the new search mode feature.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: 1. Open the address bar and select the Google one-off search button.
  1. Observe that we enter Google search mode (a blue "Google" indicator is shown in the address bar). Click on the first result.
  2. Open about:telemetry and search for urlbar.picked.searchmode.oneoff.
  3. Observe that the key 0 has a value of 1.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This patch just records new telemetry from one place where we already record telemetry. The vast majority of the code is just Telemetry definitions, tests, and documentation.

I requested QE verification, but the worst case with this patch is that no/bogus data is recorded in a new Telemetry probe I'm adding (so we'd just ignore it), so this could be uplifted before it's verified.

  • String changes made/needed:
Attachment #9185597 - Flags: approval-mozilla-beta?
Pushed by
Record telemetry for results picked in search mode. r=adw
Flags: needinfo?(htwyford)
Flags: needinfo?(htwyford)
Pushed by
Record telemetry for results picked in search mode. r=adw
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 84 Branch
Flags: qe-verify+

Comment on attachment 9185597 [details]
Bug 1674874 - Record telemetry for results picked in search mode. r?adw!

Approved for the beta branch before the merge, thanks.

Attachment #9185597 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

This needs a rebased patch for Beta83.

Flags: needinfo?(htwyford)
Attached patch beta-patch.diffSplinter Review

This patch is based on the latest mozilla-beta.

Flags: needinfo?(htwyford) → needinfo?(ryanvm)
Blocks: 1675976
Flags: needinfo?(ryanvm)
QA Whiteboard: [qa-triaged]

The issue is verified fixed using Fx84.0a1 and Fx83.0b10 on Windows 10, Ubuntu 18.04 and macOS 10.13. The "urlbar.picked.searchmode.oneoff" telemetry is correctly recorded when clicking a result from the address bar search drop-down. This has been tested using multiple one-offs with the same correct results.

Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.