1674493 - Record number of tab-to-search onboarding impressions

Status: VERIFIED FIXED

(Firefox :: Address Bar, enhancement, P2)

Description

[Harry Twyford]

We currently record the number of times tab-to-search onboarding results are picked in urlbar.searchmode.tabtosearch_onboard. We should also record the number of times these results are seen to measure their efficacy.

Comment 1

[Harry Twyford]

Attachment: Bug 1674493 - Record number of tab-to-search onboarding impressions. r?adw!

Comment 2

[Harry Twyford]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=543a9a9570b4283f165415411fc55b9f772f1282

Comment 3

[Harry Twyford]

Comment on attachment 9185198 [details]
Bug 1674493 - Record number of tab-to-search onboarding impressions. r?adw!

Request for data collection review form

1. What questions will you answer with this data?

How many tab-to-search onboarding results are seen by users.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

We'll be able to calculate the conversion rate for these results to determine if we need to tweak the frequency with which they appear.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We can look at the number of engagements with current telemetry, but without telemetry on the number of impressions, we can't calculate conversion.

4. Can current instrumentation answer these questions?

No, we only instrument engagement (i.e. how many times these results are picked).

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Adding a key to urlbar.tips: tabtosearch_onboard-shown. This is a Category 2 probe that counts the number of times tab-to-search onboarding results are shown to users.

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

On the address bar Telemetry documentation page.

7. How long will this data be collected? Choose one of the following:

I want to permanently monitor this data. Teon will look after it.

8. What populations will you measure?

All release channels, countries, and locales.

9. If this data collection is default on, what is the opt-out mechanism for users?

Standard Telemetry opt-out.

10. Please provide a general description of how you will analyze this data.
    Calculating conversion rates by comparing the number of impressions with the number of engagements. This will only be calculated as an aggregate.

11. Where do you intend to share the results of your analysis?
    Internally, with the Search Team and Desktop leadership.

12. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?
    No.

Comment 4

[Harry Twyford]

[Tracking Requested - why for this release]:

Data Science asked for this Telemetry to make important observations about the new Search Mode feature in 83. This is a small patch that records new telemetry from one place where we already record telemetry.

Comment 5

[Pascal Chevrel:pascalc (PTO until April 26)]

FYI, our last beta builds today and RC builds on Monday

Comment 6

[Megan McCorquodale]

Comment on attachment 9185198 [details]
Bug 1674493 - Record number of tab-to-search onboarding impressions. r?adw!

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   Yes, this will be documented in the probe dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Teon Brooks will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, interaction data.

5. Is the data collection request for default-on or default-off?
   default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data?
   No.

9. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 7

[Harry Twyford]

Comment on attachment 9185198 [details]
Bug 1674493 - Record number of tab-to-search onboarding impressions. r?adw!

Beta/Release Uplift Approval Request

• User impact if declined: Data Science will be lacking Telemetry they requested to monitor the success of the new search mode feature.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: 1. To make this test easier, set browser.urlbar.tabToSearch.onboard.maxShown to 100, browser.urlbar.tabToSearch.onboard.maxShownPerSession to 100, and browser.urlbar.tabToSearch.onboard.oneInteraction to true.

2. Open google.com a few times so that google.com is autofilled when you type part of google.com in the address bar.
3. When google.com is autofilled, observe that a double-sized result is shown as the second Urlbar result. Here's a screenshot of what it should look like: https://bugzilla.mozilla.org/show_bug.cgi?id=1665934#c2. Do not select the result. Close the address bar popup.
4. Go to about:telemetry and search for urlbar.tips.
5. Observe that the key tabtosearch_onboard-shown has a value of 1.

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a small patch that records new telemetry from one place where we already record telemetry.

I requested QE verification, but the worst case with this patch is that no/bogus data is recorded in a new Telemetry probe I'm adding (so we'd just ignore it), so this could be uplifted before it's verified.

• String changes made/needed:

Comment 8

[Harry Twyford]

Comment on attachment 9185198 [details]
Bug 1674493 - Record number of tab-to-search onboarding impressions. r?adw!

EDIT: Accidental post, please disregard.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5278acfcd834
Record number of tab-to-search onboarding impressions. r=adw

Comment 10

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/5278acfcd834

Comment 11

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9185198 [details]
Bug 1674493 - Record number of tab-to-search onboarding impressions. r?adw!

Approved for the beta branch before the merge, thanks.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Landed for 83.0b10.
https://hg.mozilla.org/releases/mozilla-beta/rev/daa8482d4bd5

Comment 13

[Hani Yacoub, Desktop QA]

"tabtosearch_onboard-shown" value on Windows 10 x64, Ubunutu 20.04 and on MacOS 10.15 on Firefox Nightly 84.0a1 (2020-11-08) and on Firefox 83.0b10 sometimes is 26, 29, 30 or 31. The value was never "1" as mentioned in comment 7 in step 5.

Are these the correct results we are looking for?

Comment 14

[Harry Twyford]

(In reply to Hani Yacoub from comment #13)

"tabtosearch_onboard-shown" value on Windows 10 x64, Ubunutu 20.04 and on MacOS 10.15 on Firefox Nightly 84.0a1 (2020-11-08) and on Firefox 83.0b10 sometimes is 26, 29, 30 or 31. The value was never "1" as mentioned in comment 7 in step 5.

Are these the correct results we are looking for?

No, thanks for finding this. I filed a bug. Looks like this probe is broken in 83. Recording bogus data was one of the risks I called out in the uplift request. Fortunately, no other Telemetry is affected, so this doesn't need to be backed out or anything. Data Science will likely just ignore the data from this probe recorded in 83.

Comment 15

[Hani Yacoub, Desktop QA]

Are there any further investigations should be done on this bug?
Should I verify the new bug logged once it's fixed?
Thanks.

Comment 16

[Harry Twyford]

Nothing more is needed on this bug. It would be great if you could verify the new bug once its fixed. I'll mark it qe-verify+. Thanks!

Comment 17

[Hani Yacoub, Desktop QA]

Closing this bug as verified as fixed following the steps from the follow up Bug 1676250.
Further verifications on Firefox Beta will be done on Bug 1676250.
Thanks.