1732429 - Add telemetry for counting of searching via New Tab page

Status: VERIFIED FIXED

(Firefox :: Search, enhancement, P2)

Description

[Daisuke Akatsuka (:daisuke)]

After M1, we no longer count the number of searches that originate from the New Tab page.
In this bug, to recognize how the searching feature is working, we restore the telemetry.

Comment 1

[Daisuke Akatsuka (:daisuke)]

Attachment: Bug 1732429: Add telemetry for urlbar handoff.

Comment 2

[Adrian Florinescu [:aflorinescu]]

Daisuke, could you please add details for what we should verify when this is done.

Comment 3

[Daisuke Akatsuka (:daisuke)]

Hi Adrian! I'm sorry for late replying..
As we are going to add the following telemetries, I think we should verify whether or not the telemetry will be added correctly.

Scalars:

• browser.engagement.navigation.urlbar_handoff
• browser.search.content.urlbar_handoff
• browser.search.withads.urlbar_handoff

Histogram

• SEARCH_COUNTS XXX.urlbar-handoff

We can check the value on about:telemetry.

• Scalars: about:telemetry#keyed-scalars-tab
• Histogram: about:telemetry#keyed-histograms-tab_parent

And about STR for each, the test code might help.
https://phabricator.services.mozilla.com/D126636#change-wMkjFwwBq94Q

Comment 4

[Daisuke Akatsuka (:daisuke)]

Attachment: Data Collection Review

Comment 5

[Daisuke Akatsuka (:daisuke)]

Comment on attachment 9245352 [details]
Data Collection Review

Hello Megan. Could you review this?

Comment 6

[Megan McCorquodale]

Just got back from PTO, will review today.

Comment 7

[Megan McCorquodale]

Comment on attachment 9245352 [details]
Data Collection Review

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   Yes, this is documented in the probe dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Yes, Natalie Raketic will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, interaction data.

5. Is the data collection request for default-on or default-off?
   Default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data?
   No.

9. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 8

[Daisuke Akatsuka (:daisuke)]

Thank you very much for the review, Megan!
I will land this patch.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b774b77c6b0e
Add telemetry for urlbar handoff. r=Standard8

Comment 10

[Mark Banner (:standard8)]

Comment on attachment 9242987 [details]
Bug 1732429: Add telemetry for urlbar handoff.

Beta/Release Uplift Approval Request

• User impact if declined: None. Required for new tab/Pocket experiments.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: See comments in bug.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Minor adjustment to add more detailed telemetry for the handoff.
• String changes made/needed: None

Comment 11

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/b774b77c6b0e

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9242987 [details]
Bug 1732429: Add telemetry for urlbar handoff.

Approved for 94.0b9. Please verify that the incoming data from Nightly looks sane ASAP so we have time to address any issues before next week's RC build.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/b2552509b0fe

Comment 14

[Pablo]

Attachment: telemetry.jpg

Replicated the issue on nightly 95.0a1 (2021-10-11)
Verified fix on latest nightly 95 using windows10 64bit, MacOS 10.15 and Ubuntu 20.

Comment 15

[Pablo]

On Keyed Scalar

browser.engagement.navigation.urlbar_handoff
search_enter 1

browser.search.content.urlbar_handoff
google:tagged:firefox-b-d 1

browser.search.withads.urlbar_handoff
google:tagged 1

On Keyed Histograms > Parent

SEARCH_COUNTS
google-b-d.urlbar-handoff": {
"sum": 1,
"google.in-content:sap:firefox-b-d": {
"sum": 1,

Comment 16

[Pablo]

Standard8

I will log a new bug since i just saw that Duckduckgo in a search with ads, that actually shows adds in the page is not giving me this in telemetry:

1. browser.search.withads.urlbar_handoff
   (in keyed scalars.)

Comment 17

[Pablo]

opened Bug 1737145 for the duckduckgo issue

Comment 18

[Adrian Florinescu [:aflorinescu]]

Verified on

Enviroments

Beta 94.0b9

on

Ubuntu 20.04
WIndows 10
Mac 11.5.1
Windows 11

Note:

As a separate find, we also tested on windows 11 developer edition and there we noticed that the handoff part is completely missing from the recorded telemetry pings. Meaning the pings would look like:

• browser.search.content.urlbar
• browser.engagement.navigation.urlbar
• browser.search.withads.urlbar
  even though the search made was done from the websearch and the handoff worked correctly in the browser.
  However, as detailed above when testing on windows 11 RC edition (available for public), the pings are correctly listed with their _handoff tag.

:standard8, :daisuke, should we track the windows 11 dev separately?

Comment 19

[Mark Banner (:standard8)]

(In reply to Adrian Florinescu [:aflorinescu] from comment #18)

:standard8, :daisuke, should we track the windows 11 dev separately?

Yes please, I don't really see how this could be platform specific.

Comment 20

[anicholson]

Just to confirm - this includes browser.search.adclicks.urlbar_handoff correct? So the full set of telemetry added would be:

browser.search.content.urlbar
browser.engagement.navigation.urlbar
browser.search.withads.urlbar
browser.search.adclicks.urlbar_handoff

Is that right?

Comment 21

[Mark Banner (:standard8)]

(In reply to anicholson from comment #20)

Just to confirm - this includes browser.search.adclicks.urlbar_handoff correct? So the full set of telemetry added would be:

browser.search.content.urlbar
browser.engagement.navigation.urlbar
browser.search.withads.urlbar
browser.search.adclicks.urlbar_handoff

Is that right?

There is also a new addition to the SAP counts in SEARCH_COUNTS: <engine-name>.urlbar-handoff.

Comment 22

[anicholson]

(In reply to anicholson from comment #20)

Just to confirm - this includes browser.search.adclicks.urlbar_handoff correct? So the full set of telemetry added would be:

browser.search.content.urlbar
browser.engagement.navigation.urlbar
browser.search.withads.urlbar
browser.search.adclicks.urlbar_handoff

Is that right?

(Correction)
Here I meant:
browser.search.content.urlbar_handoff
browser.engagement.navigation.urlbar_handoff
browser.search.withads.urlbar_handoff
browser.search.adclicks.urlbar_handoff

Comment 23

[Vlad Lucaci, Desktop QA (:vlucaci)]

Hello,

Since comment 18 was posted, a new Windows 11 Dev Channel update was released(22483.1011) which apparently solved the "_handoff" tag issue. Now the affected telemetry pings are displayed:

• browser.search.content.urlbar_handoff
• browser.engagement.navigation.urlbar_handoff
• browser.search.withads.urlbar_handoff