1676250 - urlbar.tips.tabtosearch_onboard-shown overcounts impressions

Status: VERIFIED FIXED

(Firefox :: Address Bar, defect, P1)

Description

[Harry Twyford]

urlbar.tips.tabtosearch_onboard-shown is counted every time UrlbarProviderTabToSearch sends an onboarding result to the muxer. This is wrong in two ways:

1. The probe is incremented on every keystroke rather than just once when the search engine domain is autofilled (see bug 1675611), and
2. The probe is incremented when the user's typed query partially matches a search engine domain, but it is not autofilled. For example, searching for "a" matches "amazon.com", but if the user's profile doesn't autofill "amazon.com", the onboarding result is not shown but the probe is incremented.

Comment 1

[Harry Twyford]

Attachment: Bug 1676250 - Stop overcounting urlbar.tips.tabtosearch_onboard-shown. r?adw!

Comment 2

[Harry Twyford]

[Tracking Requested - why for this release]:
Some Telemetry is broken in 83. Data Science and Product have asked that we prioritize a fix. The fix is limited in scope to only the affected probe.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/926adb78d9ef
Stop overcounting urlbar.tips.tabtosearch_onboard-shown. r=adw

Comment 4

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/926adb78d9ef

Comment 5

[Harry Twyford]

Comment on attachment 9188677 [details]
Bug 1676250 - Stop overcounting urlbar.tips.tabtosearch_onboard-shown. r?adw!

Beta/Release Uplift Approval Request

• User impact if declined: One Telemetry probe will continue to be broken on 84. It's a probe that measures a feature that's new to 83, so we'd like it fixed ASAP.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: 1. On a new en-US profile, visit amazon.com and bookmark it. Close the tab.

2. Start typing "amazon.com" in the address bar. Observe that a large "Search with Amazon.com" result is shown as the second result once you type a few characters.
3. Continue typing all the letters of "amazon.com". The large result should be shown the entire time.
4. Click away from the address bar so it closes. Open about:telemetry and search for "urlbar.tips".
5. Observe the key tabtosearch_onboard-shown has a value of 1.

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Fix is limited in scope to just the affected probe. Worst case is that we record bad/no data for this single probe. Verified on Nightly with tests.
• String changes made/needed:

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9188677 [details]
Bug 1676250 - Stop overcounting urlbar.tips.tabtosearch_onboard-shown. r?adw!

Approved for 84.0b4.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/caf253aecfde

Comment 8

[Hani Yacoub, Desktop QA]

Verified as fixed on Windows 10 x64, Ubunutu 20.04 and on MacOS 10.15 on Firefox Nightly 85.0a1 (2020-11-22).

Comment 9

[Hani Yacoub, Desktop QA]

I tried to verify the bug on Firefox 84.0b4 also but the result of "tabtosearch_onboard-shown" is 2 after everytime I tried on all the OS's mentioned in the previous comment.
I thought at first that the changes are not landed on Firefox 84.0b4 yet but I verified in about:buildconfig and it shows that the changes should be on Firefox 84.0b4.
Could you please take a look at this?
Thanks.

Comment 10

[Harry Twyford]

I can't reproduce on 84b6. Are you sure that the Amazon tab-to-search result wasn't shown for a moment at some other point in your testing? For example, when you typed the a in about:telemetry? If it wasn't shown a second time, could you please upload a screen recording of the steps you took to get a value of 2?

Comment 11

[Hani Yacoub, Desktop QA]

I followed the same steps while testing on Firefox Nightly 85.0a1 (2020-12-01) and on Firefox Beta 84.0b7.
"tabtosearch_onboard-shown" value while testing on Nightly was 1 and while testing on Beta the value was 2.

Below I attached a screen recording with the results of testing on Beta and on Nightly on MacOS 10.14.6
Beta: https://imgur.com/a/S3vQjJK
Nightly: https://imgur.com/a/uDxi10k

Comment 12

[Harry Twyford]

Thanks for the screen recordings. In the Beta screen recording around 20s from the end, you can see that the Amazon tab-to-search item is shown a second time when you type the "a" in about:telemetry. That means this is working as expected. Thank you!

Comment 13

[Hani Yacoub, Desktop QA]

Based on comment 12 I will mark this as verified as fixed.
Thank you!