1774305 - Add impression telemetry for autofill

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

As part of the adaptive history autofill experiment, we'll need to add impression telemetry for the different types of autofill. Per a Slack thread today, the definition of "impression" will be the same as the one we use for Suggest: An impression occurs when an autofill result is present at the end of a urlbar engagement. (We've been calling this "option 3" in threads and a doc or two.)

Comment 1

[Daisuke Akatsuka (:daisuke)]

Attachment: Bug 1774305: Add impression telemetry for autofill.

Comment 2

[Daisuke Akatsuka (:daisuke)]

Attachment: data-review-request-for-bug1774305.md

Hello Megan! Could you do data-review? Thanks!

Comment 3

[Megan McCorquodale]

Comment on attachment 9281577 [details]
data-review-request-for-bug1774305.md

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   This will be documented in the probe dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Firefox Suggest User Value team will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, Interaction Data.

5. Is the data collection request for default-on or default-off?
   Default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/299feb340bf2
Add impression telemetry for autofill. r=adw

Comment 5

[Sandor Molnar[:smolnar]]

Backed out changeset 299feb340bf2 (bug 1774305) for causing mochitest failures in browser_urlbar_telemetry_autofill

Backout link: https://hg.mozilla.org/integration/autoland/rev/2b49767efaac5d20a64ce78c1ca0e2899823530f

Push with failures

Failure log

TEST-UNEXPECTED-FAIL | browser/components/urlbar/tests/browser/browser_urlbar_telemetry_autofill.js | "undefined" == 1 - JS frame :: resource://testing-common/TelemetryTestUtils.jsm :: assertScalar :: line 23

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9ef9b0bfdfeb
Add impression telemetry for autofill. r=adw

Comment 7

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/9ef9b0bfdfeb

Comment 8

[Drew Willcoxon :adw]

STR for QA

There are three cases we should test for each kind of autofill. Before you start each case, please restart the browser so one case doesn't influence the next.

Notes:

• To verify telemetry, please visit about:telemetry#scalars-tab_search=urlbar.impression.autofill
• It will be helpful to keep the telemetry tab open during the STR so you can reload it to verify telemetry is updated correctly. You can use one tab for telemetry and another tab for triggering autofill and picking results.
• If you find that unexpected telemetry values appear, make sure you didn't accidentally hit enter or pick a result after another type of autofill happened.

Case 1: Origin autofill

1. Type moz and verify mozilla.org/ autofills
   a. If it doesn't autofill, visit https://www.mozilla.org/ a few times until it does, and then close the tab. Don't use autofill when visiting it in this step. You can copy-paste the full URL instead.
2. Press enter
3. Verify the following telemetry value appears:
   • urlbar.impression.autofill_origin: 1
4. Verify the following telemetry does not appear:
   • urlbar.impression.autofill_adaptive
   • urlbar.impression.autofill_url
5. Type moz and verify mozilla.org/ autofills
6. While the panel remains open, pick any result in the urlbar panel except the first one, e.g. a search suggestion
7. Verify the telemetry value is incremented:
   • urlbar.impression.autofill_origin: 2
8. Verify the following telemetry does not appear:
   • urlbar.impression.autofill_adaptive
   • urlbar.impression.autofill_url

Case 2: URL autofill

1. Type addons and verify addons.mozilla.org/ autofills
   a. If it doesn't autofill, visit https://addons.mozilla.org/firefox/addon/nightly-tester-tools/ a few times until it does, and then close the tab. Don't use autofill when visiting it in this step. You can copy-paste the full URL instead.
2. Move the caret to the end and type fire so that addons.mozilla.org/firefox/ autofills
3. Press enter
4. Verify the following telemetry value appears:
   • urlbar.impression.autofill_url: 1
5. Verify the following telemetry does not appear:
   • urlbar.impression.autofill_adaptive
   • urlbar.impression.autofill_origin
6. Repeat steps 1 and 2
7. While the panel remains open, pick any result in the urlbar panel except the first one, e.g. the history result
8. Verify the telemetry value is incremented:
   • urlbar.impression.autofill_url: 2
9. Verify the following telemetry does not appear:
   • urlbar.impression.autofill_adaptive
   • urlbar.impression.autofill_origin

Case 3: Adaptive history autofill

1. Set browser.urlbar.autoFill.adaptiveHistory.enabled to true
2. Type addons and verify addons.mozilla.org/ autofills
   a. If it doesn't autofill, visit https://addons.mozilla.org/firefox/addon/nightly-tester-tools/ a few times until it does, and then close the tab. Don't use autofill when visiting it in this step. You can copy-paste the full URL instead.
3. Verify the URL from step 2a appears as a history result and click it
4. Close the tab and open a new one
5. Type addons and verify the URL from step 2a autofills
6. Press enter
7. Verify the following telemetry value appears:
   • urlbar.impression.autofill_adaptive: 1
   • urlbar.impression.autofill_origin: 1 [Edit: Earlier version of STR incorrectly didn't include this]
8. Verify the following telemetry does not appear:
   • urlbar.impression.autofill_origin [Edit: This is wrong, it should appear]
   • urlbar.impression.autofill_url
9. Close the tab and open a new one
10. Type addons and verify the URL from step 2a autofills
11. While the panel remains open, pick any result in the urlbar panel except the first one, e.g. the history result
12. Verify the telemetry value is incremented:

• urlbar.impression.autofill_adaptive: 2
• urlbar.impression.autofill_origin: 1 [Edit: Earlier version of STR incorrectly didn't include this]

13. Verify the following telemetry does not appear:

• urlbar.impression.autofill_origin [Edit: This is wrong, it should appear]
• urlbar.impression.autofill_url

Comment 9

[Drew Willcoxon :adw]

I updated the STR just now for the adaptive history autofill case. They didn't properly take into account the fact that autofill_origin should also be recorded with a value of 1. Thanks again for filing bug 1776373.

Comment 10

[Cristian Baica [:cbaica], Release Desktop QA]

Thank you Drew for the updates and the clarification on this. Based on your comments I will be marking this as verified fixed. as that was the only ping left to be checked.