1780188 - [Experiment] The Autofill telemetry impressions are not recorded using the 103 Release Candidate build

Status: VERIFIED FIXED

(Firefox :: Address Bar, defect, P1)

Description

[Carmen Fat [:cfat] - Ecosystem QA]

[Notes]:

• This issue is reproducible only using the Release Candidate build; on Nightly 104.0a1 and Beta 103.0b9 builds the autofill impressions are correctly triggered and displayed in the about:telemetry page.

[Affected versions]:

• Firefox RC 103, Build ID: 20220718155818

[Affected platforms]:

• Windows 10 x64
• macOS 12.4
• Ubuntu 20.04 x64

[Prerequisites]:

• Have Firefox RC 103 (en* build) on your computer.
• Have the following user.js downloaded on your computer.
• Have the browser.search.region pref set to US.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to the Profile Folder from about:support page and paste the user.js file from the prerequisites.
3. Restart the browser and open a new tab.
4. Type moz characters in the Address Bar.
5. After the mozilla.org/ domain autofills, press the Enter key
6. Navigate to about:telemetry in the new tab.
7. Search for urlbar.impression.autofill_origin scalar.

[Expected result]:

• The urlbar.impression.autofill_origin scalar is displayed with 1 as value.

[Actual result]:

• The urlbar.impression.autofill_origin scalar is not displayed.

[Additional notes]:

• This issue is also reproducible with the following impressions:
• urlbar.impression.autofill_adaptive
• urlbar.impression.autofill_url
• urlbar.impression.autofill_about
• I have also looked over the Probe Dictionary and I have noticed that the autofill impressions are recorded only in Nightly and Beta starting with 103 version. Is it intended not having them on the Release builds?
• Here is a screen recording with the issue.

Comment 1

[Drew Willcoxon :adw]

(In reply to Carmen Fat [:cfat] - Ecosystem QA from comment #0)

• I have also looked over the Probe Dictionary and I have noticed that the autofill impressions are recorded only in Nightly and Beta starting with 103 version. Is it intended not having them on the Release builds?

That's the problem, they're incorrectly marked release_channel_collection: opt-in in Scalars.yaml. Thanks for catching this, Carmen.

Comment 2

[Drew Willcoxon :adw]

[Tracking Requested - why for this release]: This telemetry is needed for a Suggest User Value experiment targeting Release 103. We incorrectly marked it as opt-in on Release. It should be opt-out.

Comment 3

[Drew Willcoxon :adw]

Attachment: data-review-request-for-bug1780188.md

This request is the same as the previously approved request in bug 1774305 except that here we are changing the probes from release opt-in to release opt-out. The probes in question are uint scalars. The original data request is here: https://bug1774305.bmoattachments.org/attachment.cgi?id=9281577

Comment 4

[Drew Willcoxon :adw]

Attachment: Bug 1780188 - Make the autofill impression telemetry added in bug 1774305 opt out instead of opt in.

Comment 5

[Chris H-C :chutten]

Comment on attachment 9286124 [details]
data-review-request-for-bug1780188.md

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :adw and the Firefox Suggest User Value Team are responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1d32ccff62b0
Make the autofill impression telemetry added in bug 1774305 opt out instead of opt in. r=cbellini

Comment 7

[Drew Willcoxon :adw]

Comment on attachment 9286125 [details]
Bug 1780188 - Make the autofill impression telemetry added in bug 1774305 opt out instead of opt in.

Beta/Release Uplift Approval Request

• User impact if declined: This telemetry is needed for a Suggest User Value experiment targeting Release 103.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Should be clear from the bug
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch only changes some uint scalar telemetry from release-opt-in to release-opt-out. It only modifies the Scalars.yaml file and doesn't touch any other code.
• String changes made/needed:
• Is Android affected?: No

Comment 8

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/1d32ccff62b0

Comment 9

[Ryan VanderMeulen [:RyanVM]]

(In reply to Drew Willcoxon :adw (Away Jul 25–Aug 5) from comment #7)

• Is this code covered by automated tests?: Yes

Why didn't tests catch this?

Comment 10

[Donal Meehan [:dmeehan]]

Comment on attachment 9286125 [details]
Bug 1780188 - Make the autofill impression telemetry added in bug 1774305 opt out instead of opt in.

We are out of the beta cycle and mozilla-beta was merged to mozilla-release. Morphing this into a release uplift request.

Comment 11

[Drew Willcoxon :adw]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)

Why didn't tests catch this?

The test sets Services.telemetry.canRecordExtended = true. canRecordExtended is false on Release, so if the test had not done that, there would have been failures on Release and we would have caught it. (There was discussion in the Phabricator revision about canRecordExtended being required to fix some failures on try, but I don't think any of the builds on try would have disabled extended recording, so it doesn't seem like the same problem. [Edit: Looks like the ASAN and/or TSAN builds may disable it, so it is probably the same problem])

A number of urlbar tests and other front-end tests do this, and I'm not sure what we should do about it yet, but I filed bug 1780482 for it.

Comment 12

[Donal Meehan [:dmeehan]]

Comment on attachment 9286125 [details]
Bug 1780188 - Make the autofill impression telemetry added in bug 1774305 opt out instead of opt in.

Approved for 103.0.1, thanks.

Comment 13

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-release/rev/0c35eeece9a9

Comment 14

[Carmen Fat [:cfat] - Ecosystem QA]

I have verified this issue on the Firefox dot release 103.0.1 (Build ID: 20220729222726) on Windows 10 x64, macOS 12.4 and Linux Ubuntu 20.04 x64.

• All 4 autofill impressions are now displayed in the Scalars section of the "about:telemetry" page:
  urlbar.impression.autofill_adaptive
  urlbar.impression.autofill_url
  urlbar.impression.autofill_origin
  urlbar.impression.autofill_about

• I have also verified that there is no regression on Firefox Beta 104.0b4 (Build ID: 20220731190208) related to the autofill impressions and that they are correctly displayed on this channel, as well.