1796202 - The Firefox Suggest dataCollection/Merino pref is not properly recorded in environment telemetry

Status: VERIFIED FIXED

(Firefox :: Address Bar, defect, P1)

Description

[Drew Willcoxon :adw]

When the user is in an experiment that forces on Merino and they have not manually turned off Merino, the associated pref browser.urlbar.quicksuggest.dataCollection.enabled is improperly recorded as false in telemetry environment. This is because of a race condition on startup. On startup, the following things typically happen in order:

1. Telemetry environment caches initial pref values
2. Firefox Suggest sets the value of the pref to true
3. Telemetry environment starts observing pref changes

That means telemetry environment ends up caching and recording the wrong value.

This is the same problem as in bug 1731373. We fixed that bug by notifying telemetry environment when Firefox Suggest startup finishes, and then telemetry environment updates its pref cache.

When I added browser.urlbar.quicksuggest.dataCollection.enabled in bug 1735976, I should have added the pref to this startup notification path but I forgot.

Comment 1

[Drew Willcoxon :adw]

A better fix for this is to just have Suggest wait for TelemetryEnvironment initialization before updating prefs.

Comment 2

[Drew Willcoxon :adw]

Attachment: Bug 1796202 - Make Firefox Suggest wait for TelemetryEnvironment initialization before updating the scenario prefs.

This reverts the fix to bug 1731373 and instead makes Firefox Suggest wait for
TelemetryEnvironment to finish initializing before it starts to update its
preferences. That's what I should have done in the first place, no need for an
observer notification or for TelemetryEnvironment to have to know any details
about Suggest prefs.

This fixes the bug (bug 1796202) and should also avoid the problem for any other
Suggest prefs we add to TelemetryEnvironment in the future.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9e8698175fd4
Make Firefox Suggest wait for TelemetryEnvironment initialization before updating the scenario prefs. r=daisuke,chutten

Comment 4

[Drew Willcoxon :adw]

STR for QA are below. As time allows, please also do some basic smoke testing for triggering Suggest suggestions to make sure this fix didn't regress anything abvious.

1. Enroll in the treatment branch of https://experimenter.services.mozilla.com/nimbus/firefox-suggest-by-merino-beta so that Merino is enabled by default. You could also use any other experiment whose recipe sets "quickSuggestDataCollectionEnabled": true
2. Open about:config and verify that browser.urlbar.quicksuggest.dataCollection.enabled is true and not bold (i.e., it's true by default)
3. Open about:telemetry#environment-data-tab and verify userPrefs.browser.urlbar.quicksuggest.dataCollection.enabled is true
4. Restart Firefox
5. Repeat step 3. The value should still be true. If you restored your Firefox session, you may need to refresh about:telemetry#environment-data-tab before the value will be true.
6. Open about:preferences#privacy and turn off "Improve the Firefox Suggest experience" (third toggle switch)
7. Open/reload about:telemetry#environment-data-tab and verify userPrefs.browser.urlbar.quicksuggest.dataCollection.enabled is false
8. Restart
9. Repeat step 7. The value should still be false
10. Open about:preferences#privacy and turn on "Improve the Firefox Suggest experience"
11. Open/reload about:telemetry#environment-data-tab and verify userPrefs.browser.urlbar.quicksuggest.dataCollection.enabled is true
12. Restart
13. Repeat step 11. The value should still be true. If you restored your Firefox session, you may need to refresh about:telemetry#environment-data-tab before the value will be true.

Comment 5

[Drew Willcoxon :adw]

Comment on attachment 9299414 [details]
Bug 1796202 - Make Firefox Suggest wait for TelemetryEnvironment initialization before updating the scenario prefs.

Beta/Release Uplift Approval Request

• User impact if declined: This is necessary to properly record some Firefox Suggest telemetry for users enrolled in experiments.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please see comment 4
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a small fix that makes Firefox Suggest initialization wait for telemetry initialization. It has automated test coverage and with additional QA coverage we can be confident it's low risk.
• String changes made/needed:
• Is Android affected?: No

Comment 6

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/9e8698175fd4

Comment 7

[Cosmin Muntean [:cmuntean], Ecosystem QA]

We have verified this issue on the latest Nightly 108.0a1 build (Build ID: 20221021083918) on Windows 10 x64, macOS 12.4 and Ubuntu 20.04 x64.

• In order to verify this issue we have followed the STR described in comment 4.
• Besides the STR provided, we have also verified if the "browser.urlbar.quicksuggest.dataCollection.enabled" has the correct value after unenrolling from the experiment.
• We have also verified that the "browser.urlbar.quicksuggest.dataCollection.enabled" has the correct value while enrolled in the experiment,
• We have verified the impression/click pings in the browser console.
• We have verified that the Sponsored and Non-Sponsored results are correctly triggered.

Comment 8

[Donal Meehan [:dmeehan]]

Comment on attachment 9299414 [details]
Bug 1796202 - Make Firefox Suggest wait for TelemetryEnvironment initialization before updating the scenario prefs.

Approved for 107.0b4.

Comment 9

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/1f65f83c7e75

Comment 10

[Cosmin Muntean [:cmuntean], Ecosystem QA]

We have verified this issue on the latest Firefox Beta 107.0b4 on Windows 10 x64, macOS 12.4 and Ubuntu 20.04 x64.

• In order to verify this issue we have used the same scenarios as described in comment 7.