1795803 - Expose the Merino provider pref to Nimbus

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

https://mozilla-hub.atlassian.net/browse/SNT-320

Comment 1

[Drew Willcoxon :adw]

Attachment: Bug 1795803 - Expose the Merino provider pref to Nimbus.

This adds a Nimbus variable corresponding to browser.urlbar.merino.providers.

I noticed that when we added a Nimbus variable for client variants (bug
1743685), we didn't actually use it in the provider. Instead we use its fallback
pref. I fixed that too.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/09b1ee379354
Expose the Merino provider pref to Nimbus. r=daisuke

Comment 3

[Drew Willcoxon :adw]

For QA verification: This bug only exposes some hidden preferences to Nimbus, so there's not an easy way to verify it. We don't yet have an experiment that uses these new variables. So instead I'd like to ask you to do a quick smoke test to make sure this change did not break the Firefox Suggest feature, especially after you opt-in to Merino (by turning on the third toggle switch in the Suggest preferences in about:preferences#privacy, "Improve the Firefox Suggest experience"). Thanks!

Edit: STR to test Merino:

1. Set browser.urlbar.quicksuggest.dataCollection.enabled to true (corresponds to the third toggle switch in about:preferences)
2. Set browser.urlbar.quicksuggest.remoteSettings.enabled to false to disable suggestions from remote settings
3. Set browser.urlbar.merino.enabled to true (if it's not already)
4. Type "amazon" in the urlbar
5. Verify a sponsored Amazon suggestion appears
6. Restart Firefox
7. Repeat steps 4-5

If you have time, please also do some other basic smoke testing as time allows.

Comment 4

[Drew Willcoxon :adw]

Comment on attachment 9298937 [details]
Bug 1795803 - Expose the Merino provider pref to Nimbus.

Beta/Release Uplift Approval Request

• User impact if declined: This patch is necessary for experiments.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please see comment 3
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This only exposes a hidden preference to Nimbus and modifies the relevant code to use the Nimbus variables instead of the prefs.
• String changes made/needed:
• Is Android affected?: No

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/09b1ee379354

Comment 6

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have verified this issue on the latest Nightly 108.0a1 (Build ID: 20221018200231) on Windows 10 x64, macOS 12.4 and Linux Mint 20 x64.

• In order to verify this issue I have used the STR described in comment 3.

Besides these I have tested the following scenarios:

• Verified that the Sponsored results are correctly triggered after a browser update.
• Verified that the Sponsored results are triggered if changing the Merino endpoint to the MerinoPy (https://merinopy.services.mozilla.com/api/v1/suggest).
• Verified that the Sponsored results are triggered if Top Pick is enabled.
• I have verified that the impression/click pings are registered.

@Drew please let me know if there are other important scenarios that we need to cover.
Also, we could create an experiment on the stage environment with the new variables. If you can help us with instructions about how to add and use these variables we could create an experiment on stage and test them.

Comment 7

[Donal Meehan [:dmeehan]]

Comment on attachment 9298937 [details]
Bug 1795803 - Expose the Merino provider pref to Nimbus.

Approved for 107.0b3.

Comment 8

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/e433b14352f7

Comment 9

[Drew Willcoxon :adw]

Thanks Cosmin, I appreciate it! I think you covered more than I was expecting, thanks. The Nimbus variables are:

• merinoProviders -- This is the new one added in this bug. It should be set to a string of Merino provider names separated by commas, for example: adm or adm,accuweather. Currently the only Merino provider is adm, so you can't really do a full end-to-end test with multiple providers. (Merino-py does support accuweather, so you could use it. It's not so important though, see next.) However, you can set the names to anything, and the important thing is that Firefox should include them in the request URL it sends to Merino with this format at the end of the URL: ?q=<query>&providers=<providers> (the ordering of q and providers isn't important). You can use the browser toolbox's network tab to see the URLs Firefox sends to Merino.
• merinoClientVariants -- Like the providers, this should be a comma-separated list of strings, like variant1 or variant1,variant2. Merino and Merino-py don't currently support any client variants as far as I know, but similar to providers, the important thing is the URL. For this, the format at the end of the URL is: ?q=<query>&client_variants=<variants>.

Of course if your experiment includes both providers and client variants, then the URL will contain both, like: ?q=<query>&providers=<providers>&client_variants=<variants> (and the order doesn't matter). Also there will be other parameters at the end of the URL like seq and sid, that's expected.

Comment 10

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Thank you Drew for these details! I have created a recipe on the stage environment, you can find it here: Experimenter page. I have created different branches using the Nimbus variables.
Could you please take a look over the Experimenter page, if everything is correctly set?

I have already started to verify the experiment with force enrollment and I get the URLs as you described, but I will also try with natural enrollment.

Comment 11

[Drew Willcoxon :adw]

Looks good, Cosmin, thanks.

Comment 12

[Cosmin Muntean [:cmuntean], Ecosystem QA]

We have verified this issue on the latest Beta 107.0b3 build (Build ID: 20221020202724) on Windows 10 x64, macOS 12.4 and Ubuntu 20.04 x64.

• In order to verify this issue we have used the same scenarios as in comment 6.