1801244 - Make Merino opt out on Beta

Status: RESOLVED WONTFIX

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

We want to make Merino opt out on Beta to replace the recently ended long-running experiment that did the same thing.

Comment 1

[Drew Willcoxon :adw]

Attachment: Bug 1801244 - Make Merino opt out on Beta.

This makes Merino opt out on Beta. There are two different parts:

1. Decide what "Beta" means. After discussing this with RyanVM, I'm using the
   same definition used by telemetry: (a) if MOZ_UPDATE_CHANNEL is "beta"
   or (b) if it's "release" and app.update.channel is "beta". The latter case
   is true for RC builds delivered to the Beta channel. I moved the relevant
   telemetry logic to AppConstants so I can easily use it too.

   Bug 1435753 has context on this logic w/r/t telemetry.

2. Change the Merino default so it's enabled on Beta in the offline scenario. I
   kept the online scenario's disabled default since the only point of
   delivering the online scenario to the user now is to prompt them to opt in to
   Merino.

   This change is in UrlbarPrefs.sys.mjs.

The rest of the patch updates tests.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/af2c13b9fdb7
Make Merino opt out on Beta. r=daisuke,nanj,RyanVM,chutten

Comment 3

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/af2c13b9fdb7

Comment 4

[Drew Willcoxon :adw]

For verification, please see this doc.

Comment 5

[Drew Willcoxon :adw]

Comment on attachment 9304005 [details]
Bug 1801244 - Make Merino opt out on Beta.

Beta/Release Uplift Approval Request

• User impact if declined: Merino will continue to remain opt in by default on Beta. We'd like this to be on Beta sooner rather than later so we can serve those users suggestions from Firefox Suggest.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please see the test doc
• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): This enables Merino by default on Beta. If the fix is buggy it might accidentally enable it on Nightly or Release. Enabling it on Release would violate our privacy policy. However, the patch is small, it was reviewed by several people including RyanVM, and it's very unlikely to have that problem. I've also written a QA doc to make sure it's thoroughly verified. Once this fix migrates to Release/RC we'll need to be careful to verify it again. I'll coordinate with QA to do that.
• String changes made/needed:
• Is Android affected?: No

Comment 6

[Dianna Smith [:diannaS]]

Comment on attachment 9304005 [details]
Bug 1801244 - Make Merino opt out on Beta.

Approved for 108.0b6
Approved for 108.0b8
Backed out in comment 12

Comment 7

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/0b01a51a35be

Comment 8

[Cosmin Muntean [:cmuntean], Ecosystem QA]

We have verified this issue on the latest Nightly 109.0a1 (Build ID: 20221122214324) on Windows 10 x64, macOS 12.4 and Ubuntu 20.04 x64.

• In order to verify this issue, we have followed the STR for Nightly testing provided in the test plan doc from comment 4. We haven't found any issues during testing.
• As soon as the bug is uplifted in Beta 108 and a build is available we will verify the STR for Beta 108.
• Also when a RC build is available we will test the STR for Release.

Comment 9

[Dianna Smith [:diannaS]]

As requested, we will backout 0b01a51a35be from Beta to avoid any issues during the holiday. We will re-Introduce this in 108.0b8.

Comment 10

[Donal Meehan [:dmeehan]]

Backed out changeset 0b01a51a35be (bug 1801244) back out for potential risk in beta.

Backout: https://hg.mozilla.org/releases/mozilla-beta/rev/57f3340621bf

Comment 11

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/917a129b9b92

Comment 12

[Dianna Smith [:diannaS]]

Backed out changeset 917a129b9b92 (Bug 1801244) back out for potential risk in beta as requested
Backout: https://hg.mozilla.org/releases/mozilla-beta/rev/af298bbf1b6d

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Thinking about this more, I think we need QA verification that this patch is working correctly RE: ensuring that Telemetry data is/isn't being sent in the correct scenarios. Because if we mess that up, it's likely an S1 incident :)

Comment 14

[Dianna Smith [:diannaS]]

Comment on attachment 9304005 [details]
Bug 1801244 - Make Merino opt out on Beta.

It was decided that this would not roll out in fx108.

Comment 15

[Drew Willcoxon :adw]

It's looking like we'll want to back out this bug entirely. We should know more soon. If we don't come to a firm decision before the next merge, we should back it out before the merge. I'll leave the NI on myself as a reminder.

Comment 16

[Drew Willcoxon :adw]

Yes, this should be backed out on 109/m-c too. Could you back it out please, Ryan? I was able to revert it cleanly locally.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Backed out.
https://hg.mozilla.org/integration/autoland/rev/9047b4db0385f5fccf23525c41889bb637ef3bc3

Comment 18

[Drew Willcoxon :adw]

Thanks everyone, sorry for the churn. Now that this has been backed out everywhere, I'll close it as wontfix. The latest discussion is that if we go forward with making Merino opt out on Beta, we should use another experiment or Nimbus rollout like we did before rather than landing a change in Firefox.