Closed Bug 1801244 Opened 2 months ago Closed 2 months ago

Make Merino opt out on Beta

Categories

(Firefox :: Address Bar, task, P1)

task

Tracking

()

RESOLVED WONTFIX

People

(Reporter: adw, Assigned: adw)

Details

Attachments

(1 obsolete file)

We want to make Merino opt out on Beta to replace the recently ended long-running experiment that did the same thing.

This makes Merino opt out on Beta. There are two different parts:

  1. Decide what "Beta" means. After discussing this with RyanVM, I'm using the
    same definition used by telemetry: (a) if MOZ_UPDATE_CHANNEL is "beta"
    or (b) if it's "release" and app.update.channel is "beta". The latter case
    is true for RC builds delivered to the Beta channel. I moved the relevant
    telemetry logic to AppConstants so I can easily use it too.

    Bug 1435753 has context on this logic w/r/t telemetry.

  2. Change the Merino default so it's enabled on Beta in the offline scenario. I
    kept the online scenario's disabled default since the only point of
    delivering the online scenario to the user now is to prompt them to opt in to
    Merino.

    This change is in UrlbarPrefs.sys.mjs.

The rest of the patch updates tests.

Pushed by dwillcoxon@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/af2c13b9fdb7
Make Merino opt out on Beta. r=daisuke,nanj,RyanVM,chutten
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → 109 Branch

For verification, please see this doc.

Flags: qe-verify+
Flags: in-testsuite+

Comment on attachment 9304005 [details]
Bug 1801244 - Make Merino opt out on Beta.

Beta/Release Uplift Approval Request

  • User impact if declined: Merino will continue to remain opt in by default on Beta. We'd like this to be on Beta sooner rather than later so we can serve those users suggestions from Firefox Suggest.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: Please see the test doc
  • List of other uplifts needed: None
  • Risk to taking this patch: Medium
  • Why is the change risky/not risky? (and alternatives if risky): This enables Merino by default on Beta. If the fix is buggy it might accidentally enable it on Nightly or Release. Enabling it on Release would violate our privacy policy. However, the patch is small, it was reviewed by several people including RyanVM, and it's very unlikely to have that problem. I've also written a QA doc to make sure it's thoroughly verified. Once this fix migrates to Release/RC we'll need to be careful to verify it again. I'll coordinate with QA to do that.
  • String changes made/needed:
  • Is Android affected?: No
Attachment #9304005 - Flags: approval-mozilla-beta?
QA Whiteboard: [qa-triaged]

Comment on attachment 9304005 [details]
Bug 1801244 - Make Merino opt out on Beta.

Approved for 108.0b6
Approved for 108.0b8
Backed out in comment 12

Attachment #9304005 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

We have verified this issue on the latest Nightly 109.0a1 (Build ID: 20221122214324) on Windows 10 x64, macOS 12.4 and Ubuntu 20.04 x64.

  • In order to verify this issue, we have followed the STR for Nightly testing provided in the test plan doc from comment 4. We haven't found any issues during testing.
  • As soon as the bug is uplifted in Beta 108 and a build is available we will verify the STR for Beta 108.
  • Also when a RC build is available we will test the STR for Release.

As requested, we will backout 0b01a51a35be from Beta to avoid any issues during the holiday. We will re-Introduce this in 108.0b8.

Backed out changeset 0b01a51a35be (bug 1801244) back out for potential risk in beta.

Backout: https://hg.mozilla.org/releases/mozilla-beta/rev/57f3340621bf

Backed out changeset 917a129b9b92 (Bug 1801244) back out for potential risk in beta as requested
Backout: https://hg.mozilla.org/releases/mozilla-beta/rev/af298bbf1b6d

Attachment #9304005 - Flags: approval-mozilla-beta+ → approval-mozilla-beta?

Thinking about this more, I think we need QA verification that this patch is working correctly RE: ensuring that Telemetry data is/isn't being sent in the correct scenarios. Because if we mess that up, it's likely an S1 incident :)

Flags: needinfo?(adw)

Comment on attachment 9304005 [details]
Bug 1801244 - Make Merino opt out on Beta.

It was decided that this would not roll out in fx108.

Attachment #9304005 - Flags: approval-mozilla-beta? → approval-mozilla-beta-

It's looking like we'll want to back out this bug entirely. We should know more soon. If we don't come to a firm decision before the next merge, we should back it out before the merge. I'll leave the NI on myself as a reminder.

Yes, this should be backed out on 109/m-c too. Could you back it out please, Ryan? I was able to revert it cleanly locally.

Flags: needinfo?(adw) → needinfo?(ryanvm)
Status: RESOLVED → REOPENED
Flags: needinfo?(ryanvm)
Resolution: FIXED → ---
Target Milestone: 109 Branch → ---
Status: REOPENED → NEW

Thanks everyone, sorry for the churn. Now that this has been backed out everywhere, I'll close it as wontfix. The latest discussion is that if we go forward with making Merino opt out on Beta, we should use another experiment or Nimbus rollout like we did before rather than landing a change in Firefox.

Status: NEW → RESOLVED
Closed: 2 months ago2 months ago
Resolution: --- → WONTFIX
Attachment #9304005 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.