1740965 - Update Firefox Suggest online defaults and the behavior online opt-in modal now that there's a separate toggle for data collection

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

https://mozilla-hub.atlassian.net/browse/SNT-39

The way the online opt-in modal is currently implemented is that all three prefs (both suggestion types and data collection) are set according to whether the user opted in: https://searchfox.org/mozilla-central/rev/9b61a812836d63bd59a7977016f390052ac585ac/browser/components/urlbar/UrlbarQuickSuggest.jsm#273

This made sense when the Firefox Suggest prefs UI had only two checkboxes and it wasn't possible for the user to toggle data collection separately from suggestions. But now that there is a separate data-collection toggle, the modal should act only on the data-collection pref, or at least it should not disable the two suggestion types when the user does not opt in.

The Jira ticket says if the user opts in then all three toggles should be on, but I'd like to clarify with Natalie what should happen if the user had previously turned off one or both of the suggestions toggles. Should we override that previous choice and turn them on, or should we preserve the previous choice? I imagine it depends on the new design/wording of the modal.

Natalie says this does not block shipping the preferences redesign (bug 1735976) but it does block ramping up the online rollout. The online rollout is also blocked by the new modal design, which is not done yet.

Comment 1

[Drew Willcoxon :adw]

(In reply to Drew Willcoxon :adw from comment #0)

The Jira ticket says if the user opts in then all three toggles should be on, but I'd like to clarify with Natalie what should happen if the user had previously turned off one or both of the suggestions toggles. Should we override that previous choice and turn them on, or should we preserve the previous choice? I imagine it depends on the new design/wording of the modal.

I talked with Natalie today and the plan right now is for the modal to act only on the data collection pref. If the user opts in, enable data collection; otherwise leave it false, the default value. Either way, opting in or not opting in won't change whether suggestions are enabled.

Natalie updated the Jira during our meeting but it still talks about the two suggestion toggles remaining/being on when the user both opts in and doesn't opt in. Based on our conversation at the time I think that's assuming the user didn't change the offline defaults where suggestions are enabled by default.

Something this made me realize and that we talked about too is that the defaults for online aren't right anymore. Currently suggestions default to disabled in online, but they should default to enabled just like they do in offline. So we should fix that in this bug, in addition to fixing the prefs that are set when the modal is opted in to.

Comment 2

[Drew Willcoxon :adw]

(In reply to Drew Willcoxon :adw from comment #1)

Natalie updated the Jira during our meeting but it still talks about the two suggestion toggles remaining/being on when the user both opts in and doesn't opt in. Based on our conversation at the time I think that's assuming the user didn't change the offline defaults where suggestions are enabled by default.

Ah that's covered by the fourth bullet:

If a user disables sponsored and/or suggestions from the web but clicks on “allow”, only the third toggle will be turned on

I thought Natalie had updated the Jira to address that case, and yep. I had the ticket open in two tabs and was looking at the older version.

Comment 3

[Drew Willcoxon :adw]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=872ead6511739dfaec0fb877db05188c9291d5b8

Comment 4

[Drew Willcoxon :adw]

Attachment: Bug 1740965 - Part 1: Enable Firefox Suggest suggestions by default in the online scenario.

This enables suggestions by default in the online scenario. Data collection is
still disabled by default. With this revision, there's now no difference between
online and offline in terms of defaults.

Part 2 of this bug will modify the opt-in modal so that opting in or out affects
only the data-collection pref and not suggestions like it currently does.

For users who were already enrolled in online and who did not opt in or
otherwise enable suggestions, suggestions will remain disabled when they
upgrade. Otherwise they would go from not seeing suggestions to seeing
suggestions, which wouldn't be right. Users in online who did enable suggestions
will continue to see them when they upgrade.

For all other users, suggestions will default to enabled when/if they are
enrolled in online.

I added another prefs migration to implement this. The case where the user is
upgrading from the unversioned prefs (Firefox 94.0.1 and earlier) to the new
version is tricky because unless the user was shown the opt-in modal, we can't
tell whether they were in online or not just by looking at the prefs. So in
order to tell whether the user was in online in that case, I check whether they
were enrolled in the firefox-suggest-offline-vs-online treatment branch.

I wrote a test plan doc to make sure all relevant upgrade paths work correctly
and I've verified it myself. I also added new automated tests of course.

Comment 5

[Drew Willcoxon :adw]

Attachment: Bug 1740965 - Part 2: Update the Firefox Suggest online opt-in modal dialog so it affects only data collection, not suggestions.

This modifies the opt-in modal so that opting in or out affects only the
data-collection pref and not suggestions like it currently does.

However, the modal content redesign isn't done yet. I'll need to check with
Natalie whether it's OK to land this separately from the content redesign or
not. We may need to wait until that's ready so we can update the content at the
same time. There's some risk that users who are currently enrolled in online
haven't seen the modal yet. If those users see the dialog after this lands but
with the old content, it will appear to them that they're opting in/out of
suggestions and not only data collection.

Depends on D131751

Comment 6

[Drew Willcoxon :adw]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d79d424b42bc40f588c09de0e94314a7c79115b

Comment 7

[Drew Willcoxon :adw]

[Tracking Requested - why for this release]:
This is necessary for the next Firefox Suggest rollout in 96.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/00415755847e
Part 1: Enable Firefox Suggest suggestions by default in the online scenario. r=nanj
https://hg.mozilla.org/integration/autoland/rev/a9c0c089be08
Part 2: Update the Firefox Suggest online opt-in modal dialog so it affects only data collection, not suggestions. r=nanj

Comment 9

[Drew Willcoxon :adw]

For QA verification, please see the testing doc.

Comment 10

[Drew Willcoxon :adw]

Comment on attachment 9251823 [details]
Bug 1740965 - Part 1: Enable Firefox Suggest suggestions by default in the online scenario.

Beta/Release Uplift Approval Request

• User impact if declined: This is necessary for the next Firefox Suggest rollout in 96.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please see the testing doc
• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): This is a large patch but it only affects the Firefox Suggest "online scenario" experience, which currently is enabled for 1% of U.S. users via a Nimbus experiment, with plans to roll out to more users in 96. It will only affect newly enrolled users, and current users will not see any changes at all (with one minor exception for a cohort of ~1200 users). Of course, especially since this is a large patch, there may be bugs that unintentionally affect other users, but the patch comes with comprehensive tests and we have extensive manual QA test cases too.
• String changes made/needed:

Comment 11

[Drew Willcoxon :adw]

Beta try pushes:

https://treeherder.mozilla.org/jobs?repo=try&revision=c7bd1ecda0a24a2821c39903a20f4382f4778480
https://treeherder.mozilla.org/jobs?repo=try&revision=fcef87750de4b03f83b480b77e5314d63c1893ea

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/00415755847e
https://hg.mozilla.org/mozilla-central/rev/a9c0c089be08

Comment 13

[Dianna Smith [:diannaS]]

Comment on attachment 9251823 [details]
Bug 1740965 - Part 1: Enable Firefox Suggest suggestions by default in the online scenario.

Approved for 96.0b6

Comment 14

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/1dd310bc71ec
https://hg.mozilla.org/releases/mozilla-beta/rev/e976a8a8e2f8

Comment 15

[Carmen Fat [:cfat] - Ecosystem QA]

• We have finished verifying these changes on the latest Nightly 97.0a1 build (Build ID: 20211216094142) on Windows 10 x64, macOS 10.15.7 and Ubuntu 20.04 x64.

• In order to verify this bug we have followed the scenarios: Gdoc.

• During testing, we have encountered the following 2 issues on Ubuntu 20.04 x64 on a slower machine, which are tracked here: Bug 1746214 and Bug 1746459.

Comment 16

[Carmen Fat [:cfat] - Ecosystem QA]

• We have finished verifying these changes on the latest Firefox Beta 96.0b7 (Build ID: 20211219185855) on Windows 10 x64, macOS 10.15.7 and Ubuntu 20.04 x64.

• In order to verify this bug we have followed the scenarios: Gdoc.

• Besides the known issues (Bug 1746214 and Bug 1746459) we didn't encounter any new ones.