1745026 - Implement Firefox Suggest online opt-in modal redesign

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

https://mozilla-hub.atlassian.net/browse/SNT-46

This will need to be based on the patches in bug 1740965.

Comment 1

[Drew Willcoxon :adw]

[Tracking Requested - why for this release]:
This is necessary for the next Firefox Suggest rollout in 96.

Comment 2

[Daisuke Akatsuka (:daisuke)]

Attachment: Bug 1745026 - Part 1: Implement basic opt-in modal design for Firefox Suggest.

Comment 3

[Drew Willcoxon :adw]

Attachment: data-request.md

Requesting data review on the changes to the existing contextservices.quicksuggest.opt_in_dialog telemetry event

Comment 4

[Drew Willcoxon :adw]

Comment on attachment 9255302 [details]
data-request.md

Looks like Teon is on leave, switching data-review request to Megan:

Requesting data review on the changes to the existing contextservices.quicksuggest.opt_in_dialog telemetry event

Comment 5

[Daisuke Akatsuka (:daisuke)]

Attachment: Bug 1745026 - Part 2: Update messages and UI by Nimbus.

Depends on D133467

Comment 6

[Megan McCorquodale]

Comment on attachment 9255302 [details]
data-request.md

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   This will be documented in the probe dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Drew Willcoxon will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, Interaction Data.

5. Is the data collection request for default-on or default-off?
   Default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data?
   No.

9. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 7

[Daisuke Akatsuka (:daisuke)]

Attachment: Bug 1745026 - Part 3: Update telemetries for Firefox suggest onboarding dialog.

Depends on D133993

Comment 8

[Drew Willcoxon :adw]

Attachment: data-request.md v2

Megan, you approved a previous data review request for this bug, but we ended up changing the values recorded in the telemetry, so I'm asking for review again. This new request includes modifications to two probes, a telemetry event and a preference recorded in telemetry environment data. Thanks!

Comment 9

[Megan McCorquodale]

Comment on attachment 9257774 [details]
data-request.md v2

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   Yes, this will be documented in the probe dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Drew Willcoxon will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, Interaction data.

5. Is the data collection request for default-on or default-off?
   Default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data?
   No.

9. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 10

[Drew Willcoxon :adw]

Attachment: Bug 1745026 - Part 4: Hook into the upgrade- and default-browser-dialog logic when determining whether to show the Firefox Suggest opt-in modal.

Currently we check in a startup idle task whether we should show the Firefox
Suggest online modal, which means the modal competes with the upgrade dialog and
default-browser prompt. This revision moves the check to the same site where we
check for the other two dialogs. So now the sequence is:

1. Show the upgrade dialog? If yes, show it and stop. If no, continue.
2. Show the default-browser prompt? If yes, show it and stop. If no, continue.
3. Show the Suggest modal if necessary.

Only one of these dialogs will be shown per Firefox session.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/25dc0db77ea2
Part 1: Implement basic opt-in modal design for Firefox Suggest. r=adw
https://hg.mozilla.org/integration/autoland/rev/dc472d8ed5a0
Part 2: Update messages and UI by Nimbus. r=adw,flod
https://hg.mozilla.org/integration/autoland/rev/a6abdf8c9bc4
Part 3: Update telemetries for Firefox suggest onboarding dialog. r=adw
https://hg.mozilla.org/integration/autoland/rev/cbee9aaa7d7b
Part 4: Hook into the upgrade- and default-browser-dialog logic when determining whether to show the Firefox Suggest opt-in modal. r=nanj

Comment 12

[Drew Willcoxon :adw]

For QA verification, please see the QA doc.

Comment 13

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/25dc0db77ea2
https://hg.mozilla.org/mozilla-central/rev/dc472d8ed5a0
https://hg.mozilla.org/mozilla-central/rev/a6abdf8c9bc4
https://hg.mozilla.org/mozilla-central/rev/cbee9aaa7d7b

Comment 14

[Drew Willcoxon :adw]

Comment on attachment 9254715 [details]
Bug 1745026 - Part 1: Implement basic opt-in modal design for Firefox Suggest.

Beta/Release Uplift Approval Request

• User impact if declined: This is required for the Firefox Suggest experiment on a 96 dot release.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please see the testing doc
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is only a redesign of the Firefox Suggest online opt-in modal and doesn't affect anything else. Has tests and QA coverage.
• String changes made/needed:

Comment 15

[Carmen Fat [:cfat] - Ecosystem QA]

• We have verified the implementation of the new Opt-in modal using the latest Nightly 97.0a1 (Build ID: 20220109215539) build, on Windows 10 x64, macOS 10.15.7, and Linux Ubuntu 20.04 x64.
• In order to verify the implementation, we followed the guidelines from the testing document.

However, we have 1 question about the behavior of the modal and 1 concern for future experiments:

1. If you have the opt-in modal displayed and you close the browser on macOS by clicking the browser "X" button, or right click -> Quit from Docker/Taskbar (all OSes), the browser is closed, but the Modal is re-displayed after reopening the browser. Is this the expected behavior?

2. The new prefs added after enrolling in the experiment and interacting with the modal, they remain set after unenrolling from the experiment. Is this something that can affect future experiments?

Comment 16

[Drew Willcoxon :adw]

Thanks Carmen.

(In reply to Carmen Fat [:cfat] - Ecosystem QA from comment #15)

1. If you have the opt-in modal displayed and you close the browser on macOS by clicking the browser "X" button, or right click -> Quit from Docker/Taskbar (all OSes), the browser is closed, but the Modal is re-displayed after reopening the browser. Is this the expected behavior?

That's not the expected behavior. Could you file a bug please? I asked Natalie in the Slack thread if this is a blocker and she doesn't think so, and I agree. I think it will end up being wontfix because: (1) It affects all modals in Firefox AFAICT, not only this one, (2) it's a bit of an edge case, (3) it's not a terrible outcome since the user did not finish their interaction with the modal.

2. The new prefs added after enrolling in the experiment and interacting with the modal, they remain set after unenrolling from the experiment. Is this something that can affect future experiments?

Are you referring to browser.urlbar.quicksuggest.onboardingDialogChoice and browser.urlbar.quicksuggest.onboardingDialogVersion? We definitely want those prefs to stick around after the experiment ends. Otherwise Firefox would not be able to identify users who have already seen the modal, which version of the modal they saw, and whether they opted in/out. Any future experiments will take them into account as appropriate.

Comment 17

[Carmen Fat [:cfat] - Ecosystem QA]

We have verified the implementation of the new Opt-in modal using the latest Beta 97.0b2 (Build ID: 20220111185943) build, on Windows 10 x64, macOS 10.15.7, and Linux Ubuntu 20.04 x64.
In order to verify the implementation, we followed the guidelines from the testing document.

During testing we have found 4 new issues that we logged separately: Bug 1749692, Bug 1749740, Bug 1749741 and Bug 1749742.
Beside these issues we have encountered another intermittent behavior:

• After accepting the modal, we went to the "about:config" page and manually flipped the browser.urlbar.quicksuggest.dataCollection.enabled preference to the false value. However, the value did not remain set to false at restart; instead, it went back to true automatically.

• We also tried to reproduce this issue using the 3rd toggle from the “about:preferences" page, but we didn't manage to reproduce the issue from the UI settings.

• Also, this only happens rarely, but we wanted to raise awareness about this since the behavior interferes with users' choice about collecting telemetry.

• We also managed to capture part of the behavior in this screencast.

I will leave the "qe-verify" flag set, to re-verify this issue after it will be uplifted in Firefox 96 dot release.

Comment 18

[Dianna Smith [:diannaS]]

Comment on attachment 9254715 [details]
Bug 1745026 - Part 1: Implement basic opt-in modal design for Firefox Suggest.

Approved for 96.0.2

Comment 19

[Dianna Smith [:diannaS]]

Comment on attachment 9256304 [details]
Bug 1745026 - Part 3: Update telemetries for Firefox suggest onboarding dialog.

Approved for 96.0.2

Comment 20

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-release/rev/882024bac13e
https://hg.mozilla.org/releases/mozilla-release/rev/8e104fa2eaac
https://hg.mozilla.org/releases/mozilla-release/rev/983e7bb5032c
https://hg.mozilla.org/releases/mozilla-release/rev/1134f309f55f

Comment 21

[Carmen Fat [:cfat] - Ecosystem QA]

• We have verified the implementation of the new Opt-in modal using the Firefox RC 96.0.2 (Build ID: 20220119040736) build, on Windows 10 x64, macOS 10.15.7, and Linux Ubuntu 20.04 x64.
• In order to verify the implementation, we followed the guidelines from the testing document.
• We will also send a sign-off today for the upcoming Firefox Suggest experiment that is based on this modal.