1900486 - Review Checker Blank Screen after filling out Satisfaction Microsurvey and Disabling Feature

Status: RESOLVED FIXED

(Firefox :: Shopping, defect, P1)

Description

[jchen]

Attachment: Screenshot 2024-06-03 at 3.23.39 PM.png

STEPS TO REPLICATE

1. You will need to use Review Checker by going to an Amazon, Best Buy, or Walmart PDP
2. Opt in when you see the opt-in screen by hitting "Yes, try it"
3. You'll need to fill out the Satisfaction microsurvey, which is triggered by being at least on your 2nd session and having already visited 5 PDPs that have available Fakespot Reviews Analyses on them. You can also simulate this experience by manually setting these preferences in about:config while testing:

browser.shopping.experience2023.survey.enabled must be true (experiment does this)
browser.shopping.experience2023.survey.optedInTime must be 24 hours ago (try 1696550403)
browser.shopping.experience2023.survey.hasSeen must be false
browser.shopping.experience2023.survey.pdpVisits must be 5
browser.shopping.experience2023.optedIn must be 1

4. Complete the Satisfaction Microsurvey - you should be asked 2 questions, then you'll see a green confirmation message.
5. Hit "Turn off Review Checker" to disable the feature (at any point after having filled out the survey)
6. Navigate to another PDP
7. Manually turn on Review Checker again from the Shopping Icon

EXPECTED:

• Review Checker should work as expected, you should be able to see an opt-in screen telling you about using the Review Checker

ACTUAL RESULTS

• See a blank Review Checker sidebar with only "Thank you for the feedback" in green, or if you dismissed that, then you should only see a blank screen in that sidebar where there normally would be content, with Review Checker as a title

ADDITIONAL NOTES

• The blank screen doesn't seem to happen if you hit X, the Shopping icon, or turn off Auto-Open to temporarily close out the Review Checker. It seems to only happen after you fill out the Microsurvey, and, at any point after filling out the Microsurvey, disable the Review Checker feature by clicking "Turn off Review Checker"

Comment 1

[Shane Hughes [:aminomancer]]

You actually don't have to fill out the survey. Just seeing the survey is enough. The problem is the survey is showing instead of the opt-in message, which is supposed to show because you're not opted in. We need to prevent the survey showing if the user is not opted in.

Comment 2

[Shane Hughes [:aminomancer]]

Attachment: Bug 1900486 - Re-render Review Checker messaging content when opt-in state changes. r=jprickett

Comment 3

[:Gijs (he/him)]

Per the conversation on phabricator, this doesn't need to block because we can roll out without a survey to 80%, then fix this post-initial-rollout and then run a survey in the held-back remainder.

Comment 4

[Shane Hughes [:aminomancer]]

[Tracking Requested - why for this release]: To make our US Review Checker rollout schedule, this fix needs to land in 128.0. Without it, we can't show an important survey to Review Checker users. The rollout will be holding at 80% enrollment so we can launch an extra holdback experiment that deploys the survey. So how long it takes the fix to ship to release users determines the length of time 20% of US users will go without Review Checker. The sooner this fix ships, the sooner we can launch the holdback experiment, the sooner we can ramp from 80% rollout to 100% rollout.

Comment 5

[Donal Meehan [:dmeehan]]

Tracking as requested, though note we are in the final week of beta for Fx128.
The last beta for Fx128 builds on 2024-06-28, and Fx128 goes to RC on 2024-07-01

Comment 6

[Shane Hughes [:aminomancer]]

Attachment: Bug 1900486 - Re-render Review Checker messaging content when opt-in state changes. r=jprickett

Original Revision: https://phabricator.services.mozilla.com/D212498

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4ccbc99fdca7
Re-render Review Checker messaging content when opt-in state changes. r=omc-reviewers,pdahiya

Comment 8

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: To make our US Review Checker rollout schedule, this fix needs to land in 128.0. Without it, we can't show an important survey to Review Checker users. The rollout will be holding at 80% enrollment so we can launch an extra holdback experiment that deploys the survey. So how long it takes the fix to ship to release users determines the length of time 20% of US users will go without Review Checker. The sooner this fix ships, the sooner we can launch the holdback experiment, the sooner we can ramp from 80% rollout to 100% rollout.
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: In original revision
• Risk associated with taking this patch: Low
• Explanation of risk level: Behind a Nimbus experiment, and relatively low proportion of users can make it to the survey
• String changes made/needed: n/a
• Is Android affected?: no

Comment 9

[Shane Hughes [:aminomancer]]

The survey behaves oddly in verify mode on debug builds so I'm disabling the test in those conditions for now, due to the urgency. I'll continue looking into ways to make the test more stable and hopefully we can lift the restriction later on after the feature itself lands.

Comment 10

[Serban Stanca [:SerbanS]]

Backed out as requested by dev for causing a failure that went unnoticed.

• Backout link

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cba0e55d9442
Re-render Review Checker messaging content when opt-in state changes. r=omc-reviewers,pdahiya

Comment 12

[Shane Hughes [:aminomancer]]

Comment on attachment 9410020 [details]
Bug 1900486 - Re-render Review Checker messaging content when opt-in state changes. r=jprickett

beta Uplift Approval Request

• User impact if declined: To make our US Review Checker rollout schedule, this fix needs to land in 128.0. Without it, we can't show an important survey to Review Checker users. The rollout will be holding at 80% enrollment so we can launch an extra holdback experiment that deploys the survey. So how long it takes the fix to ship to release users determines the length of time 20% of US users will go without Review Checker. The sooner this fix ships, the sooner we can launch the holdback experiment, the sooner we can ramp from 80% rollout to 100% rollout.
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: In original revision
• Risk associated with taking this patch: Low
• Explanation of risk level: Behind a Nimbus experiment, and relatively low proportion of users can make it to the survey
• String changes made/needed: n/a
• Is Android affected?: no

Comment 13

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c15fda642521
Backed out changeset cba0e55d9442 for causing mochitests failures in browser_shopping_survey.js.

Comment 14

[Serban Stanca [:SerbanS]]

Backed out for causing mochitests failures in browser_shopping_survey.js.

• Backout link
• Push with failures
• Failure Log
• Failure line: TEST-UNEXPECTED-FAIL | browser/components/shopping/tests/browser/browser_shopping_survey.js | Uncaught exception in test bound test_onboarding_resets_after_opt_out - at chrome://mochitests/content/browser/browser/components/shopping/tests/browser/browser_shopping_survey.js:396 - TypeError: can't access property "wrappedJSObject", content.document.querySelector(...) is null

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b05bde6e5f9a
Re-render Review Checker messaging content when opt-in state changes. r=omc-reviewers,pdahiya

Comment 16

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/b05bde6e5f9a

Comment 18

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/a8a9f9f80cff