1878036 - [Experiment] Callout 1 is triggered after the user opts in Review Checker and then turns off the feature

Status: VERIFIED FIXED

(Firefox :: Messaging System, defect, P1)

Description

[Carmen Fat [:cfat] - Ecosystem QA]

[Affected versions]:

• Firefox Release 121.0.1 - Build ID: 20240108143603
• Firefox Release 122.0 - Build ID: 20240118164516
• Firefox Beta 123.0b5 - Build ID: 20240131091640

[Affected Platforms]:

• Windows 10 x64
• macOS 13.1
• Linux Ubuntu 22.04 x64

[Prerequisites]:

• Have a Firefox profile older than 28 days.
• Set the browser.search.region pref to US.
• Paste the user.js file in the Profile Folder from the “about:support” page and restart the browser.

[Steps to reproduce]:

1. Open the browser with the profile from the prerequisites.
2. Navigate to one of the supported websites (e.g. www.amazon.com) and click on any of the products.
3. Click the "Yes, try it" option from the bottom part of the Review Checker sidebar.
4. Click the Settings drop-down from the Review Checker sidebar.
5. Click the "Turn off Review Checker" button.
6. Observe what happens next.

[Expected result]:

• There is no Callout triggered by turning off the Review Checker feature.

[Actual result]:

• Callout 1 is triggered and displayed.

[Notes]:

• According to Figma, no Callouts should be shown if the user turns the feature off.
• Here is a screen recording of the issue.

Comment 1

[Shane Hughes [:aminomancer]]

Attachment: Bug 1878036 - Fix review checker callout trigger timing. r=jhirsch

Comment 2

[Shane Hughes [:aminomancer]]

[Tracking Requested - why for this release]: The experiment requirements are that we don't show any messages to users who have disabled Review Checker. Without this patch, we would unintentionally show messages to users right when they disable Review Checker.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox123 (beta). However, the bug still has low severity.

:aminomancer, could you please increase the severity for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 4

[Shane Hughes [:aminomancer]]

We'll set severity along with priority and points in triage later today.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/db81aa2d9b8b
Fix review checker callout trigger timing. r=jhirsch

Comment 6

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/db81aa2d9b8b

Comment 7

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/db81aa2d9b8b

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:aminomancer, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox123 to wontfix.

For more information, please visit BugBot documentation.

Comment 9

[Carmen Fat [:cfat] - Ecosystem QA]

I have verified this issue on the latest Firefox Nightly 124.0a1 (Build ID: 20240206095115) on Windows 10 x64, macOS 12.6.1, and Linux Ubuntu 20.04 x64.

• The issue is no longer reproducible with the steps from the Description.

However, if the user opens the Review Checker sidebar and then closes it after performing the steps from the Description, Callout 1 is displayed. For this specific scenario, I have logged Bug 1878932.

Comment 10

[Shane Hughes [:aminomancer]]

Comment on attachment 9377765 [details]
Bug 1878036 - Fix review checker callout trigger timing. r=jhirsch

Beta/Release Uplift Approval Request

• User impact if declined: The experiment requirements are that we don't show any messages to users who have disabled Review Checker. Without this patch, we would unintentionally show messages to users right when they disable Review Checker.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Relatively small amount of code, changes to an optional part of the browser, fix has already been verified.
• String changes made/needed:
• Is Android affected?: No

Comment 11

[Pascal Chevrel:pascalc]

Comment on attachment 9377765 [details]
Bug 1878036 - Fix review checker callout trigger timing. r=jhirsch

Approved for 123 beta 8, thanks.

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/1b4583d75b47

Comment 13

[Carmen Fat [:cfat] - Ecosystem QA]

I have verified this issue on the latest Firefox Beta 123.0b8 (Build ID: 20240207091958) on Windows 10 x64, macOS 12.6.1, and Linux Ubuntu 20.04 x64.

• The Callout is no longer displayed after tuning off the Review Checker feature.