1886215 - [Experiment] Callout 1 is displayed soon after opting out the Review Checker feature and dismissing Callout 5

Status: RESOLVED WONTFIX

(Firefox :: Messaging System, defect, P1)

Description

[Carmen Fat [:cfat] - Ecosystem QA]

[Affected versions]:

• Firefox Release 124.0 (Build ID: 20240311145044)
• Firefox Beta 125.0b1 (Build ID: 20240318085508)
• Firefox Nightly 126.0a1 (Build ID: 20240318110542)

[Affected Platforms]:

• Windows 10 x64
• macOS 13.1
• Linux Ubuntu 22.04 x64

[Prerequisites]:

• Set the browser.search.region pref to US.
• Paste the user.js file in the Profile Folder from the “about:support” page and restart the browser.

[Steps to reproduce]:

1. Open the browser with the profile from the prerequisites.
2. Navigate to one of the supported websites (e.g. www.amazon.com) and click on any of the products.
3. Click the "Yes, try it" option from the bottom part of the Review Checker sidebar.
4. Click the Settings drop-down from the Review Checker sidebar.
5. Click the "Turn off Review Checker" button.
6. Dismiss Callout 5.
7. Wait several seconds, then focus another tab.
8. Focus again the product’s page.
9. Observe what happens next.

[Expected result]:

• Nothing happens, there is no Callout triggered by this action.

[Actual result]:

• Callout 1 is triggered and displayed.

[Notes]:

• Here is a screen recording of the issue.

Comment 1

[Shane Hughes [:aminomancer]]

Looks like this is a basic regression in the shopping code. This message, callout 1, is only supposed to show when the sidebar is closed, not when activating a PDP tab. I think TabSelect used to count as a navigation, but no longer does. Either way, it should. I will try to fix this in code but that won't help with the experiment. I don't think it's disastrous to have this show up on focusing the tab, since the copy is not really specific to closing the tab. But it is a problem for it to show up for users who've already disabled RC, so I need to think of some way to avoid this off-train.

Comment 2

[Shane Hughes [:aminomancer]]

Attachment: Bug 1886215 - Fix review checker callout triggers. r=jhirsch

Comment 3

[Shane Hughes [:aminomancer]]

I've fixed the reported issue by transferring the v1.2 messages over to the experiment recipe and correcting their targeting to include only not-opted-in users (optedIn = 0), excluding opted-out users (optedIn = 2).

There's still an underlying issue that it's technically possible for callouts with isSidebarClosing targeting to show when a PDP tab is foregrounded. In that situation where a tab is foregrounded, isSidebarClosing should be false. Fortunately this doesn't cause issues for the experiment due to other targeting conditions. The only way to make a callout show in this situation is to visit a PDP, opt-in with the sidebar onboarding, close the sidebar, switch to about:config, change the optedIn pref manually to 0, then switch back to the PDP.

Since most users don't use about:config, and it's expected that tweaking prefs in about:config can cause weird behavior, and there's no particular reason for users to modify this pref manually, we don't think it's a blocker for the experiment. But we want to fix the issue anyway, since it could theoretically affect other messages in the future, as it's just a lucky coincidence that targeting prevents it from causing problems for this one. So we'll keep this bug open for my patch to land in 126.

Comment 4

[:Gijs (he/him)]

Although we'd like this to be fixed it sounds like this isn't an S2 in terms of Review Checker (or other messaging systems bits) shipping.

Comment 5

[Shane Hughes [:aminomancer]]

These callouts & triggers are being removed.