1878307 - [Experiment] [Intermittent] The Callout for Opted Out users from Control branch is displayed instead of Callout 1 in the Treatment A branch

Status: ASSIGNED

(Firefox :: Messaging System, defect, P1)

Description

[Carmen Fat [:cfat] - Ecosystem QA]

[Notes]:

• We only managed to reproduce this issue twice (with 2 different profiles) out of ~50 new profiles.

[Affected versions]:

• Firefox Release 122.0 - Build ID: 20240118164516

[Affected Platforms]:

• Windows 10 x64

[Prerequisites]:

• Have a Firefox profile older than 28 days.
• Set the browser.search.region pref to US.
• Paste the user.js file in the Profile Folder from the “about:support” page and restart the browser.

[Steps to reproduce]:

1. Open the browser with the profile from the prerequisites.
2. Navigate to one of the supported websites (e.g. www.amazon.com) and click on any of the products.
3. Click the X (Close) button on the Review Checked sidebar.
4. Observe what happens next.

[Expected result]:

• Callout 1 (v1.1) is triggered.

[Actual result]:

• The old Callout (v.1.0) for Opted Out users from the Control branch is triggered.

[Additional notes]:

• Not very confident, but this issue might be related to the ID changes mentioned in this comment.
• Unfortunately we weren’t able to catch the issue in a screen recording, but we managed to generate the Glean impression specific to this Callout (the impression also shows that we were enrolled in the treatment-a branch of the experiment.
• Also, here is the profile where we reproduced the issue in case you need it for investigation.

Comment 1

[Carmen Fat [:cfat] - Ecosystem QA]

Attachment: Callout Control branch.png

Comment 2

[Shane Hughes [:aminomancer]]

Did you happen to save any of the console log output? I'm having trouble replicating the bug. It's hard to see how this can happen, but I do know it's not related to the ID changes, as FAKESPOT_CALLOUT_NO_OP_DUMMY should block all the control callouts no matter what. Only scenario where it wouldn't is if the remote experiment loader failed to load the experiment into ASRouter, or if enrollment failed but somehow still recorded telemetry saying it was enrolled.

Comment 3

[Carmen Fat [:cfat] - Ecosystem QA]

Unfortunately, we were not able to record any logs when we encountered the issue.

However, we investigated it further and we believe that we ran into the scenario where the experiment messages fail to load to asrouter, so the default message (built-in browser) is shown instead.

The steps we used to investigate this are the following:

1. Enroll in the experiment.
2. Enable asrouter by setting to true the browser.newtabpage.activity-stream.asrouter.devtoolsEnabled pref.
3. Open the asrouter panel.
4. Block all the callout messages from the "messaging-experiment" section.
5. Navigate to a product page -> the default callout (from the browser) is displayed instead of v.1.1 callout.

Additionally, the default callout will not be displayed if we don't block the "FAKESPOT_CALLOUT_NO_OP_DUMMY", which makes us think that the messages were not loaded in asrouter.

In case there are users enrolled in the experiment who will encounter this issue, we suggest filtering out the default callout impressions in the treatment branch.
Also, in this case, where the default callout is shown, there would be the risk that the users enrolled in the treatment branch will see the v.1.1 callout as well on the same day.

Comment 4

[Shane Hughes [:aminomancer]]

(In reply to Carmen Fat [:cfat] - Ecosystem QA from comment #3)

In case there are users enrolled in the experiment who will encounter this issue, we suggest filtering out the default callout impressions in the treatment branch.

Thanks, that makes sense. We have unique screen IDs for the 1.1 callouts, so in our data analysis we'll search exclusively for those.

Also, in this case, where the default callout is shown, there would be the risk that the users enrolled in the treatment branch will see the v.1.1 callout as well on the same day.

That shouldn't be possible, since the first callout from the 1.1 experiment is configured to never show to users who've already seen the 1.0 callouts. Though it is possible for the flow to be replaced midway through, but we'll filter those users out by screen ID. Like if you only saw the 1.0 version of callout 1 but never saw callout 2/3, then you could see the 1.1 version of callout 2. But we'd be able to tell in the data, since the 1.0 version of callout 1 has a different screen ID than the 1.1 version. This was unavoidable but it's a tiny proportion of users and something we can control for in analysis.