1874051 - Rediscoverability: update callout targeting and frequency

Status: RESOLVED FIXED

(Firefox :: Shopping, enhancement, P1)

Description

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Following on the work in bug 1874047, here we want to adjust the targeting and frequency of the 3 callouts, independent of their contents.

Product and Design specs

For background, here's the product experiment brief google doc for this work: https://docs.google.com/document/d/11cP_XbmPbPagyJepebaTstEuPl_inYeKFPecNN3Wu-0/edit

The designs are in the "Final" screen of this figma

I've attached screenshots of the 3 callout updates, since bugs tend to outlive design tools.

Tweaking the targeting and frequency

The targeting is mostly driven by state stored in prefs. You can find the targeting definitions for our callouts in the asrouter devtools, or here in the source code: https://searchfox.org/mozilla-central/source/browser/components/newtab/lib/FeatureCalloutMessages.sys.mjs#630

The source docs cover targeting expressions and available targeting attributes, although I don't think we'll need to change the targeting attributes here.

I'm not sure if we can use the asrouter devtools to help us with testing here, because we need to override the existing callouts; I think we'll instead want to simulate remotely adding callouts, by appending a list of new callouts to the tail end of FeatureCalloutMessages.sys.mjs. Because there's an existing set of callouts in tree, we'll need to null those out by adding new no-op callouts with the same callout ID as the existing callouts, and a higher priority value, something roughly like:

// stuff to add to the FeatureCalloutMessages.sys.mjs file for local testing
[
    {
      // Blank out all other callouts
      "priority": 2,
      "trigger": {
        "id": "shoppingProductPageWithSidebarClosed"
      },
      "targeting": "true"
    },
    {
      // New callout 1
      "id": "SHOPPING_CALLOUT_1:A",
      "priority": 3
      // ...
    },
  // ... add the other new callouts ...
]

As with the other bug, we're not landing this in tree, so I'm not sure of the best way to get feedback and call this done. Maybe "code review" in a google doc, and drop the final working info in a comment when you close the bug?

Comment 1

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Attachment: Screenshot of revised callout 1 targeting and frequency info

Comment 2

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Attachment: Screenshot of revised callout 2 targeting and frequency info

Comment 3

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Attachment: Screenshot of revised callout 3 targeting and frequency info

Comment 4

[Shane Hughes [:aminomancer]]

I thought it might be helpful to break down the targeting expressions of the old 1.0 callouts we're replacing and explain what the attributes do, since we'll need to reuse a lot of the attributes but also add some new stuff.

This is for callout 1, id FAKESPOT_CALLOUT_CLOSED_OPTED_IN_DEFAULT in FeatureCalloutMessages:

// The `shoppingProductPageWithSidebarClosed` trigger can fire when the sidebar
// is closing OR when a product page is opened while the sidebar is already
// closed. This context attribute is used to distinguish between the two cases.
// In this case, we required that the sidebar is closing.
isSidebarClosing
// This particular callout required that the user be already opted in.
&& 'browser.shopping.experience2023.optedIn' | preferenceValue == 1
// All callouts should include this check, so that users can opt out of feature
// callouts altogether in about:preferences.
&& 'browser.newtabpage.activity-stream.asrouter.userprefs.cfr.features' | preferenceValue != false
// This is to ensure that Callout 1 is only shown to users who have never seen
// Callout 1 or 3.
&& !messageImpressions.FAKESPOT_CALLOUT_CLOSED_OPTED_IN_DEFAULT|length
&& !messageImpressions.FAKESPOT_CALLOUT_CLOSED_NOT_OPTED_IN_DEFAULT|length

This is for callout 3, the not-opted-in counterpart to callout 1, id FAKESPOT_CALLOUT_CLOSED_NOT_OPTED_IN_DEFAULT:

isSidebarClosing
// The targeting expression is exactly the same except this check is inverted,
// to exclude opted-in users.
&& 'browser.shopping.experience2023.optedIn' | preferenceValue != 1
&& 'browser.newtabpage.activity-stream.asrouter.userprefs.cfr.features' | preferenceValue != false
&& !messageImpressions.FAKESPOT_CALLOUT_CLOSED_OPTED_IN_DEFAULT|length
&& !messageImpressions.FAKESPOT_CALLOUT_CLOSED_NOT_OPTED_IN_DEFAULT|length

And this is for callout 2, the followup message that appears at least 24 hours after the other callouts, when visiting a product page while the sidebar is closed. The ID for this one is FAKESPOT_CALLOUT_PDP_OPTED_IN_DEFAULT:

// This time we're checking that the sidebar is *not* closing, i.e. that a
// product page is being opened while the sidebar is already closed. So this
// serves as a reminder that the sidebar can be opened by clicking the button.
!isSidebarClosing
&& 'browser.shopping.experience2023.optedIn' | preferenceValue == 1
&& 'browser.newtabpage.activity-stream.asrouter.userprefs.cfr.features' | preferenceValue != false
// We only want to show this callout to users who have seen Callout 1 or 3, but
// only if they saw it more than 24 hours ago. So we're checking that the last
// impression of Callout 1 or 3 was more than 24 hours ago. If there are no
// impressions, it will throw, but errors are treated as falsy.
&& (
  (
    currentDate | date - messageImpressions.FAKESPOT_CALLOUT_CLOSED_OPTED_IN_DEFAULT[
      messageImpressions.FAKESPOT_CALLOUT_CLOSED_OPTED_IN_DEFAULT | length - 1
    ] | date
  ) / 3600000 > 24
  || (
    currentDate | date - messageImpressions.FAKESPOT_CALLOUT_CLOSED_NOT_OPTED_IN_DEFAULT[
      messageImpressions.FAKESPOT_CALLOUT_CLOSED_NOT_OPTED_IN_DEFAULT | length - 1
    ] | date
  ) / 3600000 > 24
)

---

This experiment will require similar targeting. The new callout 1 is much like the old callouts 1/3, except combined - we want to remove the optedIn pref check. Let's say the new callout 1's ID will be FAKESPOT_CALLOUT_CLOSED

The new callout 2 is also like the old callout 2, showing on product page after 24 hours have passed since seeing callout 1. Except the new callout 2 actually consists of 2 separate messages, one with a checkbox and one without. The first time we show this prompt, we'll show it without a checkbox. The second and third times we show it, it'll be with a checkbox. So that requires splitting it into 2 messages.

The non-checkbox version can have a lifetime frequency cap of 1, since after that, we want to show the checkbox version. So the non-checkbox version needs to check that FAKESPOT_CALLOUT_CLOSED has been seen and 24 hours have passed since it was seen:

(
  currentDate | date - messageImpressions.FAKESPOT_CALLOUT_CLOSED[
    messageImpressions.FAKESPOT_CALLOUT_CLOSED | length - 1
  ] | date
) / 3600000 > 24

The checkbox version is similar: it needs to check that you've seen the non-checkbox version, and that 5 days have passed since then:

(
  currentDate | date - messageImpressions.FAKESPOT_CALLOUT_PDP_OPTED_IN_NO_CHECKBOX[
    messageImpressions.FAKESPOT_CALLOUT_PDP_OPTED_IN_NO_CHECKBOX | length - 1
  ] | date
) / 3600000 > 24 * 5

And it will need frequency caps like this:

frequency: {
  custom: [
    {
      cap: 1,
      period: 432000000, // Max 1 per 5 days
    },
  ],
  lifetime: 2,
}

As for callout 3, it's just like callout 2 but for not-opted-in users, and it's much simpler since the checkbox shows for all 3 times. So it can have the same impression targeting as the no-checkbox version of callout 2:

(
  currentDate | date - messageImpressions.FAKESPOT_CALLOUT_CLOSED[
    messageImpressions.FAKESPOT_CALLOUT_CLOSED | length - 1
  ] | date
) / 3600000 > 24

and its frequency cap will also be similar, but with a lifetime cap of 3 instead of 2, since all 3 showings are consolidated into one message:

frequency: {
  custom: [
    {
      cap: 1,
      period: 432000000, // Max 1 per 5 days
    },
  ],
  lifetime: 3,
}

---

The targeting expressions above are just part of what you'll need. I skipped over the straightforward stuff like the isSidebarClosing attribute and the pref checks, just to keep this post from getting any longer than it already is. But feel free to ping me on Slack (aminomancer) if you need more help.

Comment 5

[Niklas Baumgardner [:niklas]]

Attachment: Bug 1874051 - Update targeting and frequency for shopping callouts. r=aminomancer

Comment 6

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Calling this done

Comment 7

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

See attachment on bug 1874047 for the final version of the callouts + targeting updates, as used in the final experiment.

Comment 8

[Shane Hughes [:aminomancer]]

I've put the final callouts in the recipe. Note I modified the message IDs (and their references in targeting) to mimic the in-tree 1.0 callouts, per today's discussion, to prevent an issue where the 1.0 callouts might be seen when the 1.1 experiment ends. By using the same IDs, when firefox stores impressions for the 1.1 callouts, it'll think the 1.0 callouts were already seen. But I changed the screen IDs so that the 1.1 callouts can still be distinguished from the 1.0 callouts in telemetry events.

Comment 9

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

These callout changes will not land in tree. Marking as wontfix for release tracking clarity.