1871156 - Report impressions and clicks for sponsored and non-sponsored suggestions in the `events` ping

Status: RESOLVED FIXED

(Firefox for Android :: Search, task)

Description

[Lina Butler (ex-Mozilla)]

Fenix currently reports impressions for sponsored (AMP) and non-sponsored (Wikipedia) suggestions in the fx-suggest ping. The fx-suggest ping contains Category 2.5 data, so it intentionally omits the client_id, and uses the separate context_id instead.

However, omitting the client_id makes experiment analysis difficult, because our tooling relies on having it. On Desktop, we use the urlbar.engagement and urlbar.abandonment events for this analysis. These events are sent in Desktop's events ping. But Fenix has no such equivalent, which makes it hard to measure engagement for our Cross-Platform Suggest experiment.

Long-term, we'd like to port Desktop's urlbar.engagement and urlbar.abandonment events to Fenix. But this is a more substantial engineering lift, because Desktop's events have additional keys for which we'd need new instrumentation on mobile—and because we'd need to consider if collecting the same set of keys makes sense.

In the short- to medium-term, we'd still like to collect this data in some way. Fenix already has Fenix-specific events for clicks on other types of suggestions: awesomebar.bookmark_suggestion_clicked, awesomebar.history_suggestion_clicked, awesomebar.opened_tab_suggestion_clicked, etc. Since these Fenix events are already different from Desktop's, we can add new Fenix-specific events for clicks on sponsored and non-sponsored suggestions that follow this pattern: awesomebar.sponsored_suggestion_clicked and awesomebar.non_sponsored_suggestion_clicked.

Fenix doesn't currently collect impressions for any types of suggestions in its events ping, but has the instrumentation to do this—so we can also record impression events for sponsored and non-sponsored suggestions as part of this bug: awesomebar.sponsored_suggestion_impressed and awesomebar.non_sponsored_suggestion_impressed.

When we record these new impression events, we can also align with Desktop's definition of an "impression", which is for engaged and abandoned search sessions. (In contrast, the fx-suggest ping records impressions for engaged search sessions only, because that's how our partners define "impression").

Comment 1

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1871156 - Report impressions and clicks for Firefox Suggestions in the `events` ping. (#4914)

Comment 2

[BMO Github Automation]

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/5dca388356d744bb34fea161309494f00f08ff23
[main] Bug 1871156 - Report impressions and clicks for Firefox Suggestions in the events ping.

Comment 3

[Lina Butler (ex-Mozilla)]

I verified that the new events are sent correctly in today's Nightly! 🎉 Here are the scenarios in which we expect to send the new events:

1. A sponsored Firefox Suggestion is visible on the screen, and the user taps on it.
2. A non-sponsored Firefox Suggestion is visible on the screen, and the user taps on it.
3. A sponsored Firefox Suggestion is visible on the screen, and the user navigates to a different page by tapping on a different suggestion, or typing a URL or a search term and then tapping Enter.
4. A non-sponsored Firefox Suggestion is visible on the screen, and the user navigates to a different page.
5. A sponsored Firefox Suggestion is visible on the screen, and the user dismisses the awesomebar without navigating to a different page.
6. A non-sponsored Firefox Suggestion is visible on the screen, and the user dismisses the awesomebar without navigating to a different page.

And here are the STR I used for each scenario:

Scenario 1

1. Type nike into the address bar. Verify that a sponsored Nike suggestion appears in the Firefox Suggest section, and is visible on the screen.
2. Tap on the sponsored Nike suggestion.
3. Submit the Glean events ping using the Glean debug activity.
4. Verify that the events ping contains the sponsored_suggestion_impressed and sponsored_suggestion_clicked events.

The structure of the events in the ping should look like this:

{
  "events": [
    {
      "timestamp": 895277,
      "category": "awesomebar",
      "name": "sponsored_suggestion_impressed",
      "extra": {
        "provider": "amp",
        "engagement_abandoned": "false"
      }
    },
    {
      "timestamp": 895283,
      "category": "awesomebar",
      "name": "sponsored_suggestion_clicked",
      "extra": {
        "provider": "amp"
      }
    }
  ]
}

Scenario 2

1. Type cali into the address bar. Verify that a Wikipedia suggestion titled "Wikipedia - California" appears in the Firefox Suggest section, and is visible on the screen.
2. Tap on the Wikipedia suggestion.
3. Submit the Glean events ping.
4. Verify that the events ping contains the non_sponsored_suggestion_impressed and non_sponsored_suggestion_clicked events.

{
  "events": [
    {
      "timestamp": 10815,
      "category": "awesomebar",
      "name": "non_sponsored_suggestion_impressed",
      "extra": {
        "provider": "wikipedia",
        "engagement_abandoned": "false"
      }
    },
    {
      "timestamp": 10817,
      "category": "awesomebar",
      "name": "non_sponsored_suggestion_clicked",
      "extra": {
        "provider": "wikipedia"
      }
    }
  ]
}

Scenario 3

1. Type nike into the address bar. Verify that a sponsored Nike suggestion appears in the Firefox Suggest section, and is visible on the screen.
2. Tap on a different suggestion, or tap the Enter button on the keyboard to trigger a search.
3. Submit the Glean events ping.
4. Verify that the events ping contains the sponsored_suggestion_impressed event, without a corresponding sponsored_suggestion_clicked event.

{
  "events": [
    {
      "timestamp": 6252,
      "category": "awesomebar",
      "name": "sponsored_suggestion_impressed",
      "extra": {
        "engagement_abandoned": "false",
        "provider": "amp"
      }
    }
  ]
}

Scenario 4

1. Type cali into the address bar. Verify that a Wikipedia suggestion titled "Wikipedia - California" appears in the Firefox Suggest section, and is visible on the screen.
2. Tap on a different suggestion, or tap the Enter button on the keyboard to trigger a search.
3. Submit the Glean events ping.
4. Verify that the events ping contains the non_sponsored_suggestion_impressed event, without a corresponding non_sponsored_suggestion_clicked event.

{
  "events": [
    {
      "timestamp": 7528,
      "category": "awesomebar",
      "name": "non_sponsored_suggestion_impressed",
      "extra": {
        "provider": "wikipedia",
        "engagement_abandoned": "false"
      }
    }
  ]
}

Scenario 5

1. Type nike into the address bar. Verify that a sponsored Nike suggestion appears in the Firefox Suggest section, and is visible on the screen.
2. Swipe right or press the device's Back button to dismiss the list of suggestions.
3. Submit the Glean events ping.
4. Verify that the events ping contains the sponsored_suggestion_impressed event, and note that the value of the engagement_abandoned extra key is now true.

{
  "events": [
    {
      "timestamp": 0,
      "category": "awesomebar",
      "name": "sponsored_suggestion_impressed",
      "extra": {
        "engagement_abandoned": "true",
        "provider": "amp"
      }
    }
  ]
}

Scenario 6

1. Type cali into the address bar. Verify that a Wikipedia suggestion titled "Wikipedia - California" appears in the Firefox Suggest section, and is visible on the screen.
2. Swipe right or press the device's Back button to dismiss the list of suggestions.
3. Submit the Glean events ping.
4. Verify that the events ping contains the non_sponsored_suggestion_impressed event, with engagement_abandoned set to true.

{
  "events": [
    {
      "timestamp": 7365,
      "category": "awesomebar",
      "name": "non_sponsored_suggestion_impressed",
      "extra": {
        "provider": "wikipedia",
        "engagement_abandoned": "true"
      }
    }
  ]
}

Comment 4

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1871156 - Report impressions and clicks for Firefox Suggestions in the `events` ping. (backport #4914) (#5068)

Comment 5

[Lina Butler (ex-Mozilla)]

Comment on attachment 9371472 [details] [review]
[mozilla-mobile/firefox-android] Bug 1871156 - Report impressions and clicks for Firefox Suggestions in the events ping. (backport #4914) (#5068)

Beta/Release Uplift Approval Request

• User impact if declined: No visible user impact, but the impact for Mozilla is that we won't be able to restart the Enhanced Cross-Platform Suggest experiment until version 123, since the telemetry in versions 122 and older can't answer the questions about the effects of the experiment.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: Comment 3, if additional testing is desired.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch adds new Glean metrics for sponsored and non-sponsored suggestions. The additions are scoped to the new Firefox Suggestion types, which are disabled by default for users who aren't enrolled in the experiment. The patch has good unit test coverage, and has been verified in Nightly.
• String changes made/needed: None
• Is Android affected?: Yes

Comment 6

[Donal Meehan [:dmeehan]]

Comment on attachment 9371472 [details] [review]
[mozilla-mobile/firefox-android] Bug 1871156 - Report impressions and clicks for Firefox Suggestions in the `events` ping. (backport #4914) (#5068)

Approved for Fenix 122.0b7

Comment 7

[BMO Github Automation]

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/c7fa9296d722bfc10c8477d271b7f76899724d81
[releases_v122] Bug 1871156 - Report impressions and clicks for Firefox Suggestions in the events ping.