1858542 - Add Glean metrics for sponsored Firefox Suggestion impressions in Fenix

Status: RESOLVED FIXED

(Firefox for Android :: Search, enhancement, P1)

Comment 1

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1858542 - Add Glean metrics for Firefox Suggest impressions. (#4050)

Comment 2

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1858542 - Add Glean metrics for sponsored Firefox Suggest impressions. (#4146)

Comment 3

[[:harrisono]]

Comment on attachment 9359193 [details] [review]
[mozilla-mobile/firefox-android] Bug 1858542 - Add Glean metrics for sponsored Firefox Suggest impressions. (#4146)

# Data Review Form

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? 

Yes, through the metrics.yaml file and the [Glean Dictionary](https://dictionary.telemetry.mozilla.org/apps/fenix%E2%80%A8)

2) Is there a control mechanism that allows the user to turn the data collection on and off? 

Yes, through the "Send Usage Data" preference in the application settings

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

[[e-mail]](mailto:[e-mail]), [[e-mail]](mailto:[e-mail]), [[e-mail]](mailto:[e-mail]), [[e-mail]](mailto:[e-mail])  will be responsible for monitoring the data over time

4) Using the **[category system of data types](https://wiki.mozilla.org/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data

5) Is the data collection request for default-on or default-off?

default-on

6) Does the instrumentation include the addition of **any *new* identifiers**

No

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes

8) Does the data collection use a third-party collection tool?

No

# Result

data-review+

Comment 4

[BMO Github Automation]

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/d50f9f24a3a6d8fda5436179a50df37bfe1df05d
[main] Bug 1858542 - Introduce AwesomeBarState and Suggestion.metadata.

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/196e46c25beba770227dde1594034b1275a450e5
[main] Bug 1858542 - Introduce FxSuggestFactsMiddleware.

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/ca5cb4bdbdff0bf67bb763497e53ceeef43a69fd
[main] Bug 1858542 - Send Glean pings for Firefox Suggest impressions from Fenix.

Comment 5

[Lina Butler (ex-Mozilla)]

Setting firefox120 as affected, because we're targeting the experiment for Firefox 120—so we'll want to request an uplift for this bug, once it has time to bake in Nightly.

Comment 6

[Lina Butler (ex-Mozilla)]

Hi QA folks! The STR for testing this is a bit verbose, because of the different possible combinations (sorry! 😅), but I can give it a shot! There are three scenarios that we're interested in:

1. The user sees a sponsored Firefox Suggestion, and taps on it. Firefox should send two pings for that suggestion: an impression and a click.
2. The user sees a sponsored Firefox Suggestion, and navigates to a different page—by tapping on another suggestion, or typing a URL or search term and tapping the "enter" key. Firefox should only send an impression ping for that suggestion.
3. The user sees a sponsored Firefox Suggestion, and dismisses the address bar without navigating to a different page. Firefox should not send any pings for that suggestion.

There are a couple of points in those scenarios that I wanted to call out—and that would be great to test, please!

• The suggestion has to be visible on the screen to count as an "impression". For example, if the user types "nike"—which would show a sponsored suggestion—but that suggestion is hidden behind the on-screen keyboard, Firefox should not record an impression. (But if the user then hides the keyboard by scrolling the list of suggestion, and the Nike suggestion becomes visible, then Firefox should record the impression).
• Firefox should only record an impression for the sponsored suggestion that was last visible just before the user navigated to a different page. For example, if the user types "nike", then changes the text to "old"—which would hide the suggestion for Nike, and show one for Old Navy instead—and then taps on a suggestion, Firefox should only record an impression for Old Navy, not Nike.

Here are the flows I used when testing each scenario locally:

Scenario 1

1. Ensure "Enable Firefox Suggest" is switched on in Settings > Secret Settings, and "Suggestions from Firefox" and "Suggestions from sponsors" are switched on in Settings > Search. (These should all be on by default in Nightly).
2. Type nike into the address bar. Verify that a sponsored Nike suggestion appears in the Firefox Suggest section, and is visible on the screen.
3. Tap on the sponsored Nike suggestion.
4. Verify that two Glean pings are sent: one with an fx_suggest.ping_type of fxsuggest-impression, and one with an fx_suggest.ping_type of fxsuggest-click.

The impression ping should look like this:

{
   "metrics": {
      "uuid": {
         "fx_suggest.context_id": "25d840ba-15f1-4176-a457-ef7d5f5bb6a4"
      },
      "boolean": {
         "fx_suggest.is_clicked": true
      },
      "string": {
         "fx_suggest.ping_type": "fxsuggest-impression",
         "fx_suggest.advertiser": "nike",
         "fx_suggest.iab_category": "22 - Shopping"
      },
      "url": {
         "fx_suggest.reporting_url": "https://imp.mt48.net/static?v=2&partner=firefox_cla&sub1=nike&sub2=us&adv-id=74521&custom-data=128"
      },
      "quantity": {
         "fx_suggest.block_id": 128,
         "fx_suggest.position": 5
      }
   }
}

And the click ping should look like this:

{
   "metrics": {
      "quantity": {
         "fx_suggest.block_id": 128,
         "fx_suggest.position": 6
      },
      "uuid": {
         "fx_suggest.context_id": "25d840ba-15f1-4176-a457-ef7d5f5bb6a4"
      },
      "boolean": {
         "fx_suggest.is_clicked": true
      },
      "url": {
         "fx_suggest.reporting_url": "https://mozillacla.ampxdirect.com/nike?sub1=nike&sub2=us&custom-data=128&ctag=318370984us1282023102416"
      },
      "string": {
         "fx_suggest.advertiser": "nike",
         "fx_suggest.iab_category": "22 - Shopping",
         "fx_suggest.ping_type": "fxsuggest-click"
      }
   }
}

Scenario 2

1. Ensure "Enable Firefox Suggest" is switched on in Settings > Secret Settings, and "Suggestions from Firefox" and "Suggestions from sponsors" are switched on in Settings > Search.
2. Type old navy into the address bar. Verify that a sponsored Old Navy suggestion appears in the Firefox Suggest section, and is visible on the screen.
3. Tap on a different suggestion, or tap the "enter" button on the keyboard to trigger a search.
4. Verify that only one Glean ping is sent, with an fx_suggest.ping_type of fxsuggest-impression.

Here's an example of that impression ping:

{
   "metrics": {
      "string": {
         "fx_suggest.advertiser": "old navy",
         "fx_suggest.ping_type": "fxsuggest-impression",
         "fx_suggest.iab_category": "22 - Shopping"
      },
      "uuid": {
         "fx_suggest.context_id": "25d840ba-15f1-4176-a457-ef7d5f5bb6a4"
      },
      "quantity": {
         "fx_suggest.position": 1,
         "fx_suggest.block_id": 4496
      },
      "boolean": {
         "fx_suggest.is_clicked": false
      },
      "url": {
         "fx_suggest.reporting_url": "https://imp.mt48.net/static?v=2&partner=firefox_cla&sub1=oldnavy&sub2=us&adv-id=74808&custom-data=4496"
      }
   }
}

Scenario 3

1. Ensure "Enable Firefox Suggest" is switched on in Settings > Secret Settings, and "Suggestions from Firefox" and "Suggestions from sponsors" are switched on in Settings > Search.
2. Type athl into the address bar. Verify that a sponsored Athleta suggestion appears in the Firefox Suggest section, and is visible on the screen.
3. Swipe back to dismiss the address bar, or dismiss it by clearing the address bar and tapping on the page behind it.
4. Verify that no fx-suggest pings are sent.

Thanks!

Comment 7

[Mira Lobontiu (Android QA)]

Thank you, Lina for the provided steps.
I was able to test and verify the 3 scenarios.
For the 1st scenario, these are the Glean metrics.
For the 2nd scenarios these are the Glean metrics.
And, for the 3rd scenarios, there were no fx_suggest metrics generated.

Tested on today's Nightly build 121.0a1 from 10/26 with Xiaomi Mi8 Lite (Android 10).

Comment 8

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1858542 - Add Glean metrics for sponsored Firefox Suggest impressions. (backport #4146) (#4251)

Comment 9

[Lina Butler (ex-Mozilla)]

Comment on attachment 9360541 [details] [review]
[mozilla-mobile/firefox-android] Bug 1858542 - Add Glean metrics for sponsored Firefox Suggest impressions. (backport #4146) (#4251)

Beta/Release Uplift Approval Request

• User impact if declined: No visible user impact, but a product and business impact for Mozilla is that we won't be able to compare impression metrics for the Enhanced Cross-Platform Suggest experiment in Fenix to our Desktop baseline. This would likely mean deferring the experiment to 121.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: Verified in bug 1858542, comment 7, following the STR from bug 1858542, comment 6.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a large three-part patch, but has comprehensive test coverage, and has baked in Nightly for a few days. The bulk of the changes are in the first two commits, which add new types and tests to Android Components code. The integration surface with Fenix is small (limited to the third commit), is covered by extensive unit tests, has been manually verified in Nightly, and doesn't affect other features. Additionally, the feature will be disabled by default for users who aren't enrolled in the experiment, further reducing the risk of this patch causing a regression. The patch can also be backed out cleanly should issues arise.
• String changes made/needed: None
• Is Android affected?: Yes

Comment 10

[Dianna Smith [:diannaS]]

Comment on attachment 9360541 [details] [review]
[mozilla-mobile/firefox-android] Bug 1858542 - Add Glean metrics for sponsored Firefox Suggest impressions. (backport #4146) (#4251)

Approved for Mobile 120.0b3

Comment 11

[BMO Github Automation]

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/1604451b72d9d2d1ae2b839a890f20fc5b019f24
[releases_v120] Bug 1858542 - Introduce AwesomeBarState and Suggestion.metadata.

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/9ed5e251fb689048b0c33398369d066ffd80e51e
[releases_v120] Bug 1858542 - Introduce FxSuggestFactsMiddleware.

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/9b47a9d54caab00e6a528ac9d7310a69d60dab9b
[releases_v120] Bug 1858542 - Send Glean pings for Firefox Suggest impressions from Fenix.

Comment 12

[Adina Petridean QA [:apetridean]]

Verified as fixed on Firefox Beta 120.0b3 with Samsung GalaxyZ Fold 4 (Android 13).
The following pings were generated:

"metrics": {

"string": {

  "fx_suggest.advertiser": "nike",

  "fx_suggest.iab_category": "22 - Shopping",

  "fx_suggest.ping_type": "fxsuggest-click"

},

"quantity": {

  "fx_suggest.block_id": 128,

  "fx_suggest.position": 6

},

"url": {

  "fx_suggest.reporting_url": "https://mozillacla.ampxdirect.com/nike?sub1=nike&sub2=us&custom-data=128&ctag=318370984us1282023103011"

},

"uuid": {

  "fx_suggest.context_id": "ef8e5363-1910-4d93-914d-16d9f1753a8c"

},

"boolean": {

  "fx_suggest.is_clicked": true

}

}

}