1819766 - Add exposure event for tracking experiment exposure in urlbar experiments

Status: VERIFIED FIXED

(Firefox :: Address Bar, enhancement)

Description

[Wil Stuckey]

https://mozilla-hub.atlassian.net/browse/SNT-470

Comment 1

[Wil Stuckey]

Attachment: Bug 1819766 - Add exposure events and pref for excluding providers from the quicksuggest results r=adw

Comment 2

[Wil Stuckey]

Attachment: WIP: Bug 1819766 Move unsortedResults into UrlbarContext

Comment 3

[Wil Stuckey]

Attachment: WIP: Bug 1819766 Add more detail to the quicksuggest result subtype.

Depends on D173996

Comment 4

[Wil Stuckey]

Attachment: Bug 1819766 - Add support for exposure based experiments. r=adw

This change adds support for exposure based experiments by allowing
a Nimbus variable/pref to specify the urlbar provider that should
trigger an exposure event as well as a secondary boolean variable/pref
that controls the visibility of the exposed result. The exposure should
be registered when a result 'can be added' but may or may not be shown
based on the value of the displayExposureProvider variable.

• Add exposure event to metrics.yaml
• Add new Nimbus variable (exposureProvider) to specify the urlbar
  providers that should trigger exposure events .
• Add new Nimbus variable (displayExposureProvider) that controls the visibility
  of the provider results that matched the exposureProvider variable.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b6b243fba0e
Add support for exposure based experiments. r=adw

Comment 6

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/2b6b243fba0e

Comment 7

[Wil Stuckey]

Attachment: Data Review

adding data review request

Comment 8

[Wil Stuckey]

Attachment: Data Review

updated answers

Comment 9

[Travis Long [:travis_]]

Comment on attachment 9334204 [details]
Data Review

Data Review

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the telemetry and experiment preferences in the application settings.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

Permanent collection to be monitored by :wstuckey

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data

5. Is the data collection request for default-on or default-off?

Default-on, only for users enrolled in an experiment with exposures.

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

Result

data-review+

Comment 10

[Wil Stuckey]

Travis or Data Reviewers, As part of this bug we adjusted some of the values returned in the result type. Wanted to check in to confirm that the data returned in the result types are still considered Category 2 as it has been flagged that it might be Category 2.5. The possible values (linked below) have been expanded to represent the type of search suggestion shown to the user and are limited in granularity to the merino provider level. While never report on the individual content of the suggested just what provider they come from, some providers have a small number of destinations (ie weather). Looking for data steward feed back and happy to discuss in further detail.

Sheet with all result values from the past 30 days as of this comment: https://docs.google.com/spreadsheets/d/1KlY4qNME912lLaGyZ2dj08KKMIhl8uemnU2PjvkNtdA/edit

Comment 11

[Travis Long [:travis_]]

After careful consideration and consultation about this, the current data-review still stands. Since this is just an exposure event, it doesn't indicate whether or not a particular client_id went on to visit a specific site that could be associated with a provider, hence this is still going to be Category 2. Had this been an engagement event, then it would have likely crossed the threshold into 2+ as it would link a specific client_id's visit to a provider url and would be tantamount to recording a client_id's traffic, which is definitely 3+.

Comment 12

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I will mark this issue as Verified-Fixed based on the latest experiments that were successfully run and no issues were found with the exposure events.