1848160 - Record whether Nimbus control or experimental group users visited supported product pages

Status: VERIFIED FIXED

(Firefox :: Shopping, task, P1)

Description

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Data Science would like to measure the number of product pages visited by users in both the Nimbus control group and the Nimbus active experiment group.

Comment 1

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

It's not currently clear if we'll use a boolean, histogram, or counter to accomplish this; adjusting the title accordingly.

Comment 2

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Marking 119 ship-blockers as P1s for easier tracking

Comment 3

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Attachment: Bug 1848160 - Record product page exposure for users in shopping experiment. r?perry.mcmanis,#shopping-reviewers

Comment 5

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 9353311 [details]
Bug 1848160 - Record product page exposure for users in shopping experiment. r?perry.mcmanis,#shopping-reviewers

DATA REVIEW REQUEST

1. What questions will you answer with this data?

This data will measure exposure to supported product pages for users in both the control and treatment branches of the shopping review checker experiment.

2. Why does Mozilla need to answer these questions? Are there benefits for users?
   Do we need this information to address product or business requirements?

This data will be used to compare changes in shopping behavior between control and treatment groups for the duration of the experiment.

3. What alternative methods did you consider to answer these questions?
   Why were they not sufficient?

There's no alternative way to baseline the shopping behavior for the experiment group except to also observe supported product page visits for the control group.

4. Can current instrumentation answer these questions?

No, this experimental feature is new.

5. List all proposed measurements and indicate the category of data collection for each
   measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Name	Measurement Description	Data Collection Category	Tracking Bug
shopping.product_page_visits	Counts number of visits to a supported retailer product page while enrolled in either the control or treatment branches of the shopping experiment.	interaction	https://bugzilla.mozilla.org/show_bug.cgi?id=1848160

6. Please provide a link to the documentation for this data collection which
   describes the ultimate data set in a public, complete, and accurate way.

This collection is Glean so is documented in the Glean Dictionary.

It is also mapped to legacy telemetry, so it appears in the legacy telemetry dictionary as well.

7. How long will this data be collected?

This collection has expiry '122'.

8. What populations will you measure?

All channels, countries, and locales. No filters.

9. If this data collection is default on, what is the opt-out mechanism for users?

These collections are Glean. The opt-out can be found in the product's preferences.

10. Please provide a general description of how you will analyze this data.

This data will be analyzed using the usual data science tools.

11. Where do you intend to share the results of your analysis?

Internally.

12. Is there a third-party tool (i.e. not Glean or Telemetry) that you
    are proposing to use for this data collection?

No.

Comment 6

[Chris H-C :chutten]

Attachment: data collection review request

Comment 7

[Chris H-C :chutten]

Comment on attachment 9353427 [details]
data collection review request

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection can be controlled through the product's preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 122.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Though the data itself is Category 2, Interaction, it is only collected in a Category 3 context (that a retailer is being visited).

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview- for default-on collection of Category 3+ data in release, pending Sensitive Data Review.

---

To pursue Sensitive Data Collection Review, please follow these instructions to send an email. Please reference the Data Review and Response to help the facilitators understand what's collected and what risks are being evaluated.

Comment 8

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Thanks chutten! I'll escalate to the review list.

Comment 9

[Mark Reid [:mreid]]

Following the escalation process, we've completed the sensitive data collection review and approve this collection.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/36e95068e103
Record product page exposure for users in shopping experiment. r=perry.mcmanis,shopping-reviewers,kpatenio

Comment 11

[Cosmin Sabou [:CosminS]]

Backed out for bc failures on browser_preferences_usage.js.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=36e95068e10391b8f3460c36e5651343feb808ab&selectedTaskRun=c_Ya80c9RfSx2MYAP4CHlw.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=429717530&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/b672afc613ee709e8c4f59a3e2b4ebe4eeb96005

Comment 12

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Turns out we uncovered a bug with pref caching not working as intended in Nimbus. Since this patch needs to land for 119, I'm going to queue this to land with the nimbus variables cached locally inside ShoppingUtils.

I verified the tests pass in this try job: https://treeherder.mozilla.org/jobs?repo=try&revision=133f74f41830e8ef9289357952481fb580d75d45

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cf6edd8e4242
Record product page exposure for users in shopping experiment. r=perry.mcmanis,shopping-reviewers,kpatenio

Comment 14

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/cf6edd8e4242

Comment 15

[Rares Doghi, Desktop QA]

Verified as fixed in our latest Beta 119.0b4, after setting browser.shopping.experience2023.control and reaching a few product pages we are seeing shopping.product_page_visits increment in The Debug ping viewer.