1948142 - Data Review Request for Subscription Platform V3

Status: RESOLVED MOVED

(Firefox :: Untriaged, task)

Description

[dalvarez]

1) What questions will you answer with this data?

• How many customers do we support?
• When do the customers drop off during the checkout process?
• Are customers dropping off because of code or technical failures?
• Are there devices we are failing to support appropriately?
• What marketing channels are most successful at bringing in customers?
• What is the interest in our different offerings?

2) Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

• Mozilla needs to understand the user experience of the subscription platform process in order to eliminate pain points in the process, and to gain the business intelligence needed to determine how well marketing campaigns are working.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

Sentry can be used to raise errors to the Sentry dashboard, and Stripe can be used to determine the ratios of cart creations, and payment completions. Neither of these platforms record the necessary information to understand the experience of the user, nor to identify the success-rate of marketing campaigns.

4) Can current instrumentation answer these questions?

No. The Subscription Platform 3.0 is not in active use yet, and does not have prior metrics collections.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

Measurement Description	Data Collection Category	Tracking Bug #	Measurement Name
Checkout engage event	Category 2	#FXA-7531	pay_setup.engage
Checkout fail event	Category 2	#FXA-7531	pay_setup.fail
Checkout submit event	Category 2	#FXA-7531	pay_setup.submit
Checkout engage event	Category 2	#FXA-7531	pay_setup.success
Checkout view event	Category 2	#FXA-7531	pay_setup.view
The client id of the relying party	Category 1	#FXA-7265	relying_party.oauth_client_id
The service name of the relying party	Category 1	#FXA-7531	relying_party.service
The device type used. One of 'mobile', 'tablet', or ‘’	Category 1	#FXA-7531	session.device_type
Entrypoint to the service	Category 2	#FXA-7531	session.entrypoint
Identifier for the experiment the user is part of at the entrypoint	Category 2	#FXA-7531	session.entrypoint_experiment
Identifier for the experiment variation the user is part of at the entrypoint	Category 2	#FXA-7531	session.entrypoint_variation
ID generated by FxA for its flow metrics	Category 1	#FXA-7531	session.flow_id
Whether the checkout flow is for new users or existing users. One of “with-account” or “without-account"	Category 2	#FXA-7531	subscription.checkout_type
Currency of a subscription used at checkout	Category 2	#FXA-7531	subscription.currency
The error id, if any, encountered for a fxa_pay_setup - fail event	Category 1	#FXA-7531	subscription.error_id
Interval of a subscription used at checkout	Category 2	#FXA-7531	subscription.interval
ID of the offering a customer subscribed to	Category 2	#FXA-7531	subscription.offering_id
The third party service ultimately processing a user’s payments (e.g. 'stripe')	Category 2	#FXA-7531	subscription.payment_provider
Plan ID of a subscription	Category 2	#FXA-7531	subscription.plan_id
Product ID of a subscription	Category 2	#FXA-7531	subscription.product_id
The Stripe customer-facing promotion code applied, if any	Category 2	#FXA-7531	subscription.promotion_code
Comma-separated list of Stripe price/plan IDs the user is already subscribed to	Category 2	#FXA-7531	subscription.subscribed_plan_ids
A marketing campaign	Category 2	#FXA-7531	utm.campaign
The content on which the user acted (e.g. 'fx-sync-get-started')	Category 2	#FXA-7531	utm.content
The "medium" on which the user acted (e.g. 'email')	Category 2	#FXA-7531	utm.medium
The source from where the user started (e.g. 'fx-website')	Category 2	#FXA-7531	utm.source
This metric is similar to the utm.source; it is used in the Firefox browser	Category 2	#FXA-7531	utm.term

6) Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

This collection is Glean so is documented in the Glean Dictionary.

Note: The full discussion from the original data review request, which notably included the category 4 data, FxA's account.user_id field (no longer included in the dataset), can be found at this internally-available google drive document.

7) How long will this data be collected? Choose one of the following:

This collection will be collected permanently.
[e-mail], [e-mail] will be responsible for the permanent collections.

8) What populations will you measure?

All channels, countries, and locales. No filters.

9) If this data collection is default on, what is the opt-out mechanism for users?

These collections are Glean. The opt-out can be found in the product's preferences. Session data will be recorded up until a user signs in with an account that has opted-out of metrics collection, or until they opt-out of collection, whichever comes first.

10) Please provide a general description of how you will analyze this data.

• We will work with DS and DE to create dashboards for analysis of the metrics.
• The data will be analyzed as events, rather than reviewed as aggregate data.

11) Where do you intend to share the results of your analysis?

• The results of the analysis will be available to internal teams and NDA'd Mozillians only.

12) Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

No.

Comment 1

[dalvarez]

This bug report is a followup to bugzilla bug 1924601. At the time of that report, a field was included (FxA's account id), which was discovered to be a category 4 field. Per the decision brief about this field, it was believed to be a category 1 field and as such acceptable in this dataset. A sensitive review was initiated, and the decision was made by Martin Thomson that this case falls outside the purview of the decision brief, and the FxA account id is category 4. The Subscription Platform team made the decision that this is not necessary for our business purposes, and has removed it entirely from the recorded fields. As the current highest data collection category is category 2, it is not believed that this will require a similar sensitive data review process.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Fenix::Accounts and Sync' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[dalvarez]

Per the discussion in the original ticket, it should stay as Firefox::Untriaged, as that's the documented path.

Comment 4

[Francesco Lodolo [:flod]]

There is no currently a data review request, or data steward flagged?

Comment 5

[dalvarez]

Ticket moved to https://github.com/mozilla/fxa/pull/18513