2021134 - Data review: Add churn/retention metrics to SubPlat telemetry

Status: NEW

(Cloud Services :: Server: Firefox Accounts, task)

Description

[Lisa Chan]

1) What questions will you answer with this data?

• Funnel & reach
  • How many customers reach each page (Subscription Management, retention (cancel/stay), interstitial offer)?
  • What are the drop-off points (e.g., view → engage → submit → result) for retention and interstitial flows?
• Effectiveness
  • What proportion of customers complete each retention flow successfully vs. hit errors?
  • Which product subscriptions (by offering_id + interval), if any, see successful retention uptake?
• Attribution & entrypoints
  • Which entrypoints drive usage (e.g., source / entrypoint, plus utm_* when present)?
  • Do certain entrypoints correlate with higher success or error rates?
• Reliability / diagnostics
  • What are the most common error reasons (error_reason) and where do they occur in the flow (step)?
  • Are there regressions in success/error rates after releases/changes?

2) Why does Mozilla need to answer these questions? Benefits for users / product requirements?

• Customer benefit
  • We would like to identify friction and failures in cancellation/retention experiences, enable fixes that reduce dead-ends, and improve clarity and completion rates.
• Product/business requirement
  • Subscriptions need reliable, measurable flows to understand churn reduction initiatives and ensure that retention and interstitial offer experiences behave as intended.
• Operational benefit
  • Error breakdowns help engineering prioritize bugs and validate fixes (e.g., “subscription_not_found”, “customer_mismatch”, “no_*_found”).

3) What alternative methods did you consider, and why were they not sufficient?

• Server logs / API logs
  • While good for debugging individual incidents, it is not reliable for product analytics at scale (sampling, retention, querying cost, and not purpose-built for analysis dashboards).
• Stripe / provider dashboards
  • Don’t represent Mozilla’s UI journey (pages, steps, entrypoints) or eligibility/offer logic. It is also hard to join to the customer journey context.
• Manual QA / user testing
  • Helpful qualitatively, but not sufficient to quantify rates (drop-off, error distribution) across channels/locales and over time.

4) Can current instrumentation answer these questions?
No. Subscription Platform’s current telemetry does not capture page views on the Subscription Management page nor the full flow retention and interstitial offer lifecycle (page reached, step, outcome, error_reason, entrypoint attribution). We would like to do this in a consistent, queryable way for these specific journeys.

5) Proposed measurements + Firefox data collection category (Mozilla categories)
Mozilla’s long-standing categories are: Category 1 (technical), Category 2 (interaction), Category 3 (web activity), Category 4 (highly sensitive).
These metrics are interaction with Mozilla UI/flows, so Category 2 (interaction data).

Description                                          | Data Collection Category | Tracking Bug | Measurement Name
---------------------------------------------------------------------------------------------------------------
Records page views for management pages; includes   | Category 2               | PAY-3472     | subscriptions.page_view
page_name, source                                   |                          |              |

Lifecycle tracking for retention decision flows     | Category 2               | PAY-3472     | subscriptions.retention_flow
with entrypoint, flow_type, eligibility_status,     |                          |              |
step, outcome, error_reason, attribution, etc.      |                          |              |

Engagement tracking for non-retention interstitial  | Category 2               | PAY-3472     | subscriptions.interstitial_offer
offer journeys with entrypoint, step, outcome,      |                          |              |
error_reason, attribution + product fields          |                          |              |

6) Link to public documentation describing the ultimate data set
This collection is Glean - documented in the Glean Dictionary.

7) How long will this data be collected?
This collection will be collected permanently.
[e-mail] and [e-mail] will be responsible for the permanent collections.

8) What populations will you measure?
All channels, countries, and locales. No filters.

9) If default-on, what is the opt-out mechanism for users?
These collections are Glean. The opt-out can be found in the product's preferences. Session data will be recorded up until a user signs in with an account that has opted-out of metrics collection, or until they opt-out of collection, whichever comes first.

10) General description of how you will analyze this data
We would like to build dashboards and/or scheduled queries to:

• Analyze customer progression and conversion rates between lifecycle steps (view→engage→submit→result) within the retention_flow and interstitial_offer journeys
• Evaluate outcome distribution (success vs error) across retention flows and eligibilty_status, with detailed breakdowns by error_reason.
• Analyze flow participation and outcomes segment by entrypoint and associated utm_* (when present).
• Assess retention and offer effectiveness by product subscription (offering_id and interval)
• Establish baseline conversion and error metrics and monitor trends to detect deviations following releases or configuration changes.

11) Where will you share results?
The results of the analysis will be available to internal teams and NDA'd Mozillians only.

12) Any third-party tool proposed (not Glean/Telemetry)?
No - though this is for the client-side/frontend Glean (browser telemetry via @mozilla/glean/web), not server-side.

This Bugzilla report is mirrored in the FxA repository - https://github.com/mozilla/fxa/pull/20000

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Privacy: Anti-Tracking' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Manuel Bucher [:manuel]]

Moving to a Mozilla account component. Potentially also Firefox :: Firefox Accounts might also be suited for this bug.

Comment 3

[Roger Yang [:royang]]

(In reply to Lisa Chan from comment #0)

1) What questions will you answer with this data?

• Funnel & reach
  • How many customers reach each page (Subscription Management, retention (cancel/stay), interstitial offer)?
  • What are the drop-off points (e.g., view → engage → submit → result) for retention and interstitial flows?
• Effectiveness
  • What proportion of customers complete each retention flow successfully vs. hit errors?
  • Which product subscriptions (by offering_id + interval), if any, see successful retention uptake?
• Attribution & entrypoints
  • Which entrypoints drive usage (e.g., source / entrypoint, plus utm_* when present)?
  • Do certain entrypoints correlate with higher success or error rates?
• Reliability / diagnostics
  • What are the most common error reasons (error_reason) and where do they occur in the flow (step)?
  • Are there regressions in success/error rates after releases/changes?

2) Why does Mozilla need to answer these questions? Benefits for users / product requirements?

• Customer benefit
  • We would like to identify friction and failures in cancellation/retention experiences, enable fixes that reduce dead-ends, and improve clarity and completion rates.
• Product/business requirement
  • Subscriptions need reliable, measurable flows to understand churn reduction initiatives and ensure that retention and interstitial offer experiences behave as intended.
• Operational benefit
  • Error breakdowns help engineering prioritize bugs and validate fixes (e.g., “subscription_not_found”, “customer_mismatch”, “no_*_found”).

3) What alternative methods did you consider, and why were they not sufficient?

• Server logs / API logs
  • While good for debugging individual incidents, it is not reliable for product analytics at scale (sampling, retention, querying cost, and not purpose-built for analysis dashboards).
• Stripe / provider dashboards
  • Don’t represent Mozilla’s UI journey (pages, steps, entrypoints) or eligibility/offer logic. It is also hard to join to the customer journey context.
• Manual QA / user testing
  • Helpful qualitatively, but not sufficient to quantify rates (drop-off, error distribution) across channels/locales and over time.

4) Can current instrumentation answer these questions?
No. Subscription Platform’s current telemetry does not capture page views on the Subscription Management page nor the full flow retention and interstitial offer lifecycle (page reached, step, outcome, error_reason, entrypoint attribution). We would like to do this in a consistent, queryable way for these specific journeys.

5) Proposed measurements + Firefox data collection category (Mozilla categories)
Mozilla’s long-standing categories are: Category 1 (technical), Category 2 (interaction), Category 3 (web activity), Category 4 (highly sensitive).
These metrics are interaction with Mozilla UI/flows, so Category 2 (interaction data).

Description                                          | Data Collection Category | Tracking Bug | Measurement Name
---------------------------------------------------------------------------------------------------------------
Records page views for management pages; includes   | Category 2               | PAY-3472     | subscriptions.page_view
page_name, source                                   |                          |              |

Lifecycle tracking for retention decision flows     | Category 2               | PAY-3472     | subscriptions.retention_flow
with entrypoint, flow_type, eligibility_status,     |                          |              |
step, outcome, error_reason, attribution, etc.      |                          |              |

Engagement tracking for non-retention interstitial  | Category 2               | PAY-3472     | subscriptions.interstitial_offer
offer journeys with entrypoint, step, outcome,      |                          |              |
error_reason, attribution + product fields          |                          |              |

6) Link to public documentation describing the ultimate data set
This collection is Glean - documented in the Glean Dictionary.

7) How long will this data be collected?
This collection will be collected permanently.
[e-mail] and [e-mail] will be responsible for the permanent collections.

8) What populations will you measure?
All channels, countries, and locales. No filters.

9) If default-on, what is the opt-out mechanism for users?
These collections are Glean. The opt-out can be found in the product's preferences. Session data will be recorded up until a user signs in with an account that has opted-out of metrics collection, or until they opt-out of collection, whichever comes first.

10) General description of how you will analyze this data
We would like to build dashboards and/or scheduled queries to:

• Analyze customer progression and conversion rates between lifecycle steps (view→engage→submit→result) within the retention_flow and interstitial_offer journeys
• Evaluate outcome distribution (success vs error) across retention flows and eligibilty_status, with detailed breakdowns by error_reason.
• Analyze flow participation and outcomes segment by entrypoint and associated utm_* (when present).
• Assess retention and offer effectiveness by product subscription (offering_id and interval)
• Establish baseline conversion and error metrics and monitor trends to detect deviations following releases or configuration changes.

11) Where will you share results?
The results of the analysis will be available to internal teams and NDA'd Mozillians only.

12) Any third-party tool proposed (not Glean/Telemetry)?
No - though this is for the client-side/frontend Glean (browser telemetry via @mozilla/glean/web), not server-side.

This Bugzilla report is mirrored in the FxA repository - https://github.com/mozilla/fxa/pull/20000

DATA COLLECTION REVIEW RESPONSE:

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the Glean Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. These collections are Glean. The opt-out can be found in the product's preferences.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, [e-mail] and [e-mail] will be responsible for the permanent collections.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data

5. Is the data collection request for default-on or default-off?

Default on for all channels, countries, and locales. No filters.

6. Does the instrumentation include the addition of any new identifiers?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

Result: data-review+