1574618 - Support for pref-flip study to asses Fenix tracking protection modes

Status: RESOLVED FIXED

(Data Science :: Experiment Collaboration, task)

Description

[esmyth]

Brief Description of the request (required):

The Fenix team would like to understand how tracking protection modes impact page load performance and how a change in the default mode may impact user engagement. To inform this decision, we intend to conduct an experiment to test the hypothesis that Tracking Protection has perceptibly faster page loads than Enhanced Tracking Protection and that changes to the default mode will measurably impact user engagement.

Business purpose for this request (required):

There are product benefits from both modes. However we need to better understand how the modes affect user behavior, especially whether a change in the default behavior would impact 7-day retention of new users, churn of existing users, and user engagement (e.g., total_uri_count or subsession_length).

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Start the experiment on or after September 19

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

• Glean-based performance telemetry
• preference_toggled telemetry probe

Name of Data Scientist (If Applicable):

Corey Dow-Hygelund

Comment 1

[Corey Dow-Hygelund [:ccd]]

Reference material regarding mobile experiments and Mako.

Comment 2

[Corey Dow-Hygelund [:ccd]]

Sample size analysis for new user retention and churn has been calculated. Sampling 23K clients, which is ~30% of available experiment population, yields:

• 7-day new user retention:
  • < 5% measurable difference
  • 0.005 Type I Error (false positive)
  • 0.10 Type II Error (false negative)
• 7-day churn (i.e., 1 - existing user retention):
  • < 10% measurable difference
  • 0.005 Type I Error
  • 0.10 Type II Error

As 10% is quite large, this seems like the smallest viable sample size to satisfy that requirement.

Decreasing the measurable difference in new user retention to 1% increases the necessary sample size to ~70K clients. This is 84% of available experiment population. Therefore, that is close to the maximum sensitivity we can achieve with new user retention in Fenix.

Engagement metrics will be added shortly.

Comment 3

[Corey Dow-Hygelund [:ccd]]

Comparison of navigation timings to visual metrics. Idea would be to utilize a linear combo of timings as a means of getting user-perceived page load metrics.

Comment 4

[Corey Dow-Hygelund [:ccd]]

Engagements metrics have been added. Requesting 85% of population.

Comment 5

[Corey Dow-Hygelund [:ccd]]

Experiment is set to launch October 15th or 31st.

Comment 6

[Corey Dow-Hygelund [:ccd]]

Experiment has been launched. Notebook for data pull and initial analysis.

Comment 7

[Corey Dow-Hygelund [:ccd]]

Initial Analysis

Comment 8

[Corey Dow-Hygelund [:ccd]]

Report drafted and sent for review.

Comment 9

[Corey Dow-Hygelund [:ccd]]

Final report