Open Bug 1119891 Opened 5 years ago Updated 4 years ago

[Polaris] Breakdown - Telemetry Experiment 1 for Tracking Protection

Categories

(Firefox :: General, defect)

x86
All
defect
Not set
Points:
5

Tracking

()

ASSIGNED

People

(Reporter: javaun, Assigned: javaun)

References

Details

User Story

As the Polaris team, we need an experiment design for a Telemetry experiment for Tracking Protection to help us understand:
* Do TP affect usage of Firefox?
* How often are loads blocked? 
* How often do users interact with the UI to correct page load issues?
* Can an alternate UI treatment improve user awareness of when content is blocked?
* Does TP boost user sentiment and rating of Fx? (user advocacy survey and numerical rating served via experiment)
As the Polaris team, we need to expand and monitor usage of the Tracking Protection feature, complementing user research, so that we can observe:
* if and where breakage occurs,
* how users interact with the UI,
* how they rate the TP feature, and
* whether TP impacts usage or abandonment of Nightly.

Requirements. This initial TP telemetry experiment will have 3 cohorts:
1. Existing users who have already enabled Tracking Protection in about:config.
2. Users with DNT enabled (but not TP enabled) who will be recruited to TP via a door hanger  (target 10% of DNT enabled users)
3. Users with DNT enabled who will not be offered TP (A baseline control to help understand usage patterns. Target: 10% of DNT enabled users)

We seek to answer the following questions via Telemetry data:
* Does TP affect usage of Fx Nightly? For example, does cohort 2 user Fx more/less than cohort 3?
* Do users disable TP? When? 
* How often do users interact with TP navigation bar UI (enable/disable blocking)? Which sites?
* How do users feel about TP and Fx. Accomplished through a survey and numerical rating popped after a TBD period of elapsed time.

Work to accomplish:
* Create telemetry code/filters to create cohorts and run experiment.
* Create the frontend code to display door hanger that allows cohort 2 to turn on  TP
* Pop a survey for users to gather their rating and qualitative feedback on Firefox (question, TBD)
*  Confirm proper instrumentation of telemetry probes and messages to answer the questions above.
OS: Mac OS X → All
Whiteboard: [meta]
Flags: qe-verify-
Flags: firefox-backlog+
Summary: [Polaris] [Meta] Tracking Bug for Telemetry Experiment 1 for Tracking Protection in Nightly → [Polaris] Breakdown - Telemetry Experiment 1 for Tracking Protection in Nightly
Whiteboard: [meta]
I spoke with Chad. We are likely going to try to do this in the Beta channel to get a better cross-section of users. Scratch the three requirements in the description.

This breakdown will focus on the experiment design and identifying resulting work items: Design work for alternate UI treatments, Telemetry and filter coding, front end development, probe instrumentation, and user advocacy group work.
User Story: (updated)
Summary: [Polaris] Breakdown - Telemetry Experiment 1 for Tracking Protection in Nightly → [Polaris] Breakdown - Telemetry Experiment 1 for Tracking Protection
Assignee: nobody → jmoradi
Status: NEW → ASSIGNED
Points: --- → 5
(In reply to Javaun Moradi [:javaun] from comment #1)
> I spoke with Chad. We are likely going to try to do this in the Beta channel
> to get a better cross-section of users. 

Great, I will file a bug to get rid of the #ifdef NIGHTLYs that are currently surrounding the polaris prefs.


> Scratch the three requirements in
> the description.
> 
> This breakdown will focus on the experiment design and identifying resulting
> work items: Design work for alternate UI treatments,

Why are we doing design work for alternate UI treatments before testing the first iteration of the UI?

> Telemetry and filter
> coding, 

I made a telemetry dashboard for this feature at people.mozilla.org/~mchew/tracking-protection. It shows that for users who have tracking protection enabled,

* approximately 15% of page views are affected by tracking protection
* Of page views affected by tracking protection, approximately 1% of those have had tracking protection disabled
* There are fewer than 10 disable events per day

Approximately 0.5% of sessions from Nightly users (~140K) have tracking protection enabled.

I filed bug 1122122 to count the number of times the doorhanger is shown if the user has enabled tracking protection. If there are any other metrics you want, please file bugs on those.

> front end development, probe instrumentation, and user advocacy
> group work.
No longer blocks: 1029886
You need to log in before you can comment on or make changes to this bug.