1562545 - Pref-flip study of effect of fingerprinting protections on retention

Status: RESOLVED FIXED

(Data Science :: Experiment Collaboration, task)

Description

[Arthur Edelstein [:arthur]]

Brief Description of the request (required):

We want to investigate the effect of fingerprinter blocking on retention.

Business purpose for this request (required):

The fingerprinting list may cause breakage; we need to check this breakage isn't severe enough to cause problems for our users.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

We would like to run this study pretty soon.

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

A link to the experimenter entry will be posted here when it's ready.

Name of Data Scientist (If Applicable):

Requesting flawrence.

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

design-review+

--

• What is the goal of the effort the experiment is supporting?

Deploy fingerprinter blocking.

• Is an experiment a useful next step towards this goal?

Yeah!

• What is the hypothesis or research question? Are the consequences for the top-level goal clear if the hypothesis is confirmed or rejected?

The hypothesis is that fingerprinter blocking might cause breakage severe enough to lead to churn. We hope it doesn't! It if leads to a terrible UX, product could retool or decline to release the feature.

• Which measurements will be taken, and how do they support the hypothesis and goal? Are these measurements available in the targeted release channels? Has there been data steward review of the collection?

URI count and active_ticks. Yes and yes.

• Is the experiment design supported by an analysis plan? Is it adequate to answer the experimental questions?

Yes.

• Is the requested sample size supported by a power analysis that includes the core product metrics?

Yes.

• If the experiment is deployed to channels other than release, is it acceptable that the results will not be representative of the release population?

n/a

Comment 2

[Felix Lawrence]

Experimenter link

Comment 3

[Felix Lawrence]

Report

Comment 4

[Experimenter Bot]

[Experiment]Fingerprinting Protections Retention Study status has been changed to: Complete
            url:https://experimenter.services.mozilla.com/experiments/fingerprinting-protections-retention-study/

Comment 5

[Felix Lawrence]

Postscript: I tried segmenting the data by whether the profile was newish, and no convincing story jumped out of the data. The strongest effect was present for profiles of unknown age ("unknown" because of my sloppy segmentation approach that only looked for pings on the day of enrollment) - but the effect direction was the same for all three categories (new, old and unknown profile age).

There might be something down the "unknown profile" rabbit hole, but it's fairly likely to be the result of my methods of segmentation, rather than something actionable that could let us track down breakage or run more focussed investigations in the future. So I'll end this "proactive quick check" here.

Comment 6

[Felix Lawrence]

Public report