1518855 - Default-On Cookie Restrictions Rollout: Mid-65 Release

Status: RESOLVED FIXED

(Data Science :: Experiment Collaboration, task)

Description

[Arthur Edelstein [:arthur]]

Brief Description of the request:

Requesting assistance with monitoring effects of basic-list cookie restrictions as they are gradually rolled out.

Any timelines for the request or how this fits into roadmaps:

Rollout expected to begin on 2019/2/11.

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

I will provide a Mozilla Experimenter link here, if possible.

Name of Data Scientist (If Applicable):
flawrence

Comment 1

[Arthur Edelstein [:arthur]]

There are two experiments. Drafts are here:
https://experimenter.services.mozilla.com/experiments/mid-65-default-on-cookie-restrictions-for-existing-users/
https://experimenter.services.mozilla.com/experiments/mid-65-default-on-cookie-restrictions-for-new-users/

Comment 2

[Felix Lawrence]

Per yesterday's meeting with arthur and pdol, there is some important background context and multiple goals.

The background context is that we have been running a large scale experiment to test this change. So far, looking at hours of usage, URI count and search volume, there don't seem to be any notable negative impacts on retention in users 1-2, 2-3 or 3-4 weeks after enrolling. So we're confident that this change doesn't have large or even medium-sized negative effects on general usage, so we're not going to churn many users if we roll this feature out to a large sample size in a future experiment.

This bug has three somewhat-separate goals:

1. We think that negative revenue effects from cookie restrictions might be most visible on new users, so we want to run an experiment on lots of new users to gather as much information as we can in the time available.
2. Let's gather some more info on the revenue effects on existing users - we can go to a larger sample size than the existing experiment because we're confident the feature won't churn users.
3. Let's roll the functionality out to ~5% of users so that we can announce progress on the rollout, and gather feedback in the form of anecdotal breakage reports via bugzilla, twitter, etc. This goal could be met by running a large-scale experiment, or by a non-experiment rollout, or some mixture of the two.

Comment 3

[Felix Lawrence]

Quick n' dirty first interim report

Comment 4

[Felix Lawrence]

Interim report 20190315

Comment 5

[Felix Lawrence]

Interim report 20190329

Comment 6

[Felix Lawrence]

Final report