1369150 - [Shield] Pref flip study for ScreenShots

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Comment 1

[Mahendra Potharaju [:mahe]]

**Details in the works**

1. What is the preference we’ll be changing?

2. What are the values of each branch of the study? You can have multiple branches, but every study must include a control.
	
3. What percentage of users do you want in each branch? 
	
4. What Channels/locales do you intend to ship to?
	Beta. Not preference on locales.
5. What is your intended go live date and how long will the study run? 
	6/19 - 6/27. 
6. Are there specific criteria for participants? We can include/exclude based on anything available in telemetry.
	Windows users
7. What is the main effect you are looking for?  
Confirm that Screenshots does not adversely affect Firefox usage. Confirm that experimental cohort shows no significant difference with respect to usage hours, page views, default browser toggles, and crashes.

Screenshots is a test pilot experiment. Other necessary details about UR results, tests etc will be provided.

Comment 2

[John Gruen [:jgruen]]

(In reply to Mahendra Potharaju [:mahe] from comment #1)
> **Details in the works**
> 
> 1. What is the preference we’ll be changing?
> 
extensions.screenshots.system-disabled
> 2. What are the values of each branch of the study? You can have multiple
> branches, but every study must include a control.
> 

The branch is toggled to false by default. We want the experimental cohort to be set to true.

> 3. What percentage of users do you want in each branch? 

We'd like a staged roll out of the experiment to a large % of beta users over a few weeks. 

goals are as follows: 

- verify our ops perform under load
- verify that the add-on does no harm WRT browser page views, searches, and time in browser

Comment 3

[Mahendra Potharaju [:mahe]]

Hey Gregg, 

Can you please check this bug and see if necessary information is present?

Comment 4

[Matt Grimes [:Matt_G]]

We'll need to figure out what "large % of beta users" looks like in practice, but I think we can do that when we meet and then update the bug accordingly. The pref flipping action hits beta this week and then rides the train to Release in 54. That means we should be ready for both the Beta test and our Release experiment in 55.

Comment 5

[John Gruen [:jgruen]]

A few notes, the experiment should be restricted to en/en-* locales, no geo restrictions required

Let's do a 50/50 control/experiment split:

19th - 22nd: release to 2% of beta (1% in experiment)
22nd - 29th: release to 20% of beta (10% in experiment)
29th - close of beta window (8/8) release to 100% of beta (50% in experiment)

Comment 6

[John Gruen [:jgruen]]

Matt G. This makes sense right? LMK and I'll bang out an intent to ship email.

Comment 7

[Matt Grimes [:Matt_G]]

Looks good to me. You'll need to send an update to Release Drivers before each of those population changes, so give them at least a few days notice.

Comment 8

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Claim: the pref should UNSET by default, **NOT** false.

Comment 9

[John Gruen [:jgruen]]

Hey gregg, Can you provide a rationale here? We're a right at the end of 55 Nightly so making any changes now would inject significant chaos into the process.

Comment 10

[John Gruen [:jgruen]]

Mahe, let's push this back a week since we need uplift into beta for some last minute features...dates are now:

June 26th - 29nd: release to 2% of beta (1% in experiment)
June 29nd - July 6th: release to 20% of beta (10% in experiment)
July 6th - close of beta window (8/8) release to 100% of beta (50% in experiment)

Comment 11

[Mahendra Potharaju [:mahe]]

Done JGruen. Sent out the intent to ship email. Will update the ticket with RD response.

Comment 12

[Matt Grimes [:Matt_G]]

Bumped to 20% of Beta per John.

Comment 13

[Matt Grimes [:Matt_G]]

Bumped to 100% of Beta split 50/50

Comment 14

[Mahendra Potharaju [:mahe]]

Launch plan for release

9/6 : 12.5%  (25% roll out with 50-50 split)
9/13 : 25% (50% roll out with 50-50 split)
9/20 : 50% (100% roll out with 50-50 split)

Comment 15

[Matt Grimes [:Matt_G]]

Update is live to roll screenshots out to 50% of Release with a 1/1 ratio treatment to control.

Comment 16

[Matt Grimes [:Matt_G]]

I think we've completed all phases of the rollout. Closing this bug.