1377267 - SBv4 Crashrate opt-out study

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement, P1)

Description

[François Marier [:francois]]

> What is the preference we’ll be changing?
> What are the values of each branch of the study?

V2:

pref("urlclassifier.malwareTable", "goog-malware-shavar,goog-unwanted-shavar,test-malware-simple,test-unwanted-simple");
pref("urlclassifier.phishTable", "goog-phish-shavar,test-phish-simple");
pref("urlclassifier.downloadAllowTable", "goog-downloadwhite-digest256");
pref("urlclassifier.downloadBlockTable", "goog-badbinurl-shavar");

V4:

pref("urlclassifier.malwareTable", "goog-malware-proto,goog-unwanted-proto,test-malware-simple,test-unwanted-simple");
pref("urlclassifier.phishTable", "goog-phish-proto,test-phish-simple");
pref("urlclassifier.downloadAllowTable", "goog-downloadwhite-proto");
pref("urlclassifier.downloadBlockTable", "goog-badbinurl-proto");


> What percentage of users do you want in each branch?

1% of users overall, split 50/50 between V2 and V4.

> What Channels/locales do you intend to ship to?
> What is your intended go live date and how long will the study run?

We're 100% V4 in Nightly at the moment. We're planning to go 100% V4 in Beta 56.

Then 100% V2 in release 56, until the experiment concludes there. After reviewing the results of the experiment, we can hopefully go 100% V4 on release too.

We would like to run the experiment for 1 week in each channel.

> Are there specific criteria for participants?
> What is the main effect you are looking for?  

We are only interested in comparing the crash rate between the V2 users and the V4 users.

Hopefully V4 will have the same crash rate (or less) as V2.

> Who is the owner of the data analysis for this study?

Francois Marier (module owner)

> Do you plan on surveying users at the end of the study?

No.

> Link to any relevant google docs / Drive files that describe the project.

https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation

Comment 1

[Kamyar Ardekani (he/him)]

Just to update the status of this bug, per the discussions I have had with Francois, we cannot currently flip multiple prefs at the same time using Shield. So, we will wait for now and proceed with this study once the opt-out studies are enabled.

Comment 2

[François Marier [:francois]]

Since we are going to be 100% V4 on Beta 56 (the patch is currently riding the train), we'd like to have 100% of the beta population in this experiment.

Then we will put half of the study population on the old code (V2) and then the other half will stay on the new code (V4).

There should be no user-visible changes at all. We are interested in comparing the crash rate of V2 v. V4.

Comment 3

[François Marier [:francois]]

The study has run and is now finished.

Kamyar is analyzing the results.

Comment 4

[François Marier [:francois]]

We have gone to 100% on release 56 (bug 1387651) and haven't seen any problems so far.

No further analysis is needed.