1434388 - [Shield] Pref Flip Study: HTTP response throttling algorithm v2

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Honza Bambas (:mayhemer)]

Basic description of experiment: 
Comparison of Intelligent HTTP response throttling algorithm v1 (enabled in the tree up to 57) and v2 (landed disabled in 59).

What is the preference we will be changing? 
network.http.throttle.version

What are the branches of the study and what values should each branch be set to? 
Control: the pref at 1, Treatment branch (only one): the pref at 2. 

What percentage of users do you want in each branch? 
I think 50% Nightly and 25% Beta population with split of Control(=v1)/Treatment(=v2) to 50/50 each. 

What Channels and locales do you intend to ship to? 
Nightly/Beta, all locales

What is your intended go live date and how long will the study run? 
Start aprox: soon
Run time: 2 weeks since this optimization may hit with lower probability (there has to be either a download running or some network activity of a background tab(s) during an active tab page load)

Are there specific criteria for participants? 
Nothing I can think of.

What is the main effect you are looking for and what data will you use to make these decisions? 
Following telemetry probes should observe lower mean/avg values with v2 vs v1:
TIME_TO_NON_BLANK_PAINT_NETOPT_MS
TIME_TO_DOM_CONTENT_LOADED_START_ACTIVE_NETOPT_MS
TIME_TO_LOAD_EVENT_START_ACTIVE_NETOPT_MS
TIME_TO_DOM_INTERACTIVE_MS

Who is the owner of the data analysis for this study? 
Me, Honza Bambas or anyone more suitable

Will this experiment require uplift? 
No

QA Status of your code: 
I can only provide try pushes with the v2 on:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=62d2998ae31f5d16ac89505f90def758a078f185 (Nightly, ~two months old)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d00a176762332343cad8f45aa30dea82d098e46d (Nightly, today)

Do you plan on surveying users at the end of the study?
No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
Just the bug reference for the v2 patches: https://bugzilla.mozilla.org/show_bug.cgi?id=1386746

Comment 1

[Ilana]

Science review: R+

Comment 2

[Honza Bambas (:mayhemer)]

Update: run this study only on the Beta channel (59).  Reasons:
- we would have to reenable some of the probes needed for results collecting
- simpler to do QA only on one channel prior shipping the study
- having results from Beta is pretty much enough valuable to have (Beta pop is usually more reflecting Release pop)

thanks

Comment 3

[Honza Bambas (:mayhemer)]

This has been QA-signed off by Emil Pasca.

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking this for 59, sounds like we will be ending the experiment March 2.

Comment 5

[Liz Henry (:lizzard) (relman/hg->git project)]

After more discussion, we can push this out to March 5th if you think you need it.

Comment 6

[Josh Gaunt [:jgaunt]]

This study has been disabled - analysis pending...

Comment 7

[Josh Gaunt [:jgaunt]]

sharing results from this study here: https://gist.github.com/jtg567/e91d3c5a676324e0698b1975838e60c7 there's no strong effect of the pref-flip on the probes between branches.

I'm glad to answer any followup questions here.

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

Has this study now ended? I'm marking it fixed for 59 since it was running in beta.  If we're definitely done, can you close the bug? Thanks!

Comment 9

[Matt Grimes [:Matt_G]]

This study has ended. Closing this bug!