Closed Bug 1434388 Opened 2 years ago Closed 2 years ago

[Shield] Pref Flip Study: HTTP response throttling algorithm v2

Categories

(Shield :: Shield Study, defect)

defect
Not set

Tracking

(firefox59+ fixed)

RESOLVED FIXED
Tracking Status
firefox59 + fixed

People

(Reporter: mayhemer, Assigned: mayhemer)

References

()

Details

Basic description of experiment: 
Comparison of Intelligent HTTP response throttling algorithm v1 (enabled in the tree up to 57) and v2 (landed disabled in 59).

What is the preference we will be changing? 
network.http.throttle.version

What are the branches of the study and what values should each branch be set to? 
Control: the pref at 1, Treatment branch (only one): the pref at 2. 

What percentage of users do you want in each branch? 
I think 50% Nightly and 25% Beta population with split of Control(=v1)/Treatment(=v2) to 50/50 each. 

What Channels and locales do you intend to ship to? 
Nightly/Beta, all locales

What is your intended go live date and how long will the study run? 
Start aprox: soon
Run time: 2 weeks since this optimization may hit with lower probability (there has to be either a download running or some network activity of a background tab(s) during an active tab page load)

Are there specific criteria for participants? 
Nothing I can think of.

What is the main effect you are looking for and what data will you use to make these decisions? 
Following telemetry probes should observe lower mean/avg values with v2 vs v1:
TIME_TO_NON_BLANK_PAINT_NETOPT_MS
TIME_TO_DOM_CONTENT_LOADED_START_ACTIVE_NETOPT_MS
TIME_TO_LOAD_EVENT_START_ACTIVE_NETOPT_MS
TIME_TO_DOM_INTERACTIVE_MS

Who is the owner of the data analysis for this study? 
Me, Honza Bambas or anyone more suitable

Will this experiment require uplift? 
No

QA Status of your code: 
I can only provide try pushes with the v2 on:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=62d2998ae31f5d16ac89505f90def758a078f185 (Nightly, ~two months old)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d00a176762332343cad8f45aa30dea82d098e46d (Nightly, today)

Do you plan on surveying users at the end of the study?
No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
Just the bug reference for the v2 patches: https://bugzilla.mozilla.org/show_bug.cgi?id=1386746
Flags: needinfo?(isegall)
Science review: R+
Flags: needinfo?(isegall)
Update: run this study only on the Beta channel (59).  Reasons:
- we would have to reenable some of the probes needed for results collecting
- simpler to do QA only on one channel prior shipping the study
- having results from Beta is pretty much enough valuable to have (Beta pop is usually more reflecting Release pop)

thanks
This has been QA-signed off by Emil Pasca.
Tracking this for 59, sounds like we will be ending the experiment March 2.
After more discussion, we can push this out to March 5th if you think you need it.
This study has been disabled - analysis pending...
sharing results from this study here: https://gist.github.com/jtg567/e91d3c5a676324e0698b1975838e60c7 there's no strong effect of the pref-flip on the probes between branches.

I'm glad to answer any followup questions here.
Has this study now ended? I'm marking it fixed for 59 since it was running in beta.  If we're definitely done, can you close the bug? Thanks!
This study has ended. Closing this bug!
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.