1391450 - [Shield] Pref Flip Study: Foreground Tracker Throttling

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[:Harald Kirschner :digitarald]

Basic description of experiment: By throttling trackers, we can incrementally improve user interactivity by reducing the time spent executing tracker callbacks. The idea is that this will help with our overall perceived performance goals. However, we’re not quite sure if it will have any noticeable effect and want to make sure it won’t break any experience such that it drops overall engagement. Breakage can cause never-ending page loads, when timers get throttled that are on the critical path for page completion.

What is the preference we will be changing? dom.min_tracking_timeout_value

What are the branches of the study and what values should each branch be set to? Control: PREF SET TO 4, Treatment: PREF SET TO 10000

What percentage of users do you want in each branch? 100% of beta of eligible population, split 50/50.

What Channels and locales do you intend to ship to? Beta 56, all locales

What is your intended go live date and how long will the study run? 2 weeks.  Aug 21st to Sep 3rd, 2017

Are there specific criteria for participants? Firefox Beta 56

What is the main effect you are looking for and what data will you use to make these decisions? The treatment group will see an improvement in TIMEOUT_EXECUTION_FG_TRACKING_MS. Less timers running will also improve input latency, best measured with INPUT_EVENT_RESPONSE_COALESCED_MS. Improvement in _MS probes can manifest as changes in count and duration. We will need additional data analyst help to understand the results.

Comparable or improved engagement metrics (total URIs, tabs opened, etc) and usage hours (measured via active ticks).

Possible breakages result in eternal page loads, for which we expect to see users to give up on the page load, measured in BUSY_TAB_ABANDONED. This metric is not well analyzed or validated in the past and will also benefit from data science.

We lack metrics for CPU or power consumption, which we would like to get on top off for future experiments.

Who is the owner of the data analysis for this study? Experiments team?

Will this experiment require uplift? No

QA Status of your code: Green

Do you plan on surveying users at the end of the study? Not for the first experiment as the performance improvements are hard to estimate.

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists: Nothing, really.

Comment 1

[:Harald Kirschner :digitarald]

Per Matt's request, looping in Josh.

Comment 2

[:Harald Kirschner :digitarald]

Per Ilana's request, a bug list for all listed telemetry.

BUSY_TAB_ABANDONED: Bug 1307689
TIMEOUT_EXECUTION_FG_TRACKING_MS: Bug 1355480
INPUT_EVENT_RESPONSE_COALESCED_MS: Bug 1357457

Ilana or Josh, please confirm that I send the Intent with the starting date for next week, Sep 4th.

Comment 3

[Josh Gaunt [:jgaunt]]

Sept. 4th is fine with us, Harald.

Since that's Labor Day in the US Dominic has offered to enable the recipe.

Comment 4

[:Harald Kirschner :digitarald]

OK, I will prepare the Intent mail.

Comment 5

[:Harald Kirschner :digitarald]

Per release-drivers thread, the new start date is the first week of the beta 57 cycle: Sep 25th to Oct 9th

:liz, could you confirm and give your approval here?

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

Redirecting to Sylvestre for 57

Comment 7

[Sylvestre Ledru [:Sylvestre]]

OK, let's do that!

Comment 8

[Matt Grimes [:Matt_G]]

This is now live in Beta 57 for all locales. The study enrolls 100% of the Beta population split 50/50 between treatment and control. I've set an end date of 10/11.

Comment 9

[:Harald Kirschner :digitarald]

Matt, as mentioned in the PHD we'd like some help analyzing the results.

Looking at the dashboards I wasn't able to skim off how significant the slight shifts in first-non-blank-paint were; just that treatment curve looks few pixels faster. Is it possible to tell from the charts or do we need to analyze the raw data?

Comment 10

[:Harald Kirschner :digitarald]

Working through it with jgaunt, who will run an analysis.

Comment 11

[Matt Grimes [:Matt_G]]

Hey folks. Now that this study has ended can you please recap the outcome here and close the bug? A couple sentences is fine.