Closed Bug 1462158 Opened 3 years ago Closed 2 years ago

[Shield] Pref Flip Study: Lower FPS Tab Throbber Study

Categories

(Shield :: Shield Study, enhancement)

Product:
Shield
Component:
Shield Study
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mconley, Unassigned, NeedInfo)

References

Details

User Story

Basic description of experiment

	We think that by lowering the frame rate of the tab throbber, the CPU will gain cycles to more quickly process and present web pages to the user, which should improve page load time in a perceivable way.

What is the preference we will be changing
        browser.tabs.30FpsThrobber
	browser.tabs.20FpsThrobber

What are the branches of the study and what values should each branch be set to?
	Branch 1 (value: “30fps”): Sets browser.tabs.30FpsThrobber to true, activating the 30fps throbber.
	Branch 2 (value: “20fps”): Sets browser.tabs.20FpsThrobber to true, activating the 20fps throbber.
	Control (value: “off”): Default fps throbber.

What percentage of users do you want in each branch
	1% of Beta, with an even split (33%) for Branch 1, 2 and Control, 

What Channels and locales do you intend to ship to?
	Beta 61, all locales.

What is your intended go live date and how long will the study run?
	Start on Monday May 21. Running for 3 weeks.

Are there specific criteria for participants?
	No.

What is the main effect you are looking for and what data will you use to make these decisions?
	We're looking for changes to the FX_PAGE_LOAD_MS  and / or TOTAL_CONTENT_PAGE_LOAD_TIME probes for the 5th percentile to see if we see at least a 5% improvement in page load time with either branch.

Who is the owner of the data analysis for this study?
	Ilana Segall

Will this experiment require uplift?
	No

QA Status of your code:
	Green

Do you plan on surveying users at the end of the study?
	Yes

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
	https://docs.google.com/document/d/1xHnXv-UZMC5n4PuqN64rgNQ9CfWmEBRPwHJasfNW2Bo/edit
No description provided.
Hey Ilana, did I get my User Story right?
Flags: needinfo?(isegall)
(In reply to Mike Conley (:mconley) (:⚙️) (Catching up on needinfos / reviews) from comment #1)
> Hey Ilana, did I get my User Story right?

yup!

I can also give R+ for science review here.
Flags: needinfo?(isegall)
Krupa, I believe you said Justin was going to follow up w/QA status?
Flags: needinfo?(kraj)
(In reply to Marnie Pasciuto-Wood [:marnie] from comment #3)
> Krupa, I believe you said Justin was going to follow up w/QA status?

Ciprian will be testing this SHIELD study. I can update this bug on Monday if the testing has been completed. If not, we need to push the release out.
Flags: needinfo?(kraj)
We have finished testing the Lower FPS Tab Throbber experiment.

QA’s recommendation: GREEN

Reasoning: We haven’t found any issues during testing.

Testing Summary:
- Verified that the correct animations appear with each pref enabled on multiple machines.
- Measured the load time until first paint using a scenario similar to the one from Bug 1406414 comment 9 on multiple machines. We've noticed a bit of improvement on our lower end machines, while we haven't seen much difference on our higher-end machines.
 
- We've tested each variant 3 times and calculated the average values as follows: Results (https://docs.google.com/document/d/12DvaIn0_YaffJt5nm9q8vtwMpJXa00MjLDiQkRV0_AQ/edit).
- From these results, it looks like 30fps is our sweet spot.
 
Tested Platforms:
- Windows 10 x64 - Dual Core
- Windows 7 x64, Windows 10 x64 - Octa Core
- Ubuntu 16.04 - Dual Core
- MacBook Pro early 2015 10.13.3 - Dual Core

Tested Firefox versions:
- Firefox 61 Beta 6 (61.0b6)
Krupa, do I need an explicit R+ from you on this, or is Ciprian's green light sufficient?

And if we're good from a QA perspective, we only need approval from RyanVM before we're good to launch.
Flags: needinfo?(ryanvm)
Flags: needinfo?(kraj)
(In reply to Marnie Pasciuto-Wood [:marnie] from comment #6)
> Krupa, do I need an explicit R+ from you on this, or is Ciprian's green
> light sufficient?
> 
> And if we're good from a QA perspective, we only need approval from RyanVM
> before we're good to launch.

Ciprian's signoff should be good enough. But since I am already here, ship it :)
Flags: needinfo?(kraj)
(In reply to Marnie Pasciuto-Wood [:marnie] from comment #6)
> And if we're good from a QA perspective, we only need approval from RyanVM
> before we're good to launch.

Approved.
Flags: needinfo?(ryanvm)
User Story: (updated)
This study is now live. You should see data coming in shortly. We've got it slated to end Jun 18th. If that date changes for any reason please let us know.
(In reply to Matt Grimes [:Matt_G] from comment #9)
> This study is now live. You should see data coming in shortly. We've got it
> slated to end Jun 18th. If that date changes for any reason please let us
> know.

Is this beta only or is Nightly 62 included?
Per mconley above, this is Beta 61 only.
(In reply to Matt Grimes [:Matt_G] from comment #11)
> Per mconley above, this is Beta 61 only.

Great
should be in addons-experiments?
(In reply to Matt Grimes [:Matt_G] from comment #9)
> This study is now live. You should see data coming in shortly. We've got it
> slated to end Jun 18th. If that date changes for any reason please let us
> know.

Running Beta61b11

cannot find the study
You have not participated in any studies.
and shows empty for any running ones.
Flags: needinfo?(mgrimes)
Per the User Story at the top of the page, it's only going out to 1% of Beta users.
Flags: needinfo?(mgrimes)
We do random assignment for studies to maintain population representivity. Since this only went out to 1% of the Beta population you are likely just not in that sample.
(In reply to Ryan VanderMeulen [:RyanVM] from comment #14)
> Per the User Story at the top of the page, it's only going out to 1% of Beta
> users.

(In reply to Matt Grimes [:Matt_G] from comment #15)
> We do random assignment for studies to maintain population representivity.
> Since this only went out to 1% of the Beta population you are likely just
> not in that sample.

Got it

was thinking was for everyone in beta
and % was for 20FPS/30FPS/default setting running on the beta
I've ended the study. Users should unenroll over the next 24 hours. We can close the bug once analysis is complete.
where can one find the study results.
Flags: needinfo?(mgrimes)
Flagging the analyst to post results. Ilana?
Flags: needinfo?(mgrimes) → needinfo?(isegall)
Is there any plan how this bug's result will feature into firefox
Flags: needinfo?(mgrimes)
https://gist.github.com/ilanasegall/a94ca02057565d3d70bac1969ab1c628

- No statistically significant change in retention between branches.

- For the 5th slowest percentile of load times (listed as 95th in the notebook, as the times are listed increasingly):

           - for FX_PAGE_LOAD_MS, the 30 fps branch performed significantly slower than 20 fps branch (1306 vs 785)
           - for TOTAL_CONTENT_PAGE_LOAD_TIME, there was no significant difference between the branches.
Flags: needinfo?(isegall)
We re-sliced the data a little, and started looking only at machines that have 2 or fewer CPUs.

We have a new gist here:

https://gist.github.com/ilanasegall/662fda318928e7f74aad37ca50caf4aa


Interestingly, looking at the the slowest page loads (so the 95th percentile, or the 5th slowest percentile), both the 20fps and 30fps throbbing groups seemed to experience longer load times for those slowest page, at least from the perspective of the content process / network layer.


FX_PAGE_LOAD_MS
95th Percentile interval for 20fps: 10000.0 (10000.0, 10000.0)
95th Percentile interval for 30fps: 10000.0 (10000.0, 10000.0)
95th Percentile interval for Control: 10000.0 (10000.0, 10000.0)

TOTAL_CONTENT_PAGE_LOAD_TIME
95th Percentile interval for 20fps: 14925.0 (14925.0, 14925.0)
95th Percentile interval for 30fps: 14925.0 (14925.0, 14925.0)
95th Percentile interval for Control: 14081.0 (14081.0, 14081.0)


However, if we look at the 80th percentile:


FX_PAGE_LOAD_MS
80th Percentile interval for 20fps: 6011.0 (6011.0, 6011.0)
80th Percentile interval for 30fps: 3613.0 (3613.0, 3613.0)
80th Percentile interval for Control: 3613.0 (3613.0, 6011.0)

TOTAL_CONTENT_PAGE_LOAD_TIME
80th Percentile interval for 20fps: 3484.0 (3484.0, 3484.0)
80th Percentile interval for 30fps: 3287.0 (3287.0, 3287.0)
80th Percentile interval for Control: 3484.0 (3484.0, 3484.0)


We see that quicker page loads seem to be... quicker for the 30fps case. Not by a huge margin, but it's there. 20fps doesn't appear to make a difference in the content process, but the parent process sees a rather large bump in FX_PAGE_LOAD_MS for some reason.


For the 50th percentile:

FX_PAGE_LOAD_MS
50th Percentile interval for 20fps: 1306.0 (1306.0, 1306.0)
50th Percentile interval for 30fps: 1306.0 (1306.0, 1306.0)
50th Percentile interval for Control: 1306.0 (1306.0, 1306.0)

TOTAL_CONTENT_PAGE_LOAD_TIME
50th Percentile interval for 20fps: 914.0 (914.0, 914.0)
50th Percentile interval for 30fps: 914.0 (914.0, 914.0)
50th Percentile interval for Control: 969.0 (969.0, 969.0)


The content process sees both 20fps and 30fps having an impact on page load times.

In terms of retention, 30fps is right in the noise and is not significantly different from the control group. 20fps is right on the line of being statistically significant.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.