1380584 - [Shield] Pref Flip Study: Shutdown decoder

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[chsiang]

[Background]
Media team built a feature called shutdown decoder. It will shutdown the video decoder when user brings a media tab to the background for more than 10s. Video will resume when the tab is brought back to the foreground. Audio will not be affected. It is to enhance the performance of the foreground tab and save battery.

It is now feature complete and can be shipped in 55. The telemetry data shows that around 10% of beta users experience >0.5s of delay when they bring the video back to the foreground. This number increases as the resolution of the video gets higher. We would like to run a shield study in 55 and compare the usage time against a controlled group.


[Segment]
A: Users who have shutdown decoder enabled
B: Users who don't have shutdown decoder enabled


[Measurement]
To quantify success, we need to make sure the total usage time of A is >= the total usage time of B

[Time]
We would like to run this test in release 55 for two weeks and compare results.


[Link to telemetry data]
https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=1&e10s=true&end_date=2017-06-29&keys=All!AV%252C241-480!AV%252C481-720!AV%252C721-1080&max_channel_version=nightly%252F56&measure=VIDEO_SUSPEND_RECOVERY_TIME_MS&min_channel_version=nightly%252F53&os=Windows_NT%252C10.0&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-06-12&table=0&trim=1&use_submission_date=0

Comment 1

[chsiang]

Had a call with Matt today to go through PHD and pasted it under the url section.

Comment 2

[chsiang]

Just checking where this is in the process and if it's set to go with 55. 
Plan to send out the intent-to-ship email late this week/ early next week.

Thanks.

Comment 3

[Blake Wu [:bwu][:blakewu]]

I assume we will need to turn off the preference of shutdown decoder on Desktop. Please tell us know if it is not that case.

Comment 4

[chsiang]

(In reply to Blake Wu [:bwu][:blakewu] from comment #3)
> I assume we will need to turn off the preference of shutdown decoder on
> Desktop. Please tell us know if it is not that case.

That's right.

Comment 5

[Matt Grimes [:Matt_G]]

You are all lined up for 55. We just need an intent to ship email. The details can be found in the document I sent you. They will ask if the feature has passed formal QA, so please be sure that is done prior to the intent to ship. I'm adding Josh from my team as well. If you need data to help you make a final decision that is not represented in the Experiments Viewer dashboard he can coordinate that.

Comment 6

[chsiang]

Cool. Thanks Matt!
Hi Bobby,
Could you send the intent to ship email after confirming the feature has passed QA?

Comment 7

[Bobby Chien]

Sending Intent to ship email soon.

Comment 8

[Bobby Chien]

Sent the Intent to Ship email. 

Matt, as offline chat with release manager, RM will also ask the verification of deploying shield study. Do we have to stage for testing? or your team could verify the deployment? Thanks.

Comment 9

[Matt Grimes [:Matt_G]]

If we get sign off from RM then someone from my team will push the experiment live on Aug 8th and confirm that it has been deployed here in the bug. There is no staging process required.

Comment 10

[Bobby Chien]

Thanks, Matt.

Comment 11

[Bobby Chien]

As Ritu's reply (on behalf of Julien), we are good to start the shield study. 

Timeline: Aug-15-2017 ~ Aug-29-2017
Maximun Runtime:  2 weeks
Update Channels: Firefox 55 Release [Desktop]
Sample Rate: 5% of Firefox 55 release users

Comment 12

[Matt Grimes [:Matt_G]]

This experiment is live. Data should start coming in to experiments viewer in the next 24 hours.

Comment 13

[Bobby Chien]

Thanks, Matt.(In reply to Matt Grimes [:Matt_G] from comment #12)
> This experiment is live. Data should start coming in to experiments viewer
> in the next 24 hours.

Thanks, Matt.

Comment 14

[chsiang]

(In reply to Bobby Chien [:bchien] from comment #13)
> Thanks, Matt.(In reply to Matt Grimes [:Matt_G] from comment #12)
> > This experiment is live. Data should start coming in to experiments viewer
> > in the next 24 hours.
> 
> Thanks, Matt.

hooray!!!

Comment 15

[Bobby Chien]

Hi Matt, 

I understand we haven't finish the study yet. However, do we have recently results or data site we could query/preview the study data? Many thanks.

Comment 16

[Matt Grimes [:Matt_G]]

Hey Bobby. You can see the current results here: https://moz-experiments-viewer.herokuapp.com/ 

However, this dashboard does not cover all probes. You should be able to see unique URI counts, but not usage hours. If that's not something you feel comfortable pulling yourself, Ilana can probably help once the experiment ends.

Comment 17

[Ilana]

Bobby, you can actually let me know whenever you'd like which probes you'd like tracked that aren't in the experiments viewer. If you'd like to set up some time to talk about which probes might be helpful, please do so!

Comment 18

[chsiang]

Hi Ilana,
I see that uri count for both groups are sitting at ~15 million. 
But it would be great to know if there are statistically significant differences in page views & usage time between the Control and the Treatment Groups. 

Thanks.
Cindy

Comment 19

[Ilana]

Cindy, 
The issue with such a large dataset is that pretty much every result will be "significant." However, we can measure if one differs from the other by a certain margin with a large degree of certainty - say 1% or 10%. (What significance would tell us is something like "we're really, really sure that the mean of this group is larger than the mean of that group by .0000001.") How much difference would you need to see between the two groups in order to register them as different enough to worry about?

Comment 20

[chsiang]

Hi Ilana,
On a hunch, I would like to see the difference smaller than 5%.

Comment 21

[chsiang]

HI Ilana, any update on this?

Comment 22

[Matt Grimes [:Matt_G]]

I plan on ending this experiment today. Ilana should have everything she needs to complete the analysis.

Comment 23

[Ilana]

chsiang, i have everything i need. i'll take a look at the outcome this week.