1491171 - [Shield] Push Performance Shield Study, release 62 and 63

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Julie McCracken (:julie)]

What is your basic hypothesis?

We are using the shield system to aid in determining if Push can handle the potential increased traffic load for upcoming proposed services.  

What high level metric are you attempting to influence? 

Unique page views No
Usage hours No
Attracting heavy users and influencers. No
Understanding the landscape of the web (research) No

None of the above, we're trying to determine that we are NOT impacting UA performance by introducing this feature. 

Do any of the following apply to your experiment:

Partner related No
High Risk to brand or product  No
Needs to be shipped ASAP (outside normal Shield process) No
Mozilla confidential (No public bugs or notifications) No
Affects a large population of users on Release Yes

If you answered yes to any of the above, your study is considered “High Risk” and will require an executive sponsor to sign off explicitly in the bug and state the known risk. 

RASCI

Final Experiment Design:
Responsible: Ben Bangert (primary/lead) JR Conlin, Phil Jenvey
Accountable: Shield Team

Population Size:
Responsible: Ben Bangert (primary/lead) JR Conlin, Phil Jenvey
Accountable: SHIELD Team 
 

Data Analysis:
Responsible: Ben Bangert (primary/lead) JR Conlin, Phil Jenvey
Accountable: SHIELD Team

Legal Sign-Off
Responsible: Michael Feldman
Accountable: SHIELD

Shipping:
Responsible: Release Management
Accountable: Shield Team
 
Risk Matrix
Responsible: Ben Bangert (primary/lead) JR Conlin, Phil Jenvey
Accountable: Shield Team

Basic description of experiment: 

Currently, the push connection is activated from the Firefox browser only if the user is opted in. We would like to verify with certainty that the system can handle the load if all of firefox user base is connected to it.

What is the preference we will be changing? dom.push.alwaysConnect

What are the branches of the study and what values should each branch be set to? 
This will be a "ramp-up" process across Release. A "control" would be users who do not have the option set and who are not included in the Broadcast service. We are looking to leverage shield for this feature control for both availability and for population management. 

What percentage of users do you want in each branch? 
The study would start with with an initial population of 20% (with 15% in study, 5% out of study) and ramp up each week as the backend and shield systems concur for success. (Reaching 100% inclusion by 6 weeks)

What Channels and locales do you intend to ship to.
Since Push is a released feature, we're looking at targeting all locales for Releases 62 and 63. 

What is your intended go live date and how long will the study run?
Go live date is as soon as Shield can start it.

We expect that the experiment will last for approximately 6 weeks to reach 100% user reach. After which, the experiment can be terminated and the service persist.

Are there specific criteria for participants? 
No criteria (other than active user)

What is the main effect you are looking for and what data will you use to make these decisions? 
Metrics will be monitored by backend tracking, (via Datadog and other process monitoring systems)

Who is the owner of the data analysis for this study?
Ben Bangert

Will this experiment require uplift? 
No

QA Status of your code: 
Code is currently green, per https://github.com/mozilla-services/autopush/ and operation monitoring.

Do you plan on surveying users at the end of the study?
No, ideally, users will be unaware of the study.

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

Push Privacy Concerns and Answers: https://docs.google.com/document/d/19lzYseyo1YRhGb88m0j-tNXgsBe_0zhQmLOk-Wo83Lg/edit

RFC: 8030 Generic Event Delivery using HTTP Push: http://www.rfc-archive.org/getrfc.php?rfc=8030

Comment 1

[Julie McCracken (:julie)]

Attachment: Push Shield Study.pdf

Comment 2

[Michael Feldman (use NI)]

I talked with Julie and JR this morning. 

My understanding is that this study will not change the number, frequency, kind, etc. of push notifications users receive. This is a slow ramp up of rolling out a new backend for the Push system for the current pushes users receive. 

That's low legal risk. Legal approved, go ahead. 

Thanks.

Comment 3

[Chris Hartjes [:grumpy][:chartjes]]

We have a test plan to do continuous testing of the Push service on Tuesdays during the Shield study to verify that we can do deployments under the expected load.

QA signs off on going forward with the study.

Comment 4

[MarniePW [:marnie]]

Dave, this will need a peer review. Are you comfortable with Ben Bangert in that role (as one of the devs) or would you like to assign another peer?

Comment 5

[Dave Townsend [:mossop]]

Lina, looks like you've been heavily involved in this code. Can you give peer sign-off for this study to go ahead. Basically will toggling this pref negatively impact Firefox in ways that we look for in normal reviews?

Comment 6

[Dave Townsend [:mossop]]

See comment 5

Comment 7

[Lina Butler (ex-Mozilla)]

Approved! :-) Thanks, Dave.

Comment 8

[MarniePW [:marnie]]

Julien, Ryan, Pascal, can you review this study for release management? They'd like to launch Monday, October 8 to release 62 and then release 63.

Comment 9

[Julien Cristau [:jcristau]]

Sounds good to me, approving for relman.

Comment 10

[JR Conlin [:jrconlin,:jconlin]]

Blog Post describing upcoming shield study

https://blog.mozilla.org/services/2018/10/03/upcoming-push-shield-study/

Comment 11

[MarniePW [:marnie]]

Ilana, looks like your questions were answered in the PHD. Can you review and approve if complete? Thanks.

Comment 12

[Julie McCracken (:julie)]

Dave, I believe Ben Bangert has given you a head's up about the Push Performance Shield study described in this bug. Because it includes all user clients in Firefox 62, we need your sign-off before it can be launched on Monday, October 8th. The Intent to Ship email was sent October 2nd and includes links to additional related documentation, should you want to read more about why this performance study is advised.

May we have your sign-off to proceed with the launch of this study?

Comment 13

[Julie McCracken (:julie)]

I am changing the NI for VP sign off from dcamp to osunick. Nick, could you please review for your approval? Ben Bangert can answer any questions.

Comment 14

[Nick Nguyen [:osunick]]

Approved

Comment 15

[Julie McCracken (:julie)]

Ilana, the desired launch date is today. Do we have your approval? If not, what else is it you need from the team? Thanks

Comment 16

[Ilana]

Science: R+

Comment 17

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Enabled.

To disable, close 597.

Comment 18

[Ben Bangert [:benbangert]]

The study has been successfully pref'd to 0.20. We would like to increase it by 20% each week. I can file a request to increase it here if that works for everyone.

Comment 19

[Julie McCracken (:julie)]

Please increase the study participation by 20%, as indicated in the approved study PHD:

"The study would start with with an initial population of 20% (with 15% in study, 5% out of study) and ramp up each week as the backend and shield systems concur for success. (Reaching 100% inclusion by 6 weeks")

Comment 20

[Josh Gaunt [:jgaunt]]

Sampling rate increased from 20% to 40%

Comment 21

[Julie McCracken (:julie)]

Is there someone you'd like for me to NI each time I make this request?

Comment 22

[Philip Jenvey [:pjenvey]]

Please increase the rate by another 20% for this week (40% -> 60%)

Comment 23

[Josh Gaunt [:jgaunt]]

Sampling rate increased from 40% to 60%.

Feel free to NI me, Julie, since I'm familiar with the study/recipe by now.

Comment 24

[Julie McCracken (:julie)]

Please increase the rate by another 20% for this week (60% -> 80%). Thanks!

Comment 25

[Josh Gaunt [:jgaunt]]

Of course! Sampling rate increased from 60% to 80%.

Comment 26

[Julie McCracken (:julie)]

Please increase the rate by another 20% for this week (80% -> 100%). Thanks!

Comment 27

[Josh Gaunt [:jgaunt]]

Sampling rate increased from 80% to 100%.

Comment 28

[Ryan VanderMeulen [:RyanVM]]

Hi Ben, I noticed today that we're still enrolling users in this study. Can we disable it now or least restrict it to versions <64?

Comment 29

[Ben Bangert [:benbangert]]

Is there a resource cost associated with enrolling users? We're going to switch the pref to defaulting on, as our system have handled this fine and we'd like to retain the broadcast benefit for Remote Settings.

Comment 30

[Ryan VanderMeulen [:RyanVM]]

From #c0:

We expect that the experiment will last for approximately 6 weeks to reach 100% user reach. After which, the experiment can be terminated and the service persist.

It doesn't feel like we should be using a study mechanism for long-term changes like this. Please file a follow-up bug for flipping this pref so we can terminate this study.

Comment 31

[Ben Bangert [:benbangert]]

Agreed, will file a follow-up bug to flip the pref.

Comment 32

[:shell escalante]

Hi Ben, when you are read to end the study - just let me know in #ask-experimenter. We created a place holder for this one in the new study intake tool - Experimenter.

Please include the link the Experimenter link for this study, and tag me in the request when you ask to turn it off :) https://experimenter.services.mozilla.com/experiments/prefflip-push-performance-1491171/

Comment 33

[:shell escalante]

Hi Ben, When should we schedule turning off this experiment? We are seeing that studies left to run are outside the normal QA test path, so Firefox changes underneath the experiment and eventually it breaks.

We're trying to turn off all long running studies to avoid this.

Comment 34

[Ben Bangert [:benbangert]]

I'd like for https://bugzilla.mozilla.org/show_bug.cgi?id=1524655 to have shipped before turning it off, but understand if that's not desired and its urgent to turn off this study.

Comment 35

[:shell escalante]

(In reply to Ben Bangert [:benbangert] from comment #34)

I'd like for https://bugzilla.mozilla.org/show_bug.cgi?id=1524655 to have shipped before turning it off, but understand if that's not desired and its urgent to turn off this study.

We are planning to shut this down June 18th. That will be 3 weeks after 67 went to release with the fix above. Please let us know if you need that date changes.