1463491 - [Shield] Opt-out Study: Beta 61 Snippets Pref Flip Test

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Tim Spurway [:tspurway]]

This is the first time we will be running AS Router code / templates in an environment with real users. We would like to confirm that our impression data for snippets matches our intended targets and that we are receiving data about how users are engaging with them. In the future we will be introducing more targeting functionality for snippets and we would like to establish a baseline.

Comment 1

[Tim Spurway [:tspurway]]

This 'experiment' does not have A/B cohorts, it is simply an opt-out pref flip to test new functionality and establish baseline data in beta 61 for a new Snippets implementation.  It is described in this Experimenter link:

https://experimenter.services.mozilla.com/experiments/as-router-phase-1-snippets-experiment/

Comment 2

[Ciprian Muresan [:cmuresan], Ecosystem QA]

We have finished testing the Beta 61 Snippets Pref Flip experiment.

We have found the following issue:
1464019 - Snippets can no longer be turned off when browser.newtabpage.activity-stream.asrouterExperimentEnabled is true

QA’s recommendation: RED - DON'T SHIP

Reasoning: Bug 1464019 makes it impossible for users that get into the study to turn off the Snippets service, if they no longer what to see them.

Testing Summary:
- Verified that snippets correctly appear on New Tabs
- Verified that different snippets appear on each New Tab
- Verified that snippets still appear after restarting or updating the browser
- Verified that snippets no longer appear after they are dismissed
- Verified that telemetry pings are sent for each snippet impression, link/button click, and dismiss

Tested Platforms:
Windows 10 x64
Mac 10.13.3
Arch Linux x64

Tested Firefox versions
Firefox 61 Beta 6 (61.0b6)
Firefox 61 Beta 7 (61.0b7)

Comment 3

[Tim Spurway [:tspurway]]

Due to the static nature and limited set of Snippets that users will see during the study, the Marketing team asked if we could do a 'rolling sample' on this study to help lessen user fatigue and annoyance.  

Ideally, we would enroll 50% of the population over a two week period starting June 4th, with each user staying enrolled in the experiment for a maximum of 5 days.  

Is this enough info to configure the study :mgrimes?

Comment 4

[Matt Grimes [:Matt_G]]

This is perfect. We'll ensure the populations are non-overlapping so that we don't cause snippet fatigue. Thanks for the details.

Comment 5

[Kate Hudson :k88hudson]

In terms of what we expect for impression data, each of the five snippets in the experiment should have daily impressions equal to one fifth of the number of new tab pages opened by all users in the experiment cohort during that day, with a margin of acceptability of 5% (see Analysis section of https://experimenter.services.mozilla.com/experiments/as-router-phase-1-snippets-experiment for more information)

Does that sound reasonable Ilana?

Comment 6

[Ilana]

Very! Feel free to have your analyst reach out to me or the team with any questions.

Comment 7

[Ciprian Muresan [:cmuresan], Ecosystem QA]

With bug 1464019 fixed and verified in Beta, from QA's point of view, we're good to go with the Study.

Comment 8

[Tim Spurway [:tspurway]]

:matt_g, :rweiss - is there anything else we need to do to launch?

Comment 9

[Matt Grimes [:Matt_G]]

Now that QA is done you're still going to need a sign off from Relman. We normally only ship on Monday, but they may be open to an exception to push on a Tues since this is low risk and in Beta.

Comment 10

[MarniePW [:marnie]]

:tspurway, I have a note that Stephanie was going to do a Security review. Did that happen? And if so, the result should be logged here.

Comment 11

[Kate Hudson :k88hudson]

We got a sign-off on our approach for the external JSON file integration here (https://bugzilla.mozilla.org/show_bug.cgi?id=1457191) from Julien and we haven't changed anything in terms of the code significantly since our initial AS security review with Stephanie. Does that sound reasonable for now?

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Does the late start to this study affect when we'd want it to expire? It's going to be getting pretty close to RC week.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

To clarify, I'm mainly worried about this study bleeding into the Fx61 RC week since this will be running a very large group of our overall Beta population (at least I would assume that 50% of en-US users is). Also, it would be good if we could clear up comments 10/11.

Beyond that, QA is happy with it, so consider this the RelMan blessing with those points addressed.

Comment 14

[MarniePW [:marnie]]

I think we're OK to go with this, based on Stephanie and Kate's comments, above. Matt, thoughts on getting this one out ASAP / before All Hands so we don't run into the 6/26 release of 61?

Comment 15

[Matt Grimes [:Matt_G]]

I think we're ready to go. I'll work on this now.

Comment 16

[Matt Grimes [:Matt_G]]

Ok, should be live. Thanks folks.