1544066 - [Newtab] FxA / Snippet Experiment - Hold Back Study in Desktop Firefox 68 For Snippet Location Switch

Status: RESOLVED FIXED

(Data Science :: Experiment Collaboration, task)

Description

[Tim Spurway [:tspurway]]

Brief Description of the request (required):

The Pocket New Tab and Snippets teams are launching a new Snippet form factor in Firefox 68 to promote the benefits FxA enrolment. We would like to launch a 'hold back' style study that measures the effectiveness of this Snippet.

Business purpose for this request (required):

The new Snippet directly affects top-line KPIs around FxA enrolment.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

We would like to launch this in the Desktop Fx68 Release cycle (July 9, 2019)

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

Product Brief
Tech Doc

Name of Data Scientist (If Applicable):

Su

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Jessilyn Davis]

Some preliminary notes/thoughts around this experiment:
https://docs.google.com/document/d/1SqIA18-542qgs_lajmgMtvjUsmc7cVVUJFo_wQhD-c4/edit#

Comment 2

[Jessilyn Davis]

Nan, can you link to snippet dashboards and any documentation around the snippet dashboards for Su to review what telemetry we have/how we use it?

Comment 3

[Nan Jiang [:nanj]]

Hey Su, here is the telemetry document for Snippets:

https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_events.md#activity-stream-router-pings

And the dashboard:

https://sql.telemetry.mozilla.org/dashboard/activity-stream-snippets

Comment 4

[Su-Young Hong]

Update:

Met with team last week. It was decided we want to try to run this with a Normandy hold back study.

Currently, Tim is working on putting this change / feature (a subset of snippets will be shown at a new location just below the search bar) behind a pref.

I also need to check to see if existing Telemetry for the snippets is sufficient for this study, or if we need more engineering work implemented.

[Su] TO DO: EDA existing snippets probes to make sure the info / data is sufficient for identifying when snippets get shown, and we can identify the condition in which it is the subset of snippets that will be affected by the pref.

Comment 5

[Jessilyn Davis]

Hi Emily!

We have escalated this to launch in 67 as part of Trailhead - that means we need to have the holdback study ready to launch by 5pm PT this Monday, May 13th to launch on Tuesday, May 15th. (A week to have holdback people enrolled.)

Su is out this week :(

Is there anyone on the DS team that can help us make this happen?

Thank you,
Jess

Comment 6

[Jessilyn Davis]

Eep! Forgot to NI Emily. Emily, can you see comment above and let us know today?

Comment 7

[Emily Thompson]

Hi Jess, things are a bit crazy this week with the expired certificates issue, and there's a current hold on new experiments going out with Normandy. That could be lifted by next week, but would want to loop in Shell to understand that.

Also, is there any particular need you have from DS this week ahead of the launch of the study?

Comment 8

[Jessilyn Davis]

Thanks Emily! After slacking and a few convos, here's the downlow.

Since it's challenging (maybe impossible) to get this study together before Tuesday we will:

• Hold off on running the study until May 21 or 28.
• In order to make sure we have a valid hold back group, we will not start running the new snippets until this study has launched and users have enrolled.
• We will continue with the uplift and associated QA.
• We will still QA the study branches to make sure they work once the study is ready to launch

Comment 9

[Su-Young Hong]

Update: experimenter ticket has been filed here:

https://experimenter.services.mozilla.com/experiments/activity-stream-fxa-release67-snippets-holdback-1544066/

Comment 10

[Su-Young Hong]

Experiment has completed and analyzed.

• Report: FxA Snippet Holdback 67 Test

Results has been shared with Jean Collins and team.

Closing this ticket out