1553716 - [Shield] Pref-Flip Study: activity-stream-fxa-release67-snippets-holdback-1544066, 67, Release

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Experimenter Bot]

activity-stream-fxa-release67-snippets-holdback-1544066

Starting with FF67, we've enabled the ability for snippets to be shown below the search box (previously was only shown at bottom of page) in newtab page.

Placement can be determined by the snippet server. We have not run any snippets that use this feature yet. The plan is to start running FxA snippets in the new, below search box location starting at a future date (June 5th).

Before we do that, however, we want to run this hold back experiment to keep an eye on any regressions in usage metrics (search counts, URIs loaded, hours used) showing these snippets at the new location might have, as well as improvement in snippet metrics (click through rate, dismissal rate, remove rate).

The design of the experiment is this:
2 branches:

• control
• variant1

They will differ in this way:

• Control branch will receive FxA snippets in the old, bottom of page position.
• Variant1 will receive FxA snippets in the new, below searchbox position.

Note: the timeline of this holdback experiment is this:

• FF67 released: users now have the code to get snippets in new location, but no snippets that use this new feature have been delivered yet.
• Normandy enrollment period: we set the prefs for the control and test group
• June 5th (or any date after enrollment ends): we start shipping new snippets that go to the new location. The control group profiles that were tagged for this experiment do not get those new snippets, instead, they get snippets at the old, bottom of page location.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-fxa-release67-snippets-holdback-1544066/

Comment 1

[Su-Young Hong]

[Tracking Requested - why for this release]:

Comment 2

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.newtabpage.activity-stream.asrouter.providers.snippets

    What are the branches of the study and what values should
    each branch be set to?

- Treatment variant1 50%:

Value: "{\"id\":\"snippets\",\"enabled\":true,\"type\":\"remote\",\"url\":\"https://snippets.cdn.mozilla.net/%STARTPAGE_VERSION%/%NAME%/%VERSION%/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/\",\"updateCycleInMs\":14400000}"

Variant1 branch. We will send snippets to the new, below the searchbox location.
        
- Treatment control 50%:

Value: "{\"id\":\"snippets\",\"enabled\":true,\"type\":\"remote\",\"url\":\"https://snippets.cdn.mozilla.net/%STARTPAGE_VERSION%/%NAME%/68-control/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/\",\"updateCycleInMs\":14400000}"

Control branch. We will send snippets to the old location (bottom of page) for these subjects. 

Note: although it's the control branch, since this is a holdback experiment, the general population will be getting the new experience (while control reflects the old experience)
        

    What version and channel do you intend to ship to?

3.3% of Release Firefox 67.0

    Are there specific criteria for participants?

Prefs: 
* N/A

Studies: 
* N/A

Any additional filters: 
* Please target users that are on FF67 and FF68
* locale: en-US only
* no country filters

    What is your intended go live date and how long will the study run?

May 28, 2019 - Jul 16, 2019 (49 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

28 day usage metrics after being shown an FxA snippet 
* days used
* active hours
* total URIs
* total searches

Snippet metrics
* click through rate


Do you plan on surveying users at the end of the study? Yes/No.
Strategy and Insights can help create surveys if needed

* No

    Who is the owner of the data analysis for this study?

[e-mail]

    Will this experiment require uplift?

False

    QA Status of your code:

Testing has been requested via: 

https://moz-pi-test.atlassian.net/browse/PI-125

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/activity-stream-fxa-release67-snippets-holdback-1544066/

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Per Experimenter, this experiment ended on July 16.