Closed Bug 1579914 Opened 6 years ago Closed 6 years ago

[Experiment] Pref-Flip: activity-stream-bigsnippet-holdback-release69-1577614 V2 Fx 69.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: shell)

References

(
URL
)

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.asrouter.providers.snippets

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment control 50%:

Value: {"id":"snippets","enabled":true,"type":"remote","url":"https://snippets.cdn.mozilla.net/%STARTPAGE_VERSION%/%NAME%/control-snippet-holdback-1577614/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/","updateCycleInMs":14400000}

The value in the URL field for the /%VERSION%/ field has been changed to this static string: control-snippet-holdback-1577614

This will allow the snippet team to target this group and give them the classic bottom of page snippet placement. 

The experience these users will get is snippets placed at the bottom of the page as is currently the default behavior.
        
- Treatment treatment 50%:

Value: {"id":"snippets","enabled":true,"type":"remote","url":"https://snippets.cdn.mozilla.net/%STARTPAGE_VERSION%/%NAME%/treatment-snippet-holdback-1577614/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/","updateCycleInMs":14400000}

The value in the URL field for the /%VERSION%/ field has been changed to this static string: treatment-snippet-holdback-1577614

This will allow the snippet team to target this group and give them the new, below search box BigSnippet snippet placement. This placement will also be going out to the rest of population. 

The experience these users will get is snippets placed below the search box.
        

    What version and channel do you intend to ship to?

6% of Release Firefox 69.0

    Are there specific criteria for participants?

N/A
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Sep 09, 2019 - Oct 21, 2019 (42 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We will run a 2 branch study using a combination of Normandy and Snippet targeting to give users 2 different snippet experiences, one using the below-search-box BigSnippet placement, and the other using the classic bottom-of-page placement of the snippet (all other factors like timing, messaging, logic, etc. will be the same).

First, we'll enroll users into the 2 branches in the first week of FF69 release using Normandy. For each branch, Normandy will change part of the URL that the snippet pref points to, so that we can deliver custom snippets to each group independently.

Then, once enrollment is complete, the snippet team will publish the relevant snippets that will get delivered to the treatment and control groups (as well as the rest of population). It is important to note that BigSnippet snippets cannot be published to the general public before enrollment is complete, otherwise users in the control group will be exposed to the BigSnippet snippets, invalidating their status as a control group.

Once the experiment has run it's course, we'll compare the subset of users in control and treatment that saw the snippet to see if there was an effect on search metrics.

Analysis Notes:

    Since there is a secondary activation criteria other then having Normandy switch the pref (i.e. user must see the snippet being compared), there needs to be an activation period (starting from when the profile sees that experiment snippet for the first time), which will be sometime after the enrollment period ends.

The main metric analyzed will be effect on total SAP search volume over a 28 day period after activation. 

Do you plan on surveying users at the end of the experiment? Yes/No.
Strategy and Insights can help create surveys if needed
    
No

    Who is the owner of the data analysis for this experiment?

shong

    Will this experiment require uplift?

True

    QA Status of your code:

Signed off as GREEN. For more info see https://experimenter.services.mozilla.com/experiments/activity-stream-bigsnippet-holdback-release69-1577614/#comment425.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-bigsnippet-holdback-release69-1577614-v2/

activity-stream-bigsnippet-holdback-release69-1577614 V2

Launch a holdback study to understand search/revenue impact of a new, new tab snippet that is positioned directly under the new tab search bar.

The relationships team has conceived of a dynamic new snippet AKA the "Giant Snippet" (screenshot attached, though with placeholder copy and logo). The old snippet is far down on the page. This one is above the fold. Prior testing suggests this will be far more successful at getting users to engage more deeply with Firefox and adjacent services.

There is business urgency to launch this Giant Snippet in 69 to maximize the time to gain FxA MAU (relationships KPI) towards our Moco 2019 goal, therefore we are going out live with this to all users.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-bigsnippet-holdback-release69-1577614-v2/

User Story: (updated)
Summary: [Experiment]: Pref-Flip: activity-stream-bigsnippet-holdback-release69-1577614 V2 → [Experiment] Pref-Flip: activity-stream-bigsnippet-holdback-release69-1577614 V2 Fx 69.0 Release
Start Date: 2019-09-09 End Date: 2019-10-21
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.