Closed Bug 1577668 Opened 6 years ago Closed 6 years ago

[Experiment] Pref-Flip: activity-stream-bigsnippet-holdback-release69-1577614 Fx 69.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: javaun)

References

(
URL
)

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.asrouter.providers.snippets

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment treatment 50%:

Value: {"id":"snippets","enabled":true,"type":"remote","url":"https://snippets.cdn.mozilla.net/%STARTPAGE_VERSION%/%NAME%/treatment-snippet-holdback-1577614/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/","updateCycleInMs":14400000}

The value in the URL field for the /%VERSION%/ field has been changed to this static string: treatment-snippet-holdback-1577614

This will allow the snippet team to target this group and give them the new, below search box BigSnippet snippet placement. This placement will also be going out to the rest of population. 

The experience these users will get is snippets placed below the search box.
        
- Treatment control 50%:

Value: {"id":"snippets","enabled":true,"type":"remote","url":"https://snippets.cdn.mozilla.net/%STARTPAGE_VERSION%/%NAME%/control-snippet-holdback-1577614/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/","updateCycleInMs":14400000}

The value in the URL field for the /%VERSION%/ field has been changed to this static string: control-snippet-holdback-1577614

This will allow the snippet team to target this group and give them the classic bottom of page snippet placement. 

The experience these users will get is snippets placed at the bottom of the page as is currently the default behavior.
        

    What version and channel do you intend to ship to?

6% of Release Firefox 69.0

    Are there specific criteria for participants?

N/A
Countries: all

Locales: English (Australian) (en-AU) English (British) (en-GB) English (Canadian) (en-CA) English (New Zealand) (en-NZ) English (South African) (en-ZA) English (US) (en-US) French (fr) German (de) German (Austria) (de-AT) German (Germany) (de-DE) German (Switzerland) (de-CH) 

    What is your intended go live date and how long will the experiment run?

Sep 03, 2019 - Oct 15, 2019 (42 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We will run a 2 branch study using a combination of Normandy and Snippet targeting to give users 2 different snippet experiences, one using the below-search-box BigSnippet placement, and the other using the classic bottom-of-page placement of the snippet (all other factors like timing, messaging, logic, etc. will be the same).

First, we'll enroll users into the 2 branches in the first week of FF69 release using Normandy. For each branch, Normandy will change part of the URL that the snippet pref points to, so that we can deliver custom snippets to each group independently.

Then, once enrollment is complete, the snippet team will publish the relevant snippets that will get delivered to the treatment and control groups (as well as the rest of population). It is important to note that BigSnippet snippets cannot be published to the general public before enrollment is complete, otherwise users in the control group will be exposed to the BigSnippet snippets, invalidating their status as a control group.

Once the experiment has run it's course, we'll compare the subset of users in control and treatment that saw the snippet to see if there was an effect on search metrics.

Analysis Notes:

    Since there is a secondary activation criteria other then having Normandy switch the pref (i.e. user must see the snippet being compared), there needs to be an activation period (starting from when the profile sees that experiment snippet for the first time), which will be sometime after the enrollment period ends.

The main metric analyzed will be effect on total SAP search volume over a 28 day period after activation. 

Do you plan on surveying users at the end of the experiment? Yes/No.
Strategy and Insights can help create surveys if needed
    
No

    Who is the owner of the data analysis for this experiment?

shong

    Will this experiment require uplift?

True

    QA Status of your code:

Signed off as GREEN. For more info see https://experimenter.services.mozilla.com/experiments/activity-stream-bigsnippet-holdback-release69-1577614/#comment425.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-bigsnippet-holdback-release69-1577614/

activity-stream-bigsnippet-holdback-release69-1577614

Launch a holdback study to understand search/revenue impact of a new, new tab snippet that is positioned directly under the new tab search bar.

The relationships team has conceived of a dynamic new snippet AKA the "Giant Snippet" (screenshot attached, though with placeholder copy and logo). The old snippet is far down on the page. This one is above the fold. Prior testing suggests this will be far more successful at getting users to engage more deeply with Firefox and adjacent services.

There is business urgency to launch this Giant Snippet in 69 to maximize the time to gain FxA MAU (relationships KPI) towards our Moco 2019 goal, therefore we are going out live with this to all users.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-bigsnippet-holdback-release69-1577614/

We have finished testing the activity-stream-bigsnippet-holdback-release69-1577614 experiment.

QA’s recommendation: GREEN - SHIP IT

**Reasoning: **

  • We have not found any new issues during testing the experiment.

Testing Summary:

  • Verified that the snippet appears as expected (under the search bar) and that only one snippet is displayed at any one time.
  • Verified that the snippet is resized accordingly depending on the available window size.
  • Verified that snippets can be turned OFF and no longer appear even on previously opened New Tabs.
  • Verified that snippets can be turned back ON and they reappear as expected.
  • Verified that snippets only appear after the Extended Triplets are dismissed.
  • Verified that the snippet is displayed even if the Search Bar is not displayed.
  • Verified that the snippet no longer appears after it was dismissed or the button is clicked.
  • Verified that the snippet redirects accordingly when the button is clicked.
  • Verified that if a sync account is already logged in, clicking the button will autocomplete the username field.
  • Verified that the snippet correctly adapts to the dark theme.
  • Verified that the snippet sends impression, click, and block pings.

The above verification was made possible with help from Giorgos, who hosted two such snippets, one for US and one for FR.

Tested Platforms:

  • Windows 10
  • macOS 10.14
  • Arch Linux 5.10

Tested Firefox versions:

  • Firefox RC 69 build 2
User Story: (updated)
Summary: [Experiment]: Pref-Flip: activity-stream-bigsnippet-holdback-release69-1577614 → [Experiment] Pref-Flip: activity-stream-bigsnippet-holdback-release69-1577614 Fx 69.0 Release
Start Date: 2019-09-03 End Date: 2019-10-15
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.