Closed Bug 1696829 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: New Tab Default Content - Existing Users - CA Fx 86.0 to 88.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox86+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox86 + fixed

People

(Reporter: experimenter, Unassigned)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Baseline Newtab Content 25%:

Value: None

The newtab content will show the Searchbar, Topsites, and Pocket by default. The Firefox logo, Highlights, and Snippets will be off by default.
        
- Treatment Focused Newtab Content 25%:

Value: None

The newtab content will show the  Firefox logo, Searchbar, and Topsites by default. Pocket, Highlights, and Snippets will be off by default.
        
- Treatment Search Newtab Content 25%:

Value: None

The newtab content will show the Firefox logo, and Searchbar by default. Topsites, Pocket, Highlights, and Snippets will be off by default.
        
- Treatment Simple Newtab Content 25%:

Value: None

The newtab content will show the Searchbar and Topsites by default. The Firefox logo, Pocket, Highlights, and Snippets will be off by default.
        

    What version and channel do you intend to ship to?

10% of Release Firefox 86.0 to 88.0

    Are there specific criteria for participants?

Prefs:
Exclude all non-default: startup.homepage_override_url
exclude all non-default: browser.startup.homepage
exclude all non-default: Browser.newtabpage.enabled

[shell] Since it's in the CA - I think it is in the Namespace("backwards")) https://docs.google.com/document/d/1lFauJ7BOEM479mnG-fSUOg5FDcf9jx9K_KEaupsuh_c/edit#

It needs to avoid:
CA experiment https://experimenter.services.mozilla.com/experiments/topsites-launch-phase3-group2-ca/ 40%
and
CA rollout https://mana.mozilla.org/wiki/display/FPS/Rollout+Plan+of+adM+Sponsored+Topsites 30%

The rollout increases 10% on April 29 - if that helps determine which bucket to put it into now.
Countries: Canada (CA) 

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 16, 2021 - May 22, 2021 (67 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

Primary metric: Search traffic increasing - Overall searches - New tab SAP search rate

Primary metric: Days of use increasing

Secondary metric (not sizing for to detect an improvement - but want to measure to look for regression) - no noticeable increase in fraction of existing profiles with any new tab page takeover event.   

NOTE: we are not including the disable rates of any particular section that the treatments change: Top Sites, Pocket, etc.  If users go in and disable the sections (which are treatment prefs enabled in the experiment) - Normandy will unenroll them from experiment.  

We can look at this through grafana unenrollment forensics if we see it often- but we're not planning to analyze this up front.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

https://jira.mozilla.com/browse/PI-1000

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/new-tab-default-content-existing-users-cad/

New Tab Default Content - Existing Users - CA

We want to learn which default content on the new tab is most loved. We aim to meet user expectations so that users do not need to alter their new tab.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-417
More information: https://experimenter.services.mozilla.com/experiments/new-tab-default-content-existing-users-cad/

[Tracking Requested - why for this release]: The search new tab team is launching 4 experiments in different countries to discover New Tab treatments that improve user engagement. The hope is to learn what experiences users most prefer.

status-firefox86: --- → affected
tracking-firefox86: --- → ?
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: New Tab Default Content - Existing Users - CA → [Experiment] Pref-Flip Experiment: New Tab Default Content - Existing Users - CA Fx 85.0 to 88.0 Release
tracking-firefox86: ? → +
User Story: (updated)

We've sent a YELLOW sign off via email. You can find more information here.

User Story: (updated)
Summary: [Experiment] Pref-Flip Experiment: New Tab Default Content - Existing Users - CA Fx 85.0 to 88.0 Release → [Experiment] Pref-Flip Experiment: New Tab Default Content - Existing Users - CA Fx 86.0 to 88.0 Release
User Story: (updated)
Start Date: 2021-03-16 End Date: 2021-05-22
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.