Closed Bug 1695026 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: New Tab Modernized UX - Region 2 Fx 86.0 to 88.0 Release

Categories

(Shield :: Shield Study, task, P3)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: experimenter, Unassigned)

References

()

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Control newtab-control 20%:

Value: None

Existing User Experience
        
- Treatment newtab-a-newux 20%:

Value: None

Treatment A: New UX
        
- Treatment newtab-b-newbranding 20%:

Value: None

Treatment B: New Branding
        
- Treatment newtab-c-newfocused 20%:

Value: None

Treatment C: New Focused
        
- Treatment newtab-d-newsearch 20%:

Value: None

Treatment D: New Search
        

    What version and channel do you intend to ship to?

10% of Release Firefox 86.0 to 88.0

    Are there specific criteria for participants?

Prefs:

[shell] believe it should go in the same namespace AND buckets as this other new profile experiment that has finished enrolling Feb 16th.  https://experimenter.services.mozilla.com/experiments/new-tab-default-content/.  When the rollout increases 10% on April 29 - it will need to avoid this experiment - if that helps determine which of the 30% available this 9% should go into.

It specifically needs to avoid - https://experimenter.services.mozilla.com/experiments/topsites-launch-phase-3-de/ and 
https://experimenter.services.mozilla.com/experiments/rollout-sponsored-topsites/

Since it's in the DE - it's in the [US, UK, DE] AND inNamespace("forward")) https://docs.google.com/document/d/1lFauJ7BOEM479mnG-fSUOg5FDcf9jx9K_KEaupsuh_c/edit#
Countries: Germany (DE) 

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 02, 2021 - May 07, 2021 (66 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will measure <outcome> by an <increase/decrease/neutral> of <size> in <feature telemetry>. There may be multiple leading indicator statements.

Optional: We hypothesize the desired change will <increase/decrease/neutral> to the <core metric>.

Primary metric: - Decrease (>=5%) in fraction of new profiles with any new tab page takeover event.

Secondary metric: Search traffic increasing - Overall searches - New tab SAP search rate

Secondary metric: Days of use increasing

Secondary metric: increasing - Section enable/disable rates at end of study - Interaction of new tab: Top Sites, Pocket, etc...

    Who is the owner of the data analysis for this experiment?

cdowhygelund@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

https://jira.mozilla.com/browse/PI-981

QA Status: GREEN. For more information see: https://docs.google.com/document/d/1hvFo64N7nPgWfc8pM51vANhFX8FwDmimjGN4rA-Jx0U/edit

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/new-tab-modernized-ux-region-2/

New Tab Modernized UX - Region 2

We have a new about:newtab design that is landing in Firefox 86 behind a pref. We would like to measure user engagement and switching behavior similar in order to compare baseline metrics between the existing and new design.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-427
More information: https://experimenter.services.mozilla.com/experiments/new-tab-modernized-ux-region-2/

We've sent a GREEN sign-off via email. For more information see: Gdoc.

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: New Tab Modernized UX - Region 2 → [Experiment] Pref-Flip Experiment: New Tab Modernized UX - Region 2 Fx 86.0 to 88.0 Release
Start Date: 2021-03-02 End Date: 2021-05-07
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.