Create A/B test for new space theme /firstrun/ and /secondrun/ pages

RESOLVED FIXED

Status

www.mozilla.org
Analytics
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: cmore, Unassigned)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

User Story

Variations:

FC 75 Control:  Re-themed Accounts First Run / No secondrun
Firstrun with accounts sign up with space theme
Url: /firstrun/?v=1
Secondrun isn’t included in flow
Url: NA

FC 76 Variation 2:  Re-themed Accounts First Run / mobile & pb user actions second run
Potential learning: Introduction of second run and content
Firstrun with accounts sign up with space theme
Url: /firstrun/?v=2
Secondrun “user actions” design template with private browsing and mobile download
Url: /secondrun/?v=2

FC 77 Variation 3: Re-themed Accounts First Run / pin & search user actions second run
Potential learning: Introduction of second run and content
Firstrun with accounts sign up with space theme
Url: /firstrun/?v=3
Secondrun “user actions” design template with 2 actions: pin Firefox and try searching
Url: /secondrun/?v=3

FC 78 Variation 4:  Private browsing first run/Re-themed Accounts Second Run
Potential learning: Impact of moving FxA to second run
Firstrun  “user actions” design template with private browsing and mobile download
Url: /firstrun/?v=4
Secondrun with accounts sign up with space theme
Url: /secondrun/?v=4

FC 79 Variation 5:  User actions firstrun/complex accounts second run
Potential learning: Understanding the impact of “why” for an Account (can use v4 as control)
Firstrun  “user actions” design template with 2 actions: pin Firefox and try searching
Url: /firstrun/?v=5
Secondrun with accounts sign up, mobile download CTA, and context about multiple device sync
Url: /secondrun/?v=5

FC 80 Variation 6:  Complex accounts sign up firstrun/user action second run
Potential learning: Understand how leading in with mobile impacts retention
Firstrun with accounts sign up, mobile download CTA, and context about multiple device sync
Url: /firstrun/?v=6
Secondrun “user actions” design template with 2 actions: pin Firefox and try searching
Url: /secondrun/?v=6

FC 81 Variation 7:  No first run or second run
Potential learning: Understand the impact of FxA on retention by comparing to control
Firstrun: none
Url: NA
Secondrun: none
Url: NA
(Reporter)

Description

2 years ago
We need to create an Optimizely experiment to redirect users to special funnelcake builds of Firefox to test the engagement and retention of new first and second run pages.

Configuration:

Language: English any
OS: Windows
Browser: Non-Firefox
Device: Desktop
variations: 7 (including control)

All 7 will be redirect tests to the base url with the appropriate f= parameter in the URL.

This Optimizely test can be setup after the builds are created in bug 1257899.
(Reporter)

Updated

2 years ago
Depends on: 1257899
(Reporter)

Updated

2 years ago
No longer depends on: 1257899
(Reporter)

Comment 1

2 years ago
Actually, the correct bug for the funnelcakes is bug 1267305.
(Reporter)

Updated

2 years ago
Blocks: 1267305
(Reporter)

Updated

2 years ago
No longer blocks: 1267305
Depends on: 1267305
(Reporter)

Comment 2

2 years ago
We're going to need about 60k downloads per variation or about 420k downloads and thus we need to make the sample size and duration of experiment long enough to get that many downloads before Firefox 47.0.
(Reporter)

Comment 3

2 years ago
Set audience to: Non-Firefox + Desktop + English + Windows 7 or newer so to not display these to Windows XP users as not all the experiences work for them.
(Reporter)

Comment 4

2 years ago
Traffic allocation: 20%
(Reporter)

Comment 5

2 years ago
:jpetto: can you check out this experiment?

https://app.optimizely.com/edit?experiment_id=5759573117
Flags: needinfo?(jon)

Comment 6

2 years ago
Experiment looks pretty straight forward. Tested redirect URLs and all behavior is as expected (including 404s on downloads until funnelcakes are built).

r+ from me.
Flags: needinfo?(jon)
Closing after r+ but pending funnel cakes
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

2 years ago
Update URL targeting:

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/.*$

excluding:

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/.*f=(7[5-9]|8[0-1]).*$

This will ensure no infinite loop.
(Reporter)

Comment 9

2 years ago
Funnelcakes are almost ready.
(Reporter)

Comment 10

2 years ago
Funnelcake experiment started on 8:50am PST on 5/13/2016. We'll need to let this run for two weeks and the experiment is set to turn off automatically on 6/6/2016. If we get up to around 420-450k downloads total, we can shut it off then too.
(Reporter)

Comment 11

2 years ago
Funnelcakes shut off at 12:48pst on 5/19/2016. We got enough downloads per variation.
(Reporter)

Updated

2 years ago
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.