Closed Bug 1267308 Opened 8 years ago Closed 8 years ago

Create A/B test for new space theme /firstrun/ and /secondrun/ pages

Categories

(www.mozilla.org :: Analytics, defect)

Product:
www.mozilla.org
Component:
Analytics
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cmore, Unassigned)

References

Details

User Story

Variations:

FC 75 Control:  Re-themed Accounts First Run / No secondrun
Firstrun with accounts sign up with space theme
Url: /firstrun/?v=1
Secondrun isn’t included in flow
Url: NA

FC 76 Variation 2:  Re-themed Accounts First Run / mobile & pb user actions second run
Potential learning: Introduction of second run and content
Firstrun with accounts sign up with space theme
Url: /firstrun/?v=2
Secondrun “user actions” design template with private browsing and mobile download
Url: /secondrun/?v=2

FC 77 Variation 3: Re-themed Accounts First Run / pin & search user actions second run
Potential learning: Introduction of second run and content
Firstrun with accounts sign up with space theme
Url: /firstrun/?v=3
Secondrun “user actions” design template with 2 actions: pin Firefox and try searching
Url: /secondrun/?v=3

FC 78 Variation 4:  Private browsing first run/Re-themed Accounts Second Run
Potential learning: Impact of moving FxA to second run
Firstrun  “user actions” design template with private browsing and mobile download
Url: /firstrun/?v=4
Secondrun with accounts sign up with space theme
Url: /secondrun/?v=4

FC 79 Variation 5:  User actions firstrun/complex accounts second run
Potential learning: Understanding the impact of “why” for an Account (can use v4 as control)
Firstrun  “user actions” design template with 2 actions: pin Firefox and try searching
Url: /firstrun/?v=5
Secondrun with accounts sign up, mobile download CTA, and context about multiple device sync
Url: /secondrun/?v=5

FC 80 Variation 6:  Complex accounts sign up firstrun/user action second run
Potential learning: Understand how leading in with mobile impacts retention
Firstrun with accounts sign up, mobile download CTA, and context about multiple device sync
Url: /firstrun/?v=6
Secondrun “user actions” design template with 2 actions: pin Firefox and try searching
Url: /secondrun/?v=6

FC 81 Variation 7:  No first run or second run
Potential learning: Understand the impact of FxA on retention by comparing to control
Firstrun: none
Url: NA
Secondrun: none
Url: NA
We need to create an Optimizely experiment to redirect users to special funnelcake builds of Firefox to test the engagement and retention of new first and second run pages.

Configuration:

Language: English any
OS: Windows
Browser: Non-Firefox
Device: Desktop
variations: 7 (including control)

All 7 will be redirect tests to the base url with the appropriate f= parameter in the URL.

This Optimizely test can be setup after the builds are created in bug 1257899.
Depends on: 1257899
No longer depends on: 1257899
Actually, the correct bug for the funnelcakes is bug 1267305.
Blocks: 1267305
No longer blocks: 1267305
Depends on: 1267305
We're going to need about 60k downloads per variation or about 420k downloads and thus we need to make the sample size and duration of experiment long enough to get that many downloads before Firefox 47.0.
Set audience to: Non-Firefox + Desktop + English + Windows 7 or newer so to not display these to Windows XP users as not all the experiences work for them.
Traffic allocation: 20%
:jpetto: can you check out this experiment?

https://app.optimizely.com/edit?experiment_id=5759573117
Flags: needinfo?(jon)
Experiment looks pretty straight forward. Tested redirect URLs and all behavior is as expected (including 404s on downloads until funnelcakes are built).

r+ from me.
Flags: needinfo?(jon)
Closing after r+ but pending funnel cakes
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Update URL targeting:

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/.*$

excluding:

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/.*f=(7[5-9]|8[0-1]).*$

This will ensure no infinite loop.
Funnelcakes are almost ready.
Funnelcake experiment started on 8:50am PST on 5/13/2016. We'll need to let this run for two weeks and the experiment is set to turn off automatically on 6/6/2016. If we get up to around 420-450k downloads total, we can shut it off then too.
Funnelcakes shut off at 12:48pst on 5/19/2016. We got enough downloads per variation.
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.