1332130 - Optimizely test needed for Onboarding v2.0 experience (funnelcake 98/99/100)

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, defect)

Description

[Chris More [:cmore]]

The product team is building a funnelcake to experiment with the first few pieces of the v2.0 onboarding experience. There will be 3 funnelcakes created, 1 control and 2 variations.

The optimizely settings will be:

* browser: non-Firefox
* language: en-US (confirming)
* OS: Windows
* Device category: desktop

Sample size: 8% (should give us enough for all 3 cohorts)

Test plan: https://docs.google.com/document/d/1Kle7LlVQ_gCJ2IFpQOSg7XVDtrXGNYGPacKIaYO1i9A/edit#

Target URL: /firefox/new/

Variations:

FC98: redirect w/f=98
FC99: redirect w/f=99
FC100: redirect w/f=100

Goal: engagement and download button

Start date: 2/7/2017 
End date: 2/25/2017

Sample size: 

This will give two full weeks of acquisition data from Sunday through Saturday.

Comment 1

[Chris More [:cmore]]

Just to be clear on variations:

FC98 = control: existing installer, download page and first run
FC99 = variation 1: updated download page, streamlined installer, auto-migration enabled, wow moment first run
FC100 = variation 2: updated download page, streamlined installer, auto-migration DISABLED, wow moment first run

Comment 2

[Romain Testard [:RT]]

Is the Optimizely set-up is needed to test end to end?
Grover helps us with testing this, thanks for confirming when this is ready for him to start testing.

Comment 3

[Chris More [:cmore]]

Optimizely test is coming together here:

https://app.optimizely.com/edit?experiment_id=8208090261

Comment 4

[Chris More [:cmore]]

(In reply to Chris More [:cmore] from comment #3)
> Optimizely test is coming together here:
> 
> https://app.optimizely.com/edit?experiment_id=8208090261

Done:

* audience
* goals
* sample size
* url target
* created 3 variations

To do:

* Add the jqeruy redirect code for each variation. It is not working from the plane or I would do it now.

Comment 5

[Chris More [:cmore]]

Optimizely test setup:

https://app.optimizely.com/edit?experiment_id=8208090261

:hoosteeno: can you double check this experiment and verify that it matches the criteria in comment 0?

Comment 6

[Chris More [:cmore]]

:jpetto: while Justin is reviewing this experiment from a growth perspective, can you take a look at it too? thanks

Comment 7

[Justin Crawford [:hoosteeno] [:jcrawford]]

* config for audience targeting looks good
* sample size looks good
* regex for URL targeting looks good

Comment 8

[Chris More [:cmore]]

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #7)
> * config for audience targeting looks good
> * sample size looks good
> * regex for URL targeting looks good

Thanks!

jpetto: let us know when you are good with the test. Launch is Feb 7th.

Comment 9

[Jonathan Petto [:jpetto]]

Looks good to me. r+

Comment 10

[Chris More [:cmore]]

Funnelcake stopped on 2/27/2017 at 8am PST.