1259608 - [tracking] firstrun/secondrun tests for Q2 2016

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Jonathan Petto [:jpetto]]

Tracking bug for all work related to firstrun/secondrun tests to be run in Q2 2016.

Onboarding web touchpoints: worksheet for content strategy & test plan: https://docs.google.com/document/d/1lvr7RmE7z2D37lCi8JaYt8wYadrRWoyhBiIxomecFmw/edit#heading=h.xe07bt7qsd5s

Onboarding sprint - March 22-April 8, 2016: https://docs.google.com/document/d/1HGFgHyyhBjpU2Xxdfh1cnk-iW3NEAC5pFEvuJ9a4dWk/edit

Comment 1

[Jonathan Petto [:jpetto]]

Wireframes from mverdi: https://www.dropbox.com/sh/aslf2gv77k1l4us/AACZ5eARZAuK6HwF-mD-qtcda?dl=0

Comment 2

[Holly Habstritt Gaal [:Habber]]

We will have 4 flow variations tested against the control. Audience is restricted to en-us, first time Firefox users, Windows and Mac (no Linux). 

Test plan and page requirements are in the following google doc (also linked to above). Tables present details for the Control and v1-v4 tests. 

https://docs.google.com/document/d/1lvr7RmE7z2D37lCi8JaYt8wYadrRWoyhBiIxomecFmw/edit#heading=h.7zcqpf6an3eh

4 new pages are being created:
1) re-themed, accounts-focused First Run page 
2) accounts-focused page w/ more content and 3 steps
3) "user choice" page w/ 2 options - Mobile download and Private Browsing
4) "user choice" page with 2 options - pin Firefox and Search

Caveat for above items:
- Web pages created will be used in both First and Second run positions in our tests, depending on the flow variation. 
- Depending on position in flow, a unique headline may need to be written. 
- Items 2-4 will be used interchangeably in the First and Second run positions

Comment 3

[Holly Habstritt Gaal [:Habber]]

(In reply to Holly Habstritt Gaal [:Habber] from comment #2)
> We will have 4 flow variations tested against the control. Audience is
> restricted to en-us, first time Firefox users, Windows and Mac (no Linux). 
> 
> Test plan and page requirements are in the following google doc (also linked
> to above). Tables present details for the Control and v1-v4 tests. 
> 
> https://docs.google.com/document/d/
> 1lvr7RmE7z2D37lCi8JaYt8wYadrRWoyhBiIxomecFmw/edit#heading=h.7zcqpf6an3eh
> 
> 4 new pages are being created:
> 1) re-themed, accounts-focused First Run page 
> 2) accounts-focused page w/ more content and 3 steps
> 3) "user choice" page w/ 2 options - Mobile download and Private Browsing
> 4) "user choice" page with 2 options - pin Firefox and Search
> 
> Caveat for above items:
> - Web pages created will be used in both First and Second run positions in
> our tests, depending on the flow variation. 
> - Depending on position in flow, a unique headline may need to be written. 
> - Items 2-4 will be used interchangeably in the First and Second run
> positions

need-info Jbertsch and Cmore: Could you notify Jpetto in this bug if anything in the updated test plan details impact his implementation work or negates anything said in comment 2? 

The updates to the test plan won't impact the 4 pages already defined in comment 2 and no additional templates will be needed. However, there are now more flow variations than the 4 mentioned in the original plan.

Comment 4

[Jonathan Petto [:jpetto]]

:cmore and I met today to finalize the test plan. The full details can be found in the doc [1]. Below are the URLs broken out by variation. Note that all templates, unless noted otherwise, use the space theme.

# Control
/firstrun/?v=1 - simple/existing FxA template

## Variation 2
/firstrun/?v=2 - simple/existing FxA template
/secondrun/?v=2 - mobile download/private browsing template

## Variation 3
/firstrun/?v=3 - simple/existing FxA template
/secondrun/?v=3 - pin it/try search template

## Variation 4
/firstrun/?v=4 - mobile download/private browsing template
/secondrun/?v=4 - simple/existing FxA template

## Variation 5
/firstrun/?v=5 - pin it/try search template
/secondrun/?v=5 - complex/3-step FxA

## Variation 6
/firstrun/?v=6 - complex/3-step FxA
/secondrun/?v=6 - pin it/try search template

## Variation 7
(no first or second run)

## Invalid variation/non en-US visitor
simple/existing FxA template (no space theme)


[1] https://docs.google.com/document/d/1yd0coXNMWTSdZ16rr2KUn3Vz9XwZrWuno_e3VazbTpY/edit?ts=56f190c9

Comment 5

[Chris More [:cmore]]

Thanks for the help Jon and working through this. We aren't getting every combination of potential first run and second run combos, but there isn't enough traffic to do more variations. Even if we did more variations, the findings may not be currently useful. This is a good first funnelcake test and there will surely be lots of insights and probably unexpected results.

I also updated the onboarding touchpoint testing matrix with the download page, first run, and second run tests:

https://docs.google.com/spreadsheets/d/13yqwS_g-pZlRlTASQ0KGxKEk-4II7f1ezy4aR6jdDPs/edit#gid=0

Comment 6

[Jonathan Petto [:jpetto]]

The variations are now up on demo5:

## Control
https://www-demo5.allizom.org/en-US/firefox/46.0/firstrun/?v=1 - simple/existing FxA template

## Variation 2
https://www-demo5.allizom.org/en-US/firefox/46.0/firstrun/?v=2 - simple/existing FxA template
https://www-demo5.allizom.org/en-US/firefox/46.0/secondrun/?v=2 - mobile download/private browsing template

## Variation 3
https://www-demo5.allizom.org/en-US/firefox/46.0/firstrun/?v=3 - simple/existing FxA template
https://www-demo5.allizom.org/en-US/firefox/46.0/secondrun/?v=3 - pin it/try search template

## Variation 4
https://www-demo5.allizom.org/en-US/firefox/46.0/firstrun/?v=4 - mobile download/private browsing template
https://www-demo5.allizom.org/en-US/firefox/46.0/secondrun/?v=4 - simple/existing FxA template

## Variation 5
https://www-demo5.allizom.org/en-US/firefox/46.0/firstrun/?v=5 - pin it/try search template
https://www-demo5.allizom.org/en-US/firefox/46.0/secondrun/?v=5 - complex/3-step FxA

## Variation 6
https://www-demo5.allizom.org/en-US/firefox/46.0/firstrun/?v=6 - complex/3-step FxA
https://www-demo5.allizom.org/en-US/firefox/46.0/secondrun/?v=6 - pin it/try search template

Comment 7

[Chris More [:cmore]]

Going to keep this open until I complete the analysis.

Comment 8

[Chris More [:cmore]]

Analysis here: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=61938912

tl;dr: none of the first run and second run pages tested had a positive or negative impact on retention. The only specific learning was doing nothing had the lowest usage compared to other variations tested.