1136937 - Mozilla.org /firefox/firstrun/ - Development

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Development of the /firstrun Tour and Post Tour page.

Wiki: https://wiki.mozilla.org/Firefox_Growth_Campaign_2015

Comment 1

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

We'll be developing only a /firstrun Landing page for FF38, no tour.

Comment 2

[Jonathan Petto [:jpetto]]

UX bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1129548
Wireframes: http://cl.ly/image/251H1W10373B

Comment 3

[Chris More [:cmore]]

cmore: generate google play store link links with non-https links.

Comment 4

[Chris More [:cmore]]

FYI: I am varying the utm_campaign variable by android button placement.

For Android device and Google Play button:

market://details?id=org.mozilla.firefox&referrer=utm_source%3Dmozilla%26utm_medium%3DReferral%26utm_campaign%3Dsync-android-snippet

For desktop and Google Play button:

https://play.google.com/store/apps/details?id=org.mozilla.firefox&referrer=utm_source%3Dmozilla%26utm_medium%3DReferral%26utm_campaign%3Dsync-desktop-snippet

I can't find documentation on how to do UTM-like campaign performance tracking with itunes app store buttons.

Comment 5

[Chris More [:cmore]]

Here's placeholder iOS links that we can update later:

for iOS and the iTunes App Store button:

https://itunes.apple.com/us/app/sync-for-firefox/id468995230

For desktop and iTunes App Store Button:

itms://itunes.apple.com/us/app/sync-for-firefox/id468995230

We'll replace the URLs later when iOS is pushed to the store.

Comment 6

[Jonathan Petto [:jpetto]]

In-progress version of the firstrun page is up on demo5:

https://www-demo5.allizom.org/en-US/firefox/38.1/firstrun/

***
Notice!

You must have demo5 white listed in browser.uitour.testingOrigins in about:config for the UITour functionality to work.
***

Both getting video shown on page load and having the CTA link directly to the accounts page have a 50/50 chance of being true on each page load.

You can test specific cases with the following URLs:

https://www-demo5.allizom.org/en-US/firefox/38.1/firstrun/?v=1a (no video on load, hamburger menu)
https://www-demo5.allizom.org/en-US/firefox/38.1/firstrun/?v=1b (no video on load, accounts page)
https://www-demo5.allizom.org/en-US/firefox/38.1/firstrun/?v=2a (video on load, hamburger menu)
https://www-demo5.allizom.org/en-US/firefox/38.1/firstrun/?v=2b (video on load, accounts page)

If you load a locale without a video (es-ES for now), the CTA will always open the accounts page.

https://www-demo5.allizom.org/es-ES/firefox/38.1/firstrun/?v=2a

Comment 7

[Chris More [:cmore]]

:jpetto: can you ensure these templates appears on 38.0.5? The decision was just made to make the campaign release 38.0.5 instead of 38.1.

Comment 8

[Jonathan Petto [:jpetto]]

Okay, will change things around to match 38.0.5.

Comment 9

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

:jpetto - /firstrun copy is approved in bug 1149709. Note we're using different headlines/CTA's for the different versions.

Comment 10

[Jonathan Petto [:jpetto]]

Copy is in: https://github.com/mozilla/bedrock/tree/bug-1136937-fx38.0.5-firstrun

I still need the PSD for the latest design.

Comment 11

[Chris More [:cmore]]

:jpetto: We going to be doing funnelcakes for this firstrun test too so that we can measure the effectiveness of the tests in the Firefox Health Report. Without the funnelcakes, we are going be missing that funnel.

Here's what will change:

If f=37 in URL then { do a 50/50 split between v=1a and v=2a } //hamburger menu
else
If f=38 in URL then { do a 50/50 split between v=1b and v=2b } //fxa form
else 
{ do a 50/50 split between v=1b and v=2b } //fxa form

This change means a few things:

1) You don't need to do a 50/50 split between version a or b, I will be doing that by distributing funnelcake 37 to 50% of people and funnelcake 38 to 50% of people.

2) You will need to use the funnelcake number as the trigger on what variation to load.

3) If there is no f is not a URL parameter at all or if the value of f is not 37 or 38, drop into the final else that will just do version b, which is the fxa form button target. All 3 conditions still do a 50/50 2 variation split with the video before an after.

Let me know if you have any questions.

Comment 12

[Chris More [:cmore]]

The funnelcake test on /firefox/new/ will be created by myself in bug 1157564.

Comment 13

[Jonathan Petto [:jpetto]]

Should the funnelcake test apply only to version 38.0.5, or will it need to live on for future versions (e.g. 38.1)?

How will this impact locales that do not have a translation of the video? I'm guessing the video condition should be of secondary importance. For example, if a user gets ?f=37 in a locale that does not have a corresponding video, that user will just get the hamburger menu CTA and no video content.

Does that sound right?

Comment 14

[Chris More [:cmore]]

(In reply to Jon Petto [:jpetto] from comment #13)
> Should the funnelcake test apply only to version 38.0.5, or will it need to
> live on for future versions (e.g. 38.1)?
> 
> How will this impact locales that do not have a translation of the video?
> I'm guessing the video condition should be of secondary importance. For
> example, if a user gets ?f=37 in a locale that does not have a corresponding
> video, that user will just get the hamburger menu CTA and no video content.
> 
> Does that sound right?

The funnelcake is only going to be made with 38.0.5 and thus the shelf life of the test is until 39.0 is released. 

In all of the conditions in comment 11, with funnelcake or without funnelcake, there is a 50/50 split with the video before or at the bottom regardless of the flow. The 50/50 split for the video should have a condition to only happen for specific locales.

Some logic like this:

If (user.locale in video.locale)
// Check to see if the user's locale is one of the locales we have the video translated in
// The video test is a 50/50 split of showing half the user the video as an overlap onLoad and 
// the other half as a secondary CTA below Fxa Form.
{
 set doVideotest=1
}

If ( f=37 parameter of URL )
// Funnelcake Hamburger menu flow
{
 if (doVideotest==1) {
   flow.videoTest(1a,2a)
 }
 flow.fxaFlow(hamburgerMenu)
}
Else If ( f=38 parameter of URL ) OR (f == undefined) OR ( f != (37|38) )
// Funnelcake Form flow or exception flow
{
 if (doVideotest==1) {
   flow.videoTest(1b,2b)
 }
 flow.fxaFlow(form)
}

Obliviously, this could be refactored, but just wanted to make sure we are on the same page from a logic perspective. :)

Comment 15

[Jonathan Petto [:jpetto]]

Yep - I believe I have exactly that logic in the branch:

https://github.com/mozilla/bedrock/pull/2932/files#diff-53f45955dc30833e483c5b53fd753987R12

Comment 16

[Jonathan Petto [:jpetto]]

Latest test links:

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37 - hamburger menu CTA random video
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=1 - hamburger menu no video on load
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=2 - hamburger menu with video on load

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38 - signup form CTA random video
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=1 - signup form no video on load
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=2 = signup form with video on load

:garethc - Can you provide GA details?

Comment 17

[Chris More [:cmore]]

for f=38, v=1 and v=2 appear to be reversed compared to f=37. jpetto will check and update if needed.

Comment 18

[Chris More [:cmore]]

jpetto found the bug and he's updating. it was random for f=38

Comment 19

[Jonathan Petto [:jpetto]]

Actually, both ?f=37 and ?f=38 were totally random due to a bug when converting to funnelcake decision making.

Fixed in the latest push. &v=1 will now never show the video on load, &v=2 will show the video on load.

Comment 20

[Chris More [:cmore]]

Designs are in already. clearning needinfo for Ty.

Comment 21

[Chris More [:cmore]]

Jon: Why is the font size in the green button in https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=1 different than https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=1. The fxa button flow is smaller than the sync button flow.

Comment 22

[Jonathan Petto [:jpetto]]

Yep, I decreased the font size for the fxa button flow in an attempt to get all copy on one line in the button for en-US.

Can easily be undone if it's unwanted.

Comment 23

[Chris More [:cmore]]

(In reply to Jon Petto [:jpetto] from comment #22)
> Yep, I decreased the font size for the fxa button flow in an attempt to get
> all copy on one line in the button for en-US.
> 
> Can easily be undone if it's unwanted.

How much work would it be to do a 3rd variation that is like f=38 (form), but just don't show any video at all (f=39). Not before and not after. Just the page, copy, and CTA. Why? So we have a control to see how the video impacts retention. 

That would mean we would do 3 variations total like this:

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37 - hamburger menu CTA random video
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=1 - hamburger menu no video on load
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=2 - hamburger menu with video on load

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38 - signup form CTA random video
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=1 - signup form no video on load
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=2 = signup form with video on load

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=39 - signup form CTA no video

We have to do it with f=39 because I am going to funnelcake the non-video variation so we can compare the impact of the video overall. Without a control, we will only know the video's impact to CTR of the button. With a funnelcake control, we can see if there is a usage impact.

Comment 24

[Jonathan Petto [:jpetto]]

Not a problem at all. Only thing to keep in mind is that there will be no video for users in locales without a translated video, regardless of funnelcake value.

Assuming the "hu" locale does not have a translated video, the following URLs would result in the exact same experience:

https://www-demo5.allizom.org/hu/firefox/38.0.5/firstrun/?f=38
https://www-demo5.allizom.org/hu/firefox/38.0.5/firstrun/?f=39

Comment 25

[Chris More [:cmore]]

(In reply to Jon Petto [:jpetto] from comment #24)
> Not a problem at all. Only thing to keep in mind is that there will be no
> video for users in locales without a translated video, regardless of
> funnelcake value.
> 
> Assuming the "hu" locale does not have a translated video, the following
> URLs would result in the exact same experience:
> 
> https://www-demo5.allizom.org/hu/firefox/38.0.5/firstrun/?f=38
> https://www-demo5.allizom.org/hu/firefox/38.0.5/firstrun/?f=39

Understood. I just need one funnelcake with no videos ever so that I can use that as a control for comparison. Let me know when it is available to view on demo5.

Funnelcakes will be ready soon to test the full experience.

Comment 26

[Jonathan Petto [:jpetto]]

?f=39 is up on demo5:

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=39

Comment 27

[Chris More [:cmore]]

Actions for :jpetto:

for the video container

* change margin to 20px instead of 60px
* header font from 3.75rem to 3rem

For all variations

This will fit height-wise 99% of visitors to first run. Previously, 40% of visitors would have had the video slightly cut off on the bottom.

Comment 28

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

We were seeing that the video modal was cutting off the bottom for 768 and lower heights. The changes above will allow these resolutions (which are the most popular on moz.org) to see the full video on load.

NI agibson to keep informed

Comment 29

[Alex Gibson [:agibson]]

(In reply to Chris More [:cmore] from comment #27)
> * change margin to 20px instead of 60px
> * header font from 3.75rem to 3rem

This is shared code in our common modal dialog CSS. To be consistent, we should update it in master and not just on /firstrun. I'll make it so.

Comment 30

[Alex Gibson [:agibson]]

Attachment: GitHub pull request

I made a quick PR here to master, which both myself and Jon should be able to use (as well as any other pages that feature videos)

Comment 31

[Chris More [:cmore]]

(In reply to Alex Gibson [:agibson] from comment #30)
> Created attachment 8608312 [details] [review]
> GitHub pull request
> 
> I made a quick PR here to master, which both myself and Jon should be able
> to use (as well as any other pages that feature videos)

Looks good now on demo5!

Jpetto: is your only action with this bug to get the final videos in the page?

Comment 32

[Chris More [:cmore]]

3 variations to test:

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37 (Sync w/ 50:50 video split)
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38 (Accounts w/ 50:50 video split)
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=39 (Accounts w/o video)

Comment 33

[Jonathan Petto [:jpetto]]

Yep, afaik the only remaining task is to add the videos.

Comment 34

[Gareth Cull [:garethc]]

Here are the GTM tracking requirements for the video:

Can we please track the following interactions on the 

1) Video Interactions - Play:

On video play please call:

window.dataLayer.push({
'event': 'firstrun-38.0.5-video',
'interaction': 'start',
'videoTitle': 'When its Personal Campaign Video'
});


2) On completion of the Campaign video playing, please call:

window.dataLayer.push({
'event': 'firstrun-38.0.5-video',
'interaction': 'finish',
'videoTitle': 'When its Personal Campaign Video'
});

We'll track FxA link and button clicks from within GTM. One thing I'm waiting to discuss with AP is how we hook into and track the split variations with custom dimensions. I'll update the bug with any GTM reqs once I talk with them later today.

Comment 35

[Jeremy Dillow]

In the core dataLayer push (see example below), please add a property named "variation" whose value is either "1" or "2", depending upon which variation is being rendered.  This will provide a way of tracking which variation was rendered for a given session.

For example:

<script>
    window.dataLayer = window.dataLayer || [];
    window.dataLayer.push({
        'event': 'dataLayer-initialized',
        'page': {
            'category': 'firstrun-38.0.5',
            'variation': '<1|2>',                            // either '1' or '2'
            'type': '<template_or_however_you_want_to_use>', // page template names e.g. "First Run Landing Page"
            'breadcrumb': '<page_breadcrumbs>'               // full site breadcrumb
        }
    });
</script>

Comment 36

[Chris More [:cmore]]

Matt:

To verify on demo5, a pref change needs to be made:

1) Start with fresh profile
2) In about:config, add browser.uitour.testingOrigins with value of https://www-demo5.allizom.org
3) Try variations below

Main areas to verify the variations:

1) For f=37, test to ensure, 50% get video as overlay and 50% get video below CTA
2) For f=37, test to ensure the CTA opens the hamburger menu and highlights sync

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37 - hamburger menu CTA random video
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=1 - hamburger menu no video on load
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=2 - hamburger menu with video on load

3) For f=38, test to ensure, 50% get video as overlay and 50% get video below CTA
4) For f=38, test to ensure the CTA opens the Firefox Accounts form

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38 - signup form CTA random video
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=1 - signup form no video on load
https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=2 = signup form with video on load

5) test to ensure the CTA opens the Firefox Accounts form and *no* video is displayed

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=39 - signup form CTA no video

6) Verify that FxA email verification snippet points to Android download and iOS coming soon link.

7) Verify the "Already have one? Sign in." works for people who already have an account

8) Verify that when no funnelcake parameter is in the URL, the user gets the Firefox Accounts flow (exactly the same as f=38)

https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/ (no funnelcake parameter)

Comment 37

[Gareth Cull [:garethc]]

jpetto,

Can you confirm that we have added 'variation': '<1|2>' to the core dataLayer which appears above the GTM tag? Let us know if you are able to do this, else we may have to re-configure how we capture the custom dimension. Thanks.

Comment 38

[Bob Silverberg [:bsilverberg]]

(In reply to Chris More [:cmore] from comment #36)

I have verified almost everything in comment #36:
> 
> 1) For f=37, test to ensure, 50% get video as overlay and 50% get video
> below CTA
- verified that the versions appear with what seems like 50% frequency

> 2) For f=37, test to ensure the CTA opens the hamburger menu and highlights
> sync
> 
> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37 -
> hamburger menu CTA random video
- verified

> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=1 -
> hamburger menu no video on load
- verified

> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=37&v=2 -
> hamburger menu with video on load
- verified

> 3) For f=38, test to ensure, 50% get video as overlay and 50% get video
> below CTA
- as above, seems like approx 50%

> 4) For f=38, test to ensure the CTA opens the Firefox Accounts form
> 
> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38 - signup
> form CTA random video
- verified

> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=1 -
> signup form no video on load
- verified

> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=38&v=2 =
> signup form with video on load
- verified

> 5) test to ensure the CTA opens the Firefox Accounts form and *no* video is
> displayed
> 
> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/?f=39 - signup
> form CTA no video
- verififed

> 6) Verify that FxA email verification snippet points to Android download and
> iOS coming soon link.
- verified that snippet points to Android download
- could not verify iOS coming soon link as it is not live on FxA server yet (discussed this with jpetto)

> 7) Verify the "Already have one? Sign in." works for people who already have
> an account
- verified

> 8) Verify that when no funnelcake parameter is in the URL, the user gets the
> Firefox Accounts flow (exactly the same as f=38)
> 
> https://www-demo5.allizom.org/en-US/firefox/38.0.5/firstrun/ (no funnelcake
> parameter)
- verified

Comment 39

[Gareth Cull [:garethc]]

Video events, button and link clicks are now being tracked as events. Here's a screencap of the events coming through to GA:
https://drive.google.com/file/d/0B6voOaUZL-jwWUxwVVd6a0YtUVU/view?usp=sharing

Still waiting on custom dimension implementation and will wrap that up today.

Gareth

Comment 40

[Chris More [:cmore]]

Thanks, :bsilverberg for the QA!

Looks like we are good to go here except for the custom dimension. 

Let us know Gareth when that is great.

Comment 41

[Gareth Cull [:garethc]]

Custom dimensions are now verified. 

Here's a couple screen shots showing them come into cd7:
http://cl.ly/image/0A3B3x2H2342
http://cl.ly/image/2e0Q1N06361e

Comment 42

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/e66925c8791b352546098d225ccd83a101e37208
Bug 1136937. Fx 38.0.5 firstrun.

https://github.com/mozilla/bedrock/commit/0bddac5c521bb5066b091ebe85271a6105a779a8
Merge pull request #2932 from jpetto/bug-1136937-fx38.0.5-firstrun

Bug 1136937. Fx 38.0.5 firstrun.

Comment 43

[Chris More [:cmore]]

On production:

https://www.mozilla.org/en-US/firefox/38.0.5/firstrun/

Funnelcakes:

https://www.mozilla.org/en-US/firefox/38.0.5/firstrun/?f=37
https://www.mozilla.org/en-US/firefox/38.0.5/firstrun/?f=38
https://www.mozilla.org/en-US/firefox/38.0.5/firstrun/?f=39