695473 - [brand-Q4] Launch & Monitor new Firefox download page with stronger brand messaging (en-US)

Status: VERIFIED FIXED

(www.mozilla.org :: General, defect)

Description

[John Slater]

One of Engagement's Q4 goals is to communicate our brand messaging more boldly and effectively during the Firefox download experience. 

I've filed bug 695462 and bug 695465 to track the copy and design portion of the work, but also want to make sure this is on the web team's radar. 

The assets are due on 11/3...Laura, what's the next release after that we can realistically get this into?

Comment 1

[Laura Forrest]

Hey John - Thanks for filing. For next steps please enter the final copy - or copy variations we'd like to test, here and then Steven can help us build out a new page with this new copy. 

In terms of testing: we'll definitely want to test this new homepage copy before pushing live to 100% of users, so let us know the copy the sooner the better. In the past we've found that modifying copy does not affect download conversion so I'd expect to see the same this time, although we still want to make sure we test.

Comment 2

[Laura Forrest]

Steven - Please create a separate URL for this so we can test how this converts. We'll want to direct 50% of the traffic here, 50% to /firefox/new. This can be called something like /firefox/new2. 

Thanks!

John will update this bug with the new copy when it's ready.

Comment 3

[John Slater]

Actually, can we test two here? If so, we'd have 50% going to the regular page and 25% going to both test pages.

The test copy has been finalized and is ready to go. Here is is:

headline 1: Different by Design
bullets 1: Proudly non-profit   User-focused innovation   Fast, flexible, secure

headline 2: The People's Browser
bullets 2: Proudly non-profit   User-focused innovation   Fast, flexible, secure

All other elements (button, etc) are the same.

Thanks!

Comment 4

[Kristin Baird]

Lets make sure we are clear on what we are testing for. Are we testing for downloads w/ the headlines?

Comment 5

[Laura Forrest]

(In reply to John Slater from comment #3)
> Actually, can we test two here? If so, we'd have 50% going to the regular
> page and 25% going to both test pages.
> 
> The test copy has been finalized and is ready to go. Here is is:
> 
> headline 1: Different by Design
> bullets 1: Proudly non-profit   User-focused innovation   Fast, flexible,
> secure
> 
> headline 2: The People's Browser
> bullets 2: Proudly non-profit   User-focused innovation   Fast, flexible,
> secure
> 
> All other elements (button, etc) are the same.
> 
> Thanks!

Yes, that's fine. 

Steven - can you create two new URLs to test this? Perhaps /firefox/new1 and /firefox/new2?

This is ready for Dev! Sooner the better on this one. HTML changes only (and, er, two new pages) so should be semi easy. Getting this into tomorrows batch would be great.

Comment 6

[Laura Forrest]

(In reply to Kristin Baird from comment #4)
> Lets make sure we are clear on what we are testing for. Are we testing for
> downloads w/ the headlines?

In this instance we're testing to make sure the new versions perform at a similar download conversion rate as our current version. We want to monitor any change to this high traffic, important page.

For any additional tests (retention rates, non-profit awareness) let's talk off-bug and create new bugs/talk to metrics team, etc.

Comment 7

[Laura Forrest]

Steven - Please do use "new1" and "new2" as the URLs - I have created custom reporting based on those URLs and we won't be able to track download conversion rate if any other URLs are used. Thanks!

Comment 8

[Steven Garrity [:sgarrity]]

These are now setup in trunk:

http://www-dev.allizom.org/en-US/firefox/new/
http://www-dev.allizom.org/en-US/firefox/new1/
http://www-dev.allizom.org/en-US/firefox/new2/

Comment 9

[Steven Garrity [:sgarrity]]

Made a text update requested by Kristin Baird via email in trunk (replaced "User-focused innovation" with "Innovating for you").

Comment 10

[Steven Garrity [:sgarrity]]

Now that we have the text changes implemented and Laura has a stats/logging setup implemented via webtrends, we just need to get some actual traffic to these pages.

The simplest method I can thing of would be a javascript redirect, but this is ugly in a few respects:

A. It will involve a performance hit - the extra round-trip to the server with slow the pages down a bit. It may also have a brief flash of the old page before the new page loads.
B. Noise in the stats - if we're comparing pages and some redirect and others don't, then we're polluting the test results. We may learn that the page is less effective, but won't know if it's due to the text changes, or the redirect.

One possible (but ugly) solution would be to setup four distinct paths for visitors:

1. The default page - no redirect
2. The default page with a redirect
3. The /new1   page with a redirect
4. the /new2   page with a redirect   

Most people can be sent to option 1., which don't have any performance impact. The test can be based on #2, #3, and #4, which will give the default, new1, and new2 an even playing field with the redirect performance hit.

As a side-effect of this method, we'll also be able to compare the default with with and without a redirect which would help determine how much the redirect is influencing our other results.

Any other suggestions? Something server-side would be ideal, but I'm not sure we can do that easily with our current setup.

Comment 11

[mcbmoz]

adding Corey here too, since this bug depends on bug 700746

Comment 12

[mcbmoz]

(In reply to Steven Garrity from comment #10)
> Now that we have the text changes implemented and Laura has a stats/logging
> setup implemented via webtrends, we just need to get some actual traffic to
> these pages.
> 
> The simplest method I can thing of would be a javascript redirect, but this
> is ugly in a few respects:
> 
> A. It will involve a performance hit - the extra round-trip to the server
> with slow the pages down a bit. It may also have a brief flash of the old
> page before the new page loads.
> B. Noise in the stats - if we're comparing pages and some redirect and
> others don't, then we're polluting the test results. We may learn that the
> page is less effective, but won't know if it's due to the text changes, or
> the redirect.
> 
> One possible (but ugly) solution would be to setup four distinct paths for
> visitors:
> 
> 1. The default page - no redirect
> 2. The default page with a redirect
> 3. The /new1   page with a redirect
> 4. the /new2   page with a redirect   
> 
> Most people can be sent to option 1., which don't have any performance
> impact. The test can be based on #2, #3, and #4, which will give the
> default, new1, and new2 an even playing field with the redirect performance
> hit.
> 
> As a side-effect of this method, we'll also be able to compare the default
> with with and without a redirect which would help determine how much the
> redirect is influencing our other results.
> 
> Any other suggestions? Something server-side would be ideal, but I'm not
> sure we can do that easily with our current setup.

Steven, just added you to bug 700746 - question is over to Corey on when/if sitespect can be redeployed. If not, we'll need to move forward with your recommended solution in comment 10.

Comment 13

[John Slater]

Resolving this per https://bugzilla.mozilla.org/show_bug.cgi?id=704645#c6. Thanks all.

Comment 14

[Steven Garrity [:sgarrity]]

Reverted the .htaccess changes from this bug in trunk in r98082.

Comment 15

[raymond [:retornam] (inactive)]

qa-verified trunk http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/.htaccess?r1=98082&r2=98081&pathrev=98082

Comment 16

[Laura Forrest]

Looped back with John and we don't want to launch the new landing page without measuring the affect it has on download. That's not the direct objective of this campaign, but it is the objective of the page, and knowing the affect this page has on download is important before diving straight in.

Plan: Launch the new page at the /new1 URL. Send 75% of traffic to this new page. Send 25% of traffic to the current /new page.  

Dependencies: None - we can measure this manually through Webtrends and do not need Sitespect. Sitespect would be a lot easier, but we can do this manually for now, and I've setup the reporting to do so.

Comment 17

[Laura Forrest]

(In reply to Laura Forrest from comment #16)
> Looped back with John and we don't want to launch the new landing page
> without measuring the affect it has on download. That's not the direct
> objective of this campaign, but it is the objective of the page, and knowing
> the affect this page has on download is important before diving straight in.
> 
> Plan: Launch the new page at the /new1 URL. Send 75% of traffic to this new
> page. Send 25% of traffic to the current /new page.  
> 
> Dependencies: None - we can measure this manually through Webtrends and do
> not need Sitespect. Sitespect would be a lot easier, but we can do this
> manually for now, and I've setup the reporting to do so.

Looks like we aren't able to technically split traffic between two pages after-all. If we could, then we'd be able to use Webtrends to measure, but we can't without Zeus or SiteSpect. 

That said, let's launch this to 100% of our users and then monitor the affect it has on downloads and non-profit awareness in real-time. Not ideal, but we will be able to monitor closely to see the affect this change has and compare it to conversion wk/wk. 

For those results follow Bug 702414.

Comment 18

[James Long (:jlongster)]

pushed to production r98231

Comment 19

[James Long (:jlongster)]

I changed the large image on the page to be served from the CDN in r98241

Comment 20

[Steven Garrity [:sgarrity]]

(In reply to James Long (:jlongster) from comment #19)
> I changed the large image on the page to be served from the CDN in r98241

Thanks James. Since the CSS itself is usually served from the CDN, we use relative URLs for the images in CSS, so they come from the CDN too. I forgot that in this case we are including the CSS inline in the HTML and that doesn't work.

Comment 21

[James Long (:jlongster)]

(In reply to Steven Garrity from comment #20)
> (In reply to James Long (:jlongster) from comment #19)
> > I changed the large image on the page to be served from the CDN in r98241
> 
> Thanks James. Since the CSS itself is usually served from the CDN, we use
> relative URLs for the images in CSS, so they come from the CDN too. I forgot
> that in this case we are including the CSS inline in the HTML and that
> doesn't work.

Oh that's right, that's how that works. I'm not convinced that we need to serve everything inline; as long as we compress everything into one file, we should be able to serve it from the CDN without problems.

We'll fix that in bedrock though!

Comment 22

[raymond [:retornam] (inactive)]

verified fixed http://www.mozilla.org/en-US/firefox/new/