If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

[brand-Q4] Launch & Monitor new Firefox download page with stronger brand messaging (en-US)

VERIFIED FIXED in 4.8

Status

www.mozilla.org
General
VERIFIED FIXED
6 years ago
5 years ago

People

(Reporter: John Slater, Assigned: sgarrity)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

6 years ago
One of Engagement's Q4 goals is to communicate our brand messaging more boldly and effectively during the Firefox download experience. 

I've filed bug 695462 and bug 695465 to track the copy and design portion of the work, but also want to make sure this is on the web team's radar. 

The assets are due on 11/3...Laura, what's the next release after that we can realistically get this into?
(Reporter)

Updated

6 years ago
OS: Mac OS X → All
Hardware: x86 → All
Summary: [brand=Q4] update Firefox download page with stronger brand messaging → [brand-Q4] update Firefox download page with stronger brand messaging
(Reporter)

Updated

6 years ago
Blocks: 695478

Comment 1

6 years ago
Hey John - Thanks for filing. For next steps please enter the final copy - or copy variations we'd like to test, here and then Steven can help us build out a new page with this new copy. 

In terms of testing: we'll definitely want to test this new homepage copy before pushing live to 100% of users, so let us know the copy the sooner the better. In the past we've found that modifying copy does not affect download conversion so I'd expect to see the same this time, although we still want to make sure we test.
Assignee: nobody → jslater
Target Milestone: --- → 4.5

Updated

6 years ago
Target Milestone: 4.5 → 4.6

Comment 2

6 years ago
Steven - Please create a separate URL for this so we can test how this converts. We'll want to direct 50% of the traffic here, 50% to /firefox/new. This can be called something like /firefox/new2. 

Thanks!

John will update this bug with the new copy when it's ready.
(Reporter)

Comment 3

6 years ago
Actually, can we test two here? If so, we'd have 50% going to the regular page and 25% going to both test pages.

The test copy has been finalized and is ready to go. Here is is:

headline 1: Different by Design
bullets 1: Proudly non-profit   User-focused innovation   Fast, flexible, secure

headline 2: The People's Browser
bullets 2: Proudly non-profit   User-focused innovation   Fast, flexible, secure

All other elements (button, etc) are the same.

Thanks!

Comment 4

6 years ago
Lets make sure we are clear on what we are testing for. Are we testing for downloads w/ the headlines?

Comment 5

6 years ago
(In reply to John Slater from comment #3)
> Actually, can we test two here? If so, we'd have 50% going to the regular
> page and 25% going to both test pages.
> 
> The test copy has been finalized and is ready to go. Here is is:
> 
> headline 1: Different by Design
> bullets 1: Proudly non-profit   User-focused innovation   Fast, flexible,
> secure
> 
> headline 2: The People's Browser
> bullets 2: Proudly non-profit   User-focused innovation   Fast, flexible,
> secure
> 
> All other elements (button, etc) are the same.
> 
> Thanks!

Yes, that's fine. 

Steven - can you create two new URLs to test this? Perhaps /firefox/new1 and /firefox/new2?

This is ready for Dev! Sooner the better on this one. HTML changes only (and, er, two new pages) so should be semi easy. Getting this into tomorrows batch would be great.
Assignee: jslater → steven

Comment 6

6 years ago
(In reply to Kristin Baird from comment #4)
> Lets make sure we are clear on what we are testing for. Are we testing for
> downloads w/ the headlines?

In this instance we're testing to make sure the new versions perform at a similar download conversion rate as our current version. We want to monitor any change to this high traffic, important page.

For any additional tests (retention rates, non-profit awareness) let's talk off-bug and create new bugs/talk to metrics team, etc.

Comment 7

6 years ago
Steven - Please do use "new1" and "new2" as the URLs - I have created custom reporting based on those URLs and we won't be able to track download conversion rate if any other URLs are used. Thanks!

Updated

6 years ago
Blocks: 702414
(Assignee)

Comment 8

6 years ago
These are now setup in trunk:

http://www-dev.allizom.org/en-US/firefox/new/
http://www-dev.allizom.org/en-US/firefox/new1/
http://www-dev.allizom.org/en-US/firefox/new2/
Keywords: qawanted
Whiteboard: r=97874, b=trunk
Target Milestone: 4.6 → 4.7
(Assignee)

Updated

6 years ago
Whiteboard: r=97874, b=trunk → r=97874,97875 b=trunk
(Assignee)

Comment 9

6 years ago
Made a text update requested by Kristin Baird via email in trunk (replaced "User-focused innovation" with "Innovating for you").
Whiteboard: r=97874,97875 b=trunk → r=97874,97875,97901 b=trunk
(Assignee)

Comment 10

6 years ago
Now that we have the text changes implemented and Laura has a stats/logging setup implemented via webtrends, we just need to get some actual traffic to these pages.

The simplest method I can thing of would be a javascript redirect, but this is ugly in a few respects:

A. It will involve a performance hit - the extra round-trip to the server with slow the pages down a bit. It may also have a brief flash of the old page before the new page loads.
B. Noise in the stats - if we're comparing pages and some redirect and others don't, then we're polluting the test results. We may learn that the page is less effective, but won't know if it's due to the text changes, or the redirect.

One possible (but ugly) solution would be to setup four distinct paths for visitors:

1. The default page - no redirect
2. The default page with a redirect
3. The /new1   page with a redirect
4. the /new2   page with a redirect   

Most people can be sent to option 1., which don't have any performance impact. The test can be based on #2, #3, and #4, which will give the default, new1, and new2 an even playing field with the redirect performance hit.

As a side-effect of this method, we'll also be able to compare the default with with and without a redirect which would help determine how much the redirect is influencing our other results.

Any other suggestions? Something server-side would be ideal, but I'm not sure we can do that easily with our current setup.

Comment 11

6 years ago
adding Corey here too, since this bug depends on bug 700746
Depends on: 700746

Updated

6 years ago
Summary: [brand-Q4] update Firefox download page with stronger brand messaging → [brand-Q4] A/B Test Firefox download page with stronger brand messaging (en-US)

Comment 12

6 years ago
(In reply to Steven Garrity from comment #10)
> Now that we have the text changes implemented and Laura has a stats/logging
> setup implemented via webtrends, we just need to get some actual traffic to
> these pages.
> 
> The simplest method I can thing of would be a javascript redirect, but this
> is ugly in a few respects:
> 
> A. It will involve a performance hit - the extra round-trip to the server
> with slow the pages down a bit. It may also have a brief flash of the old
> page before the new page loads.
> B. Noise in the stats - if we're comparing pages and some redirect and
> others don't, then we're polluting the test results. We may learn that the
> page is less effective, but won't know if it's due to the text changes, or
> the redirect.
> 
> One possible (but ugly) solution would be to setup four distinct paths for
> visitors:
> 
> 1. The default page - no redirect
> 2. The default page with a redirect
> 3. The /new1   page with a redirect
> 4. the /new2   page with a redirect   
> 
> Most people can be sent to option 1., which don't have any performance
> impact. The test can be based on #2, #3, and #4, which will give the
> default, new1, and new2 an even playing field with the redirect performance
> hit.
> 
> As a side-effect of this method, we'll also be able to compare the default
> with with and without a redirect which would help determine how much the
> redirect is influencing our other results.
> 
> Any other suggestions? Something server-side would be ideal, but I'm not
> sure we can do that easily with our current setup.

Steven, just added you to bug 700746 - question is over to Corey on when/if sitespect can be redeployed. If not, we'll need to move forward with your recommended solution in comment 10.
(Assignee)

Updated

6 years ago
Target Milestone: 4.7 → 4.9
(Reporter)

Comment 13

6 years ago
Resolving this per https://bugzilla.mozilla.org/show_bug.cgi?id=704645#c6. Thanks all.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → WONTFIX
(Assignee)

Comment 14

6 years ago
Reverted the .htaccess changes from this bug in trunk in r98082.
Whiteboard: r=97874,97875,97901 b=trunk → r=97874,97875,97901,98082 b=trunk
qa-verified trunk http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/.htaccess?r1=98082&r2=98081&pathrev=98082
Status: RESOLVED → VERIFIED
Keywords: qawanted

Comment 16

6 years ago
Looped back with John and we don't want to launch the new landing page without measuring the affect it has on download. That's not the direct objective of this campaign, but it is the objective of the page, and knowing the affect this page has on download is important before diving straight in.

Plan: Launch the new page at the /new1 URL. Send 75% of traffic to this new page. Send 25% of traffic to the current /new page.  

Dependencies: None - we can measure this manually through Webtrends and do not need Sitespect. Sitespect would be a lot easier, but we can do this manually for now, and I've setup the reporting to do so.
Status: VERIFIED → REOPENED
No longer depends on: 700746
Resolution: WONTFIX → ---
Summary: [brand-Q4] A/B Test Firefox download page with stronger brand messaging (en-US) → [brand-Q4] Launch & Monitor new Firefox download page with stronger brand messaging (en-US)
Target Milestone: 4.9 → 4.8

Comment 17

6 years ago
(In reply to Laura Forrest from comment #16)
> Looped back with John and we don't want to launch the new landing page
> without measuring the affect it has on download. That's not the direct
> objective of this campaign, but it is the objective of the page, and knowing
> the affect this page has on download is important before diving straight in.
> 
> Plan: Launch the new page at the /new1 URL. Send 75% of traffic to this new
> page. Send 25% of traffic to the current /new page.  
> 
> Dependencies: None - we can measure this manually through Webtrends and do
> not need Sitespect. Sitespect would be a lot easier, but we can do this
> manually for now, and I've setup the reporting to do so.

Looks like we aren't able to technically split traffic between two pages after-all. If we could, then we'd be able to use Webtrends to measure, but we can't without Zeus or SiteSpect. 

That said, let's launch this to 100% of our users and then monitor the affect it has on downloads and non-profit awareness in real-time. Not ideal, but we will be able to monitor closely to see the affect this change has and compare it to conversion wk/wk. 

For those results follow Bug 702414.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
pushed to production r98231
Whiteboard: r=97874,97875,97901,98082 b=trunk
I changed the large image on the page to be served from the CDN in r98241
(Assignee)

Comment 20

6 years ago
(In reply to James Long (:jlongster) from comment #19)
> I changed the large image on the page to be served from the CDN in r98241

Thanks James. Since the CSS itself is usually served from the CDN, we use relative URLs for the images in CSS, so they come from the CDN too. I forgot that in this case we are including the CSS inline in the HTML and that doesn't work.
(In reply to Steven Garrity from comment #20)
> (In reply to James Long (:jlongster) from comment #19)
> > I changed the large image on the page to be served from the CDN in r98241
> 
> Thanks James. Since the CSS itself is usually served from the CDN, we use
> relative URLs for the images in CSS, so they come from the CDN too. I forgot
> that in this case we are including the CSS inline in the HTML and that
> doesn't work.

Oh that's right, that's how that works. I'm not convinced that we need to serve everything inline; as long as we compress everything into one file, we should be able to serve it from the CDN without problems.

We'll fix that in bedrock though!
verified fixed http://www.mozilla.org/en-US/firefox/new/
Status: RESOLVED → VERIFIED
Component: www.mozilla.org/firefox → www.mozilla.org
Product: Websites → Websites
Component: www.mozilla.org → General
Product: Websites → www.mozilla.org
You need to log in before you can comment on or make changes to this bug.