689746 - [FX Download Testing] Implement A/B Testing on Firefox Download Page en-US

Status: RESOLVED WONTFIX

(www.mozilla.org :: General, defect)

Description

[mcbmoz]

Hey Steven,

I've set this for next week's release, but it can be pushed to 4.1 or 4.2 based on your workload. 

We just need the markup, because we'll work with Blake to implement in sitespect to a small testing audience.

Here are 3 versions we would like to test & links to the PSDs:

JPGs for your viewing pleasure:

http://mozilla.seanmartell.com/engagement/mocodotcom/testing/home_nonfxusers_test_v1.jpg

http://mozilla.seanmartell.com/engagement/mocodotcom/testing/home_nonfxusers_test_v2.jpg

Blue version:
http://cl.ly/192b3t3K0j3o1B3r3s2y

PSDs:

http://mozilla.seanmartell.com/engagement/mocodotcom/testing/home_nonfxusers_test.psd


http://mozilla.seanmartell.com/engagement/mocodotcom/testing/home_nonfxusers_revised.psd

Comment 1

[mcbmoz]

Hey Steven, just moved to mileston 4.4, think you guys can make this happen?

Comment 2

[mcbmoz]

Ping Steven. Thoughts on when you can get to this?

Comment 3

[Laura Forrest]

+1. 

The highest priority in terms of testing are the two Photo versions (red and blue). Since it looks like we're de-prioritizing mobile at the moment, there may not be a need to build out that version. 

URLs can be: 

www.mozilla.org/en-US/firefox/new/red
www.mozilla.org/en-US/firefox/new/blue
(Or www.mozilla.org/en-US/firefox/red if you prefer)

Thanks!

Comment 4

[mcbmoz]

ping on this steven - how are we looking for this week?

Comment 5

[Laura Forrest]

Hey Steven - We'd like to launch the blue group photo version next week. How does that sound to you?

Comment 6

[Steven Garrity [:sgarrity]]

I see the comments on this bug have become a trail of shame of my non-responsiveness... Sorry about that. Other short-term issues kept cropping up and pushing this back.

I'll work on the blue version for 4.6 - I can't guarantee that it will be ready for Tuesday, but will push early in the week anyhow. Maybe we can push out of the release cycle if we're a bit late for 4.6.

Comment 7

[Laura Forrest]

(In reply to Steven Garrity from comment #6)
> I see the comments on this bug have become a trail of shame of my
> non-responsiveness... Sorry about that. Other short-term issues kept
> cropping up and pushing this back.
> 
> I'll work on the blue version for 4.6 - I can't guarantee that it will be
> ready for Tuesday, but will push early in the week anyhow. Maybe we can push
> out of the release cycle if we're a bit late for 4.6.

Sounds like a good plan - pushing out of the cycle is totally fine. Thanks!

Comment 8

[Steven Garrity [:sgarrity]]

These three variations are now built out in trunk and ready for QA:

http://www-dev.allizom.org/en-US/firefox/new/red/
http://www-dev.allizom.org/en-US/firefox/new/blue/
http://www-dev.allizom.org/en-US/firefox/new/light/

Do they need any special Webtrends magic applied? How should we go about distributing traffic to them?

Comment 9

[mcbmoz]

(In reply to Steven Garrity from comment #8)
> These three variations are now built out in trunk and ready for QA:
> 
> http://www-dev.allizom.org/en-US/firefox/new/red/
> http://www-dev.allizom.org/en-US/firefox/new/blue/
> http://www-dev.allizom.org/en-US/firefox/new/light/
> 
> Do they need any special Webtrends magic applied? How should we go about
> distributing traffic to them?

Adding Blake & Brad to answer both questions:

How do we direct traffic to the three tests, how much traffic?

Do we need any special webtrends for tracking?

Comment 10

[Bradley Gross]

I just took a look at the (staging) pages and they're sending hits to Webtrends!  We're all go here.

Comment 11

[mcbmoz]

(In reply to Bradley Gross from comment #10)
> I just took a look at the (staging) pages and they're sending hits to
> Webtrends!  We're all go here.

Brad, are you able to tell which page is sending # of downloads? We're trying to figure out how to do A/B testing these versions. Unfortunately the tool we usually use is not working right now.

Comment 12

[Bradley Gross]

When a user clicks the download button it will send a hit to our data collection server that identifies the source page ( ../red/).  We may have to set up a custom report to track that, but we will have the data regardless.

Comment 13

[mcbmoz]

(In reply to Bradley Gross from comment #12)
> When a user clicks the download button it will send a hit to our data
> collection server that identifies the source page ( ../red/).  We may have
> to set up a custom report to track that, but we will have the data
> regardless.

Brad, this is great, would you be able to create three custom reports? Should I file a new bug?

Comment 14

[Bradley Gross]

We should be able to do it in a single custom report.  I will create it

Comment 15

[mcbmoz]

(In reply to Bradley Gross from comment #14)
> We should be able to do it in a single custom report.  I will create it

Thank YOU!

Comment 16

[raymond [:retornam] (inactive)]

qa-verified-trunk

http://www-dev.allizom.org/en-US/firefox/new/red/
http://www-dev.allizom.org/en-US/firefox/new/blue/
http://www-dev.allizom.org/en-US/firefox/new/light/

Comment 17

[Bradley Gross]

The custom report has already been created in the Mozilla.org/firefox (50% sample) profile, it is entitled:

Firefox Download Clicks by Source Page