Closed Bug 820974 Opened 12 years ago Closed 12 years ago

Implement A/B Test for new /firefox/new page

Categories

(www.mozilla.org :: General, defect)

Product:
www.mozilla.org
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cmore, Assigned: pmac)

References

Details

(Whiteboard: u=admin c=analytics p=) 

We need a way to send a sampled amount of traffic to the new version of the /firefox/new/ page in Bedrock or .htaccess.
This will be an en-US *only* test.

The current page lives at:

http://www.mozilla.org/en-US/firefox/new/

The new page should live at:

http://www.mozilla.org/en-US/firefox/new-b/

I'm leaning toward making the current page live at:

http://www.mozilla.org/en-US/firefox/new-a/

and then we make traffic to http://www.mozilla.org/en-US/firefox/new/ redirect on a sampled rate to either new-a or new-b.

This should only affect the en-US version of the existing /firefox/new/ page and not other locales.

90% of traffic should go to new-a
10% of traffic should go to new-b (new version)

We should also make sure that we can easily back out these changes once the test is over with.
Whiteboard: u=admin c=analytics p=
Jon/Pmac: I'm talking to jakem in IT to see how they did something like this for the Firefox Stub installer A/B test.
We may be able to do this with Apache instead of Django.
Assignee: nobody → pmac
pmac has decided that we are going use Google Content Experiments for this A/B test and it looks very easy. We will need to add JavaScript to the top of the <head> of the existing /firefox/new/ page. We will need to add this on the demo2 servers for test and then new code will be generated for production.
Please provide the full URL here to the new-b page on demo2 and I will include it in the experiment. Once that is in place, I will provide the GA code for demo2.
Two things need to happen on demo2:

1) All the Google Analytics code that is on production needs to be on demo2.

2) Add this code to /firefox/new/ (php) page just after the opening <head>

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='66842818-1',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->

The new page is here:

http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/en-US/firefox/new/index.html?view=markup

but I think the head is here on line 48:

http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/includes/header.inc.php?revision=111396&view=markup

We will need a condition on it so that it only displays when the CGI URL contains "en-US/firefox/new/" and that should catch when it has the /b/ or not and would only display the GA experiment code only one one page with a shared header.
something like this:

if(substr($_SERVER['PATH_INFO'], "/en-US/firefox/new/", 1)) {

// show GA experiment code

}
Update: /en-US/firefox/new/ is already in bedrock and not PHP, but the same thing would need to happen with the header to only display the code when the path include /en-US/firefox/new/ or some other slick way to do it in DJango.

https://github.com/mozilla/bedrock/blob/master/apps/firefox/templates/firefox/new.html
Changes are made and the branch has been rebased to master and thus has the GA code. Demo2 still doesn't have the GA account # in the settings, so it's not working, but it will once that's added. Everything should be ready for the test, and has been pushed to demo2.

\o/
Depends on: 822443
The A/B test is up and running at 25% on this page:

https://www-demo2.allizom.org/en-US/firefox/new/

Please visit the page, click the download buttons, clear your cookies/cache and try again. We need to get data to this page to test the GA reports.
The A/B test for this project suddenly got more complicated. GA wants one goal assign as a metric to an experiment. The issue is that the existing and new page have different goals.

Existing page:

Step 1: visit /en-US/firefox/new/
Step 2: visit /en-US/products/download.html

New page:

Step 1: visit /en-US/firefox/new-b/
Step 2: visit /en-US/firefox/download/

When you create the experiment it ones one goal and the destination URL is different from the existing and new page as seen above. 

As a test, I tried to create a goal with different destination pages and step 1 pages. I created a destination URL like:

Goal URL: ^/en-US/(products/download\.html|firefox/new-b/download/).*$

Step 1: ^/en-US/firefox/(new/|new-b/).*$

The regex says that if either the download.html page or the /new-b/download/ page reached when the referring page is /firefox/new/ or /firefox/new-b/.

While it saved and I was able to create an experiment, I don't know if conversions are going to be measured across the experiment.

An alternative would be to make the virtual page view for the button click on new-b to be /products/download.html instead of /firefox/new-b/download/ so that it has the same goal destination. That idea is documented in this comment: https://bugzilla.mozilla.org/show_bug.cgi?id=821016#c8
The experiment is running at 100% sample at: https://www-demo2.allizom.org/en-US/firefox/new/

You must clear cache and cookies to test the experiment again.
The variation 1 URL is: https://www-demo2.allizom.org/b/en-US/firefox/new-b/, but visitors should never be visiting this page directly. The Google Analytics code is on the URL in comment 12 that samples the audience.
This is the message I sent to Google support this evening:

Issue Summary: 

We are conducting an en-US only A/B test of a new product download page. We are still working on the page on our demo servers and it is not on our production website. We would like to do the A/B test this week to stay on track with the project.

You can view the existing product download page on our production website:

http://www.mozilla.org/en-US/firefox/new/

The interaction on the existing page works as follows:

1) User visits http://www.mozilla.org/en-US/firefox/new/.
2) User decides to click download button.
3) User is redirected to: http://www.mozilla.org/en-US/products/download.html?product=firefox-17.0.1&os=osx&lang=en-US
4) Save-as dialog box pop ups.
5) Instructions are displayed.

We have a goal funnel set up that has the URL destination is set to:

^.+\/products\/download\.html.*$

The first and required step of the goal funnel is set to:

^.+\/firefox\/new\/.*$

This goal funnel works and is straightforward.

This is a working version of the new version that we are building:

https://www-demo2.allizom.org/b/en-US/firefox/new-b/

The steps on how the new-b version works:

1) User visits new-b page.
2) User decides to download and clicks button.
3) Save-as box pops
4) Instructions are displayed

They key difference is that a user is *not* redirected to another page and the URL does not change.

This makes the conversion funnel set up a bit more tricky because we cannot use a different destination URL. What we did is set to virtual page view URL like:

_gaq.push(['_trackPageview', '/en-US/firefox/new-b/download/']);

This made the conversion funnel work on the new page even though the URL always stayed the same.

So, now we want to do an A/B test of the existing page (/firefox/new/) vs the new version (/firefox/new-b/). We set up the experiment in GA on our www-demo2.allizom.org profile, got the experiment code, and inserted it in this control page:

https://www-demo2.allizom.org/en-US/firefox/new/

We set the following URL as the variation 1:

https://www-demo2.allizom.org/b/en-US/firefox/new-b/

We were able to do a sampled A/B test without much effort, but we are having a problem with coming up with the "goal" for the experiment conversion. We are having an issue on how to set the "Select an Experiment objective" given that the objective is tied to a goal and the goal of each version is different.

The goal of the current page is to go to from /firefox/new/ --> /products/download.html and the goal of the new version is to go from /firefox/new-b/ to /firefox/new-b/download/. 

Question 1: Can we do an A/B test and get a true A/B conversion winner if the "goal" of each variation is different?

Question 2: Would it be easier if the virtual page view of the /firefox/new-b/ version is set to /products/download.html so that both the current and new version could use the same goal? If so, would I have to use the following regex for step 1?

Step 1: ^\/en-US\/firefox\/(new\/|new-b\/).*$

I tried to do a regex | for the destination URL, so I could have a single goal with different destination URLs, but it didn't seem to work. I was pretty sure this wasn't going to work, but I wanted to try it.

I'm leaning toward changing the virtual page view to be the same as the existing same as the current page, but I need some GA expertise to review our situation.

We could go back and forth over email, but I'm afraid this will take too long. If if it is a quick answer, email would be fine. If not, can we work through the issues and set up over a phone/video call?

Thanks in advance!
Update: in bug 821016, I've asked Jon to change the virtual page view to be the same URL as the destination URL of the existing /firefox/new/ page. I am still waiting to hear from Google.
Please add the following "new" code to the <head> of /firefox/new/:

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='65789850-0',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->

This code will only work on www.mozilla.org since the profile is different for both production and the demo servers. We can get this on production at any point because the experiment is turned off in Google and thus the trigger is GA and not the website.
Code is in branch and ready to be pushed.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
No longer depends on: 822443
New code for production:

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='65789850-8',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->
You need to log in before you can comment on or make changes to this bug.