568492 - Set up A/B testing infrastructure on mozilla.org

Status: RESOLVED FIXED

(www.mozilla.org :: General, defect)

Description

[David Boswell]

We can do some crude testing by manually updating things on the site manually, but it would be nice to have something in place to help us do real A/B testing.  Ned, suggested we look at Genetify.  We could also look at what other Mozilla sites, like mozilla.com, use for A/B testing.

http://wiki.github.com/gregdingle/genetify/

An example use case: randomly show one of multiple promos on a page to see which one has the higher click through rate.  We could specify each of the available promos in a meta tag at the top of the page:

<META NAME="WT.ad" CONTENT="test1, test2..."> 

and then each promo needs to be tagged so we can see which is being served.  You can do this in the URL like:

<a href="/foo/?WT.ac=test1">promo link</a>

Comment 1

[David Boswell]

A minor change to the note above:

I think the number of impressions might be counted by the amount of times the WT.ad value shows up on a page, so that might need to be dynamic instead of hard-coded with all available promos.  So that line in the page would be:

<META NAME="WT.ad" CONTENT="test1"> 

or 

<META NAME="WT.ad" CONTENT="test2"> 

Depending on which promo was being served and it would match the WT.ac value.

Comment 2

[David Boswell]

We're talking about setting up an A/B test of the intro blurb on the new home page.  To do that, we'd need to set up the following:

* In the header, we need a WT.ad tag that corresponds to the blurb being shown.  For instance, if 'We Believe in an Open Web.' is showing the header can include:

<meta name="WT.ad" CONTENT="blurb1">

If 'We're Building a Better Internet.' is showing the header can include:

<meta name="WT.ad" CONTENT="blurb2">

Note that I'm open to better tag names.  Also note that both tags can't just be hard-coded since WebTrends counts each appearance of that tag as an appearance of the item you are testing against.

* In the Learn more link next to the rotating blurb, we'd want another tag that corresponded with the tag in the header.  So:

<a href="/about/mission.html?WT.ac=blurb1">Learn&nbsp;more&nbsp;»</a>
<a href="/about/mission.html?WT.ac=blurb2">Learn&nbsp;more&nbsp;»</a>

* WebTrends then uses the first tag divided by clicks on the second tag to generate a click through rate number.  The trick is just having the tags and blurbs rotating to get the A/B test set up.

Comment 3

[Michael Gauthier [:gauthierm]]

David,

I don't think we can set the meta-tags in PHP due to the static page cache. Can the meta-tags get inserted in the document with JavaScript?

If so, does anything special need to be done with webtrends to get it to see the new elements?

Comment 4

[David Boswell]

Would the difference between setting things with PHP vs. Javascript be adding the meta tags to the head section or not?  I haven't experimented with adding those meta tags outside of the head, but WebTrends may be happy as long as they show up somewhere on the page.  We can try it out and see.

I don't believe WebTrends is picking up any of the stats from the staging site though, so we'd have to confirm this is working after things go live.  I'm fine with getting the rotating blurbs working and potentially revisitng the WebTrends issues after we go live if our initial approach doesn't work.

Comment 5

[Michael Gauthier [:gauthierm]]

The difference would be if done in PHP, the <meta> tag would be part of the HTML sent to the user. If done in JavaScript, the HTML would not have the <meta> tag initially, and it would be added to the DOM after the page loads.

I'll set it up to work using JavaScript for now.

Comment 6

[David Boswell]

OK, thanks for clarifying.  I could dig through WebTrends docs to see if this would work, but don't want to hold things up for confirmation.  Let's try this and then dig into the docs if it's not working.

Comment 7

[Michael Gauthier [:gauthierm]]

Committed in r76877.

Comment 8

[David Boswell]

I'm not seeing this rotation yet on stage.  Maybe the changes haven't propagated yet, but it's been about 50 minutes which seems like enough time.

Comment 9

[Michael Gauthier [:gauthierm]]

The code uses cookies to keep the text stable for a single user. Try deleting your cookies for the stage site and refreshing.

Comment 10

[David Boswell]

Re comment #9, what's the downside of just serving things randomly without worrying about someone potentially seeing both blurbs if they came back to the page or refreshed?

I haven't done a bunch of A/B testing, so I'm not sure what's standard practice.  I'd also prefer to only use cookies if we really need them (which we may here).

Comment 11

[David Boswell]

I was hoping to get a response to my question in comment #10 sometime today.  Without any information about the need for a cookie here, I'd prefer to remove it and do a simple randomization here.  Please let me know when that can be done.

Comment 12

[Michael Gauthier [:gauthierm]]

I think the test data will be better if the blurb is static for users. Otherwise, you don't know if they're clicking on learn more because the current text is better, because they noticed it change or because they remember the old text. Keeping it static reduces the number of test variables and will allow accurate results.

Comment 13

[Michael Gauthier [:gauthierm]]

I forgot to make the link tag dynamic as well. That's done in r76918.

Comment 14

[David Boswell]

OK, let's go with the cookie-based method and revisit as needed after it's live.

Comment 15

[David Boswell]

The test1 and test2 tags are showing up in WebTrends, but I'm not entirely sure it's working correctly.  There are a couple of things I want to check into:

* The CTR results are almost identical.  This may mean that the rotating blurb just has no influence over if people click on that link or not.

* The adopt_h tag isn't working correctly after the new design went up (the impressions are missing but the clicks are there).  Maybe this is unrelated, but my concern is that WebTrends is getting confused by something and is not tracking either adopt_h or the test1/test2 tags correctly.

I'm going to give WebTrends a call and see if they can answer any of these questions.

Comment 16

[Steven Garrity [:sgarrity]]

Can multiple WT.ad meta tags exist on the same page?

We've got:
<meta content="adopt_h" name="WT.ad">

but we're also adding:
<meta name="WT.ad" content="test1">
for the test page.

Perhaps they are conflicting?

Comment 17

[David Boswell]

You can have multiple WT.ad values, but they need to be in the same meta tag and separated by commas.  That explains why the impressions are missing from adopt_h.  Could we tweak the script to combine those meta tags?

Comment 18

[David Boswell]

Attachment: Patch to combine duplicated WT.ad meta tags

Comment 19

[Michael Gauthier [:gauthierm]]

David, r77147 contains a fix for the duped meta tags. I opted for leaving the tag in the header in case WebTrends can use it when JS is disabled, and si it will remain there if/when the A/B test gets removed.

Comment 20

[David Boswell]

OK, great.  Thanks for making that change.  I'll update the bug tomorrow after the stats get processed for today.

Comment 21

[David Boswell]

Re comment #19, it looks like that was just made on the kildare branch.  Should we move it over to production?

Comment 22

[Michael Gauthier [:gauthierm]]

Oops, I didn't know the kildare branch had gone live. Is it ok to apply my change to trunk so it will be automatically synced live, or is there a QA step that needs to happen first?

Comment 23

[Michael Gauthier [:gauthierm]]

Ok, Steven let me know I should test myself and commit to trunk. I checked in IE6/7/8, FF, Chrome, Safari and Opera with no issues, so I went ahead and committed the change in r77267.

Comment 24

[David Boswell]

If you've tested on your end then posting to trunk is fine.  I'll check the stats tomorrow and let you know what I see.

Comment 25

[David Boswell]

WebTrends didn't seem to like that change -- now impressions for test1 and test2 are not being counted along with adopt_h.  I'll call WebTrends and ask about this.

Comment 26

[David Boswell]

Just got off the phone with WebTrends.

The good news: It's an easy fix -- we should be using ; instead of , to separate the WT.ad values

The bad news: I gave you the wrong information in comment #17.

Comment 27

[David Boswell]

I checked in a fix for this in r 77369.

Comment 28

[David Boswell]

OK, I'm seeing impressions on test1, test2 and adopt_h now, so things are working as intended now.  Closing as fixed.