1061841 - Set up split & tracking for rAC on donation page

Status: NEW

(Websites :: donate.mozilla.org, defect)

Description

[Bobby Richter [:secretrobotron]]

Tasks:
- get urls for donation pages
- get estimate for monthly (and total) donations from mobile
- add ecommerce tracking tags to the thank you page to track conversion and donation amount
- propose timing for split and optimization

Comment 1

[Bobby Richter [:secretrobotron]]

Donation page to drive traffic at containing rAC: https://sendto.mozilla.org/page/contribute/Give-Mobile

Andrea: were there bug fixes in that page which will make it work differently than a page without rAC in it?

Comment 2

[Adam Lofting (:adamlofting)]

This might be throwing a spanner in the testing works:

We've had ~1,700 mobile donations, most of which occurred during end of year campaign. We currently get about 30 mobile donations p/month (including tablets). 

If you filter this down to: on mobile + using chrome + in US, the total is 143 including EOY. Of which about 40 are in 2014 to-date. (Let me know if I've missing something here, but I think that's the current audience for rAC.)

As an absolute minimum for generating significant results we'd need >200 transactions from a relevant audience across the two variants, and even then it would require a decent improvement in conversion rate for statistical significance. 

So...

This test would take a couple of years to maybe conclude... 

Unless someone has an audience we can promote this to before then.

Comment 3

[Bobby Richter [:secretrobotron]]

+ the fact that users would need to initialize their wallet setup.

There is something to be said for the number of users that bounce because they see a poor mobile experience and (maybe) shift to their desktop. Do you know if the number of mobile+chrome+us donations includes failures as well?

How should we move forward? Just point to the rAC-enabled page during EOY to gather data? Google folks were hoping to harvest some data before then to do a report. Maybe Maker Party campaign will yield some interesting results though.

Comment 4

[Andrea Wood [:andreawood]]

I'm not surprised at how low this is. I recall checking GA during EOY and noticing about 3% of our form traffic was not desktop. 

There are a couple of opportunities prior to EOY to test this form. 
-- Testing on the new FirefoxOS about:home (a mobile channel we have access to for the first time)
-- Pre-testing on the snippet itself (desktop, US-EN)
-- Email fundraising campaign at the end of Maker Party (people often read email on mobile, no idea what %, but it's an option)

Probably not a wildly giant # of donations prior to EOY, but we could probably get some data. I would recommend we do our focused testing on this form during EOY or late Nov (before the big traffic push) so we have the "winner" in place.

Comment 5

[Adam Lofting (:adamlofting)]

(In reply to Bobby Richter [:secretrobotron] from comment #3)
> How should we move forward? Just point to the rAC-enabled page during EOY to
> gather data? Google folks were hoping to harvest some data before then to do
> a report. Maybe Maker Party campaign will yield some interesting results
> though.

I think the best we can do here is switching this on for all traffic now and then looking for any long term shift in conversion rates for that audience segment. 

If we have any extra traffic before EOY we can look for possible results earlier. But EOY will be when we get most of the data.

(In reply to Andrea Wood from comment #4)
> There are a couple of opportunities prior to EOY to test this form. 
> -- Testing on the new FirefoxOS about:home (a mobile channel we have access
> to for the first time)
> -- Pre-testing on the snippet itself (desktop, US-EN)

These two will mean the users are using FF, so they won't get the rAC experience.

> -- Email fundraising campaign at the end of Maker Party (people often read
> email on mobile, no idea what %, but it's an option)

I think if we switch rAC on for all traffic, we'll pick up some extra data from these kind of activities.

Comment 6

[Andrea Wood [:andreawood]]

+1 agreed. The reality is that most people use Firefox to view our web properties, so traffic is relatively light since this is not a browser-agnostic form. EOY is the best timing.