1093761 - Conduct analysis of EOY form test

Status: RESOLVED FIXED

(Webmaker Graveyard :: Metrics, defect)

Description

[hannah:hkane]

Provide direction for which EOY campaign form to use

Comment 1

[Matt Thompson (@OpenMatt :OpenMatt)]

* Adam: once you're back, let us know if you're good to go on this.

Comment 2

[Andrea Wood [:andreawood]]

I was doing some analysis on donations so far and discovered that for some reason the new form isn't recording PayPal donation #s. Here's a screenshot: http://note.io/1EjPvLg

Maybe Aki can help sort out why?

Comment 3

[Aki Rose Braun [:akirose]]

No idea. If there was special tracking code to install, I wasn't aware and don't know how without Adam. (also the only way to record would be *theoretical* donations, i.e. that someone clicked paypal, not confirming they actually donated)

Comment 4

[Andrea Wood [:andreawood]]

Right. Once someone clicks paypal they're off our site and not trackable, except in PayPal. The BSD total isn't reflecting donations from PayPal coming through the new form at all, we've actually brought in $50,000 in revenue since Nov 1.

So we're stuck until Adam sets up optimizely / google analytics reports we can watch.

Comment 5

[Matt Thompson (@OpenMatt :OpenMatt)]

* Simon: is there anyone else that can fix? Or are we blocked on Adam's return?

Comment 6

[Adam Lofting (:adamlofting)]

(In reply to Andrea Wood from comment #4)
> Right. Once someone clicks paypal they're off our site and not trackable,
> except in PayPal. The BSD total isn't reflecting donations from PayPal
> coming through the new form at all, we've actually brought in $50,000 in
> revenue since Nov 1.
> 
> So we're stuck until Adam sets up optimizely / google analytics reports we
> can watch.

We'll have the same problem with GA/Optimizely.

The only way we're going to know if people donate (and how much( in GA or Optimizely is if the user is passed back to our site at the end of the PayPal transaction and then we log an event on the thank you page.

I just made a test PayPal donation and after a 5 second delay I am redirected to https://sendto.mozilla.org/page/s/mozilla-manifesto?tx=86Y054254L596410P&st=Completed&amt=0.50&cc=USD&cm=& ... (URL continues)

The URL parameters shown there does include the donation amount, so we have the data to record into GA if we script this on the thank you page. 

But, this looks like a generic (and maybe old) thank you page at the moment? Though I think you can set alternate redirect pages when you pass the user into PayPal. 

You can also pass 'sourcecodes' (I can't remember the Paypal term for them) along with the donation amount when you pass the user to PayPal. This would allow us to look at the donations in Paypal and segment them by sign-up form (if we passed the form name as the sourcecode).

Comment 7

[Adam Lofting (:adamlofting)]

(In reply to Aki Rose Braun [:akirose] from comment #3)
> No idea. If there was special tracking code to install, I wasn't aware and
> don't know how without Adam. (also the only way to record would be
> *theoretical* donations, i.e. that someone clicked paypal, not confirming
> they actually donated)

As above, the way to do this is by redirecting the user to a thank you page and Paypal passes along the donation amount for us to push into GA / Optimizely. 

Ideally the PayPal thank you page should be the same experience as the CC donation thank you page (with whatever our next step call to action is)

(Theoretically you might be able to to log this donation amount in BSD too, but I don't know their system)

Comment 8

[Adam Lofting (:adamlofting)]

Following up on this... I've looked at the data in BSD, and PayPal donations via the old form *are* recorded in BSD, but as Andrea noted these are not showing up for the new form. I'll file this as a new bug to action as we'll want to capture this data.

RE: Analysis, I've started this spreadsheet here:
https://docs.google.com/a/mozillafoundation.org/spreadsheet/ccc?key=0AnUkMzboH_XAdFQ3N0xqbVVXbmdpd2JRZGJ0X2JZUUE#gid=0

Currently the old form *appears* to perform the best, but we need to add in Paypal numbers for the new form to get a true picture. We can work out the Paypal numbers by taking the total, and subtracting the PayPal donations from the old form which are accounted for.

@Andrea, for the period 2014-11-01 to 2014-11-07 (inclusive), can you let the know:
1) Total number of PayPal donations (excluding recurring payments)
2) Total value of those PayPal donations

Comment 9

[Andrea Wood [:andreawood]]

Adam were you able to pull the data you needed from PayPal since I gave you login credentials? If not please let me know. I'm not clearing this needinfo until you tell me just in case.

Also, i want to make sure you saw that there was an error on my part with the control form. (An email that may have been buried in your inbox.) Initially I launched the test using the current live donation form:
https://sendto.mozilla.org/page/contribute/Give-Now/?&source=snippetEOY14_oldform1pg_control&utm_source=firefox&utm_medium=snippet&utm_campaign=snippetEOY14_oldform1pg_control&sample_rate=0.1&snippet_name=4628

However I caught my mistake, and at 8:40am PST Nov 2 Fabio updated the snippet to instead point to the correct control form we used in 2013 that was optimized for snippet: https://sendto.mozilla.org/page/contribute/bsd-test-give/?&source=snippetEOY14_oldform1pg_control2&amount=10&utm_source=firefox&utm_medium=snippet&utm_campaign=snippetEOY14_oldform1pg_control 

Note the default amount on the correct form is $10 vs. $25 on the incorrect form, which makes a significant impact on metrics.

Comment 10

[Adam Lofting (:adamlofting)]

(In reply to Andrea Wood from comment #9)
> Adam were you able to pull the data you needed from PayPal since I gave you
> login credentials? 

Yes. Painful but done :)

> Also, i want to make sure you saw that there was an error on my part with
> the control form. 

Yes, the analysis will only be against the correct form (you can see both pages noted in the analysis spreadsheet). Because the correct form went live later it has a bit less traffic, but it's still fine for % comparison (conversion rate and $ p/unique etc).

Comment 11

[Adam Lofting (:adamlofting)]

In this case we've been able to manually unpick the data and analyze the results of this test, but only because this was the only thing running and the new form happened to choose a new 'product name' in PayPal so I could identify the donations that way. Bug 1096293 will be essential for analysis when the level of activity ramps up.

==== RESULTS ====

All the numbers can be seen here:
https://docs.google.com/a/mozillafoundation.org/spreadsheet/ccc?key=0AnUkMzboH_XAdFQ3N0xqbVVXbmdpd2JRZGJ0X2JZUUE#gid=0

* New Form increases total conversion rate by 32% (from 5% to 6.7%)
* New Form increases total income by 6% (from $482 to $514 per 1000 unique visitors)
* New Form shifts the balance of PP/CC payment type from ~50% PP to ~70% PP
* New Form reduces CC conversion (-22%) and reduces CC income (-30%)
* New Form increases PP conversion (+85%) and increases PP income (+51%)

Hypothosis: Some donors who would have donated by card see the PP option, and switch to this instead (potential cause of CC decline) and some aditional more 'casual' donors are more likely to engage because the PP option reduces friction for them (the significant increase in PP).

I then looked at the distribution of donation amounts to get the story behind the changes in avg gift. See attached graph. The decrease in avg gift is just a statistical artifact of picking up greater numbers of donors at the lower end of the scale. 

* The New Form has relatively little effect on the more committed donors ($5-50) but has significant impact on the low end (perhaps more casual or more cautious) donors (<= $5)

Comment 12

[Adam Lofting (:adamlofting)]

Attachment: Distribution.jpg