Run on-page optimization tests on welcome.webmaker.org

RESOLVED FIXED

Status

Webmaker
Metrics
P2
normal
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: adamlofting, Assigned: adamlofting)

Tracking

Details

(Whiteboard: [metrics][aug8])

Attachments

(3 attachments)

(Assignee)

Description

4 years ago
Now that we're using landing pages on webmaker.org, e.g. http://welcome.webmaker.org/

We should plan some content optimization tests to put to use our existing guidelines: https://wiki.mozilla.org/Foundation/Metrics/Optimizely_Process
(Assignee)

Updated

4 years ago
Priority: -- → P2
Adam / Andrea, lets include in this testing adding messaging around teaching.

Updated

4 years ago
Whiteboard: [metrics][July11] → [metrics][July11]teachcta
(Assignee)

Updated

4 years ago
See Also: → bug 1036273
(Assignee)

Comment 2

4 years ago
I've started up a doc with some text variations we can drop into the existing page.

https://etherpad.mozilla.org/wm-landing-page-text-variations

Please hack and edit, and I'll aim to setup our initial tests when I'm home from this workshop tonight.

Try not to filter *too* much at this stage, as the data will quickly let us know what does and doesn't work. Too many ideas is better than too little.
(Assignee)

Comment 3

4 years ago
Created attachment 8453348 [details]
https://welcome.webmaker.org/?optimizely_snippet=s3-206878104&optimizely_x1458031682=1&optimizely_token=707148613ce197dda1b28172aaa8985a&optimizely_show_preview=true

Quick R? check before I put any of this content into live testing rotation. 

In case the Optimizely preview link doesn't work, here's are the text variations I'd like to start with:


1) We believe anyone can be a tinkerer, creator, builder of the Web. Including you.
> The control

2) Teach and learn the web this summer with Mozilla's Maker Party. Together, we can build a better web for everyone.
> Front load the word Teach

3) Teach and learn the web this summer with Mozilla's Maker Party. 
> Test shorter version of the Teach variation

4) Welcome to Webmaker, Mozilla's open source education project, where you can teach and learn the web through making. Join us today.
> Adding in 'open source' and 'education' to the vocab.

5) Mozilla, the makers of Firefox, believe anyone can be a tinkerer, creator, builder of the Web. Including you.
> Test impact of linking to Firefox (may be useful for snippet traffic)

CTA Button text:

1) Sign up
> The control
2) Register
3) Tell me more
4) Yes, I'm in
5) Join the party
Attachment #8453348 - Flags: review?(brett)
(Assignee)

Comment 4

4 years ago
We're between snippet tests at the moment so traffic to this page is decent, but not at it's peak. I think tomorrow morning is a good time to switch this on if you're happy with the content.
(Assignee)

Comment 5

4 years ago
And just to note, we can add more variations into the mix as this is running.
* Adam: move to [July25]?
Adam I'd love it if you could add these, just so we're also seeing if we have any dramatic reactions to adding these teach variants. I say move to live, the content works for me.
(Assignee)

Comment 9

4 years ago
These initial text variations are now in the testing rotation.

And +1 to pushing this further with image testing, and so on.
(Assignee)

Comment 10

4 years ago
Renaming this, and moving to [July25] as it's an ongoing task
Summary: Webmaker Landing page test plan → Run on-page optimization tests on welcome.webmaker.org
Whiteboard: [metrics][July11]teachcta → [metrics][July25]teachcta
Whiteboard: [metrics][July25]teachcta → [metrics][July25]
(Assignee)

Comment 11

4 years ago
Created attachment 8458077 [details]
Screen Shot 2014-07-17 at 17.16.39.png

We have a statistically significant winner from the initial test variations.

This increases conversion rate by 17.6% (from 6.27% to 7.37%) and includes the Teach messaging we were keen to include up front.

Screenshot of the winner to follow.
(Assignee)

Comment 12

4 years ago
Created attachment 8458078 [details]
Screen Shot 2014-07-17 at 17.16.49.png

The winning intro text was:
> Welcome to Webmaker, Mozilla's open source education project, where you can teach and learn the web through making. Join us today.

And the winning button text was:
> Register
(Assignee)

Updated

4 years ago
Depends on: 1040156
(Assignee)

Comment 13

4 years ago
The next test, prompted by Bug 1027377 is to see the impact of adding a Maker Party logo to the page.
* Adam: I'd like to work with you to write a blog post on "words that work" -- synthesizing some of the data you've gathered here on which copy is performing most strongly. 

* moving to [aug8] as this work is on-going
Whiteboard: [metrics][July25] → [metrics][aug8]
(Assignee)

Updated

4 years ago
See Also: → bug 1044974
(Assignee)

Comment 15

4 years ago
(In reply to Matt Thompson (@OpenMatt :OpenMatt) from comment #14)
> * Adam: I'd like to work with you to write a blog post on "words that work"
> -- synthesizing some of the data you've gathered here on which copy is
> performing most strongly. 

+1 : Filed it here: Bug 1044974
(Assignee)

Comment 16

4 years ago
We are now testing variations of the Maker Party Logo in combination with (and in place of) the existing Webmaker logo on this landing page.

See screenshots of the variations here: 
https://docs.google.com/a/mozillafoundation.org/document/d/1Ui7MRsgu9lD_eiJV89ugm5RdlEpA0WyALM1Bf-C193o/edit
(Assignee)

Comment 17

4 years ago
After the logo test has significant results, we will then test adding imagery into the main banner area. 

Ideas so far include: 

* Cat on browser gif from webmaker (https://bugzilla.mozilla.org/show_bug.cgi?id=1027377#c52)
* Visual assets and patterns from https://party.webmaker.org/en-US/resources

Please add other content suggestions to this ticket.
(Assignee)

Comment 18

4 years ago
The results are in for the Maker Party logo test: https://docs.google.com/a/mozillafoundation.org/document/d/1Ui7MRsgu9lD_eiJV89ugm5RdlEpA0WyALM1Bf-C193o/edit#

TL;DR - adding the Maker Party logo reduces conversion rate, so we're sticking with the Webmaker logo.
(Assignee)

Comment 19

4 years ago
We've started a new test this morning which splits 50% of traffic into another landing page route here: https://welcome.webmaker.org/from/firefox-2014/

This asks the user about their interests and gives them a task to make a meme with Thimble, and depending on their response to the survey gives people the option to pledge to teach the web to a friend.

Once we get initial results of this A/B test we can look at which of these two possible routes we want to focus on optimizing. And if there are other user flows we want to test.
(Assignee)

Comment 20

4 years ago
We've just concluded another test, and here are the results:

https://docs.google.com/a/mozillafoundation.org/document/d/1aJmpreb6LyrAHReFO-8r3n481D5VrPKuwzAfcGUPcYQ/edit#heading=h.wo58ridhujzj

The survey based landing page is our new highest performing landing page, so further optimization of the content and the ongoing user flow through subsequent pages is being tested from there.
(Assignee)

Updated

4 years ago
Depends on: 1048328
(Assignee)

Comment 21

4 years ago
We've tested routing 5% of traffic into an immediate account sign-up flow which improved our end-to-end conversion. This email capture page is no longer in active use.

We're now shipping variations of the account sign-up flow for the next phase of testing:
http://bugzilla-sprinter.herokuapp.com/sprint/301
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Assignee)

Comment 22

4 years ago
@OpenMatt: Here's the write-up.
Flags: needinfo?(matt)
Flags: needinfo?(matt)
You need to log in before you can comment on or make changes to this bug.