Closed Bug 1032224 Opened 5 years ago Closed 5 years ago

Run on-page optimization tests on welcome.webmaker.org

Categories

(Webmaker Graveyard :: Metrics, defect, P2)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: adam, Assigned: adam)

References

Details

(Whiteboard: [metrics][aug8])

Attachments

(3 files)

Now that we're using landing pages on webmaker.org, e.g. http://welcome.webmaker.org/

We should plan some content optimization tests to put to use our existing guidelines: https://wiki.mozilla.org/Foundation/Metrics/Optimizely_Process
Priority: -- → P2
Adam / Andrea, lets include in this testing adding messaging around teaching.
Whiteboard: [metrics][July11] → [metrics][July11]teachcta
See Also: → 1036273
I've started up a doc with some text variations we can drop into the existing page.

https://etherpad.mozilla.org/wm-landing-page-text-variations

Please hack and edit, and I'll aim to setup our initial tests when I'm home from this workshop tonight.

Try not to filter *too* much at this stage, as the data will quickly let us know what does and doesn't work. Too many ideas is better than too little.
Quick R? check before I put any of this content into live testing rotation. 

In case the Optimizely preview link doesn't work, here's are the text variations I'd like to start with:


1) We believe anyone can be a tinkerer, creator, builder of the Web. Including you.
> The control

2) Teach and learn the web this summer with Mozilla's Maker Party. Together, we can build a better web for everyone.
> Front load the word Teach

3) Teach and learn the web this summer with Mozilla's Maker Party. 
> Test shorter version of the Teach variation

4) Welcome to Webmaker, Mozilla's open source education project, where you can teach and learn the web through making. Join us today.
> Adding in 'open source' and 'education' to the vocab.

5) Mozilla, the makers of Firefox, believe anyone can be a tinkerer, creator, builder of the Web. Including you.
> Test impact of linking to Firefox (may be useful for snippet traffic)

CTA Button text:

1) Sign up
> The control
2) Register
3) Tell me more
4) Yes, I'm in
5) Join the party
Attachment #8453348 - Flags: review?(brett)
We're between snippet tests at the moment so traffic to this page is decent, but not at it's peak. I think tomorrow morning is a good time to switch this on if you're happy with the content.
And just to note, we can add more variations into the mix as this is running.
* Adam: move to [July25]?
Adam I'd love it if you could add these, just so we're also seeing if we have any dramatic reactions to adding these teach variants. I say move to live, the content works for me.
These initial text variations are now in the testing rotation.

And +1 to pushing this further with image testing, and so on.
Renaming this, and moving to [July25] as it's an ongoing task
Summary: Webmaker Landing page test plan → Run on-page optimization tests on welcome.webmaker.org
Whiteboard: [metrics][July11]teachcta → [metrics][July25]teachcta
Whiteboard: [metrics][July25]teachcta → [metrics][July25]
We have a statistically significant winner from the initial test variations.

This increases conversion rate by 17.6% (from 6.27% to 7.37%) and includes the Teach messaging we were keen to include up front.

Screenshot of the winner to follow.
The winning intro text was:
> Welcome to Webmaker, Mozilla's open source education project, where you can teach and learn the web through making. Join us today.

And the winning button text was:
> Register
Depends on: 1040156
The next test, prompted by Bug 1027377 is to see the impact of adding a Maker Party logo to the page.
* Adam: I'd like to work with you to write a blog post on "words that work" -- synthesizing some of the data you've gathered here on which copy is performing most strongly. 

* moving to [aug8] as this work is on-going
Whiteboard: [metrics][July25] → [metrics][aug8]
See Also: → 1044974
(In reply to Matt Thompson (@OpenMatt :OpenMatt) from comment #14)
> * Adam: I'd like to work with you to write a blog post on "words that work"
> -- synthesizing some of the data you've gathered here on which copy is
> performing most strongly. 

+1 : Filed it here: Bug 1044974
We are now testing variations of the Maker Party Logo in combination with (and in place of) the existing Webmaker logo on this landing page.

See screenshots of the variations here: 
https://docs.google.com/a/mozillafoundation.org/document/d/1Ui7MRsgu9lD_eiJV89ugm5RdlEpA0WyALM1Bf-C193o/edit
After the logo test has significant results, we will then test adding imagery into the main banner area. 

Ideas so far include: 

* Cat on browser gif from webmaker (https://bugzilla.mozilla.org/show_bug.cgi?id=1027377#c52)
* Visual assets and patterns from https://party.webmaker.org/en-US/resources

Please add other content suggestions to this ticket.
The results are in for the Maker Party logo test: https://docs.google.com/a/mozillafoundation.org/document/d/1Ui7MRsgu9lD_eiJV89ugm5RdlEpA0WyALM1Bf-C193o/edit#

TL;DR - adding the Maker Party logo reduces conversion rate, so we're sticking with the Webmaker logo.
We've started a new test this morning which splits 50% of traffic into another landing page route here: https://welcome.webmaker.org/from/firefox-2014/

This asks the user about their interests and gives them a task to make a meme with Thimble, and depending on their response to the survey gives people the option to pledge to teach the web to a friend.

Once we get initial results of this A/B test we can look at which of these two possible routes we want to focus on optimizing. And if there are other user flows we want to test.
We've just concluded another test, and here are the results:

https://docs.google.com/a/mozillafoundation.org/document/d/1aJmpreb6LyrAHReFO-8r3n481D5VrPKuwzAfcGUPcYQ/edit#heading=h.wo58ridhujzj

The survey based landing page is our new highest performing landing page, so further optimization of the content and the ongoing user flow through subsequent pages is being tested from there.
Depends on: 1048328
We've tested routing 5% of traffic into an immediate account sign-up flow which improved our end-to-end conversion. This email capture page is no longer in active use.

We're now shipping variations of the account sign-up flow for the next phase of testing:
http://bugzilla-sprinter.herokuapp.com/sprint/301
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
@OpenMatt: Here's the write-up.
Flags: needinfo?(matt)
Flags: needinfo?(matt)
You need to log in before you can comment on or make changes to this bug.