Closed Bug 1042126 Opened 10 years ago Closed 10 years ago

Change mozilla.org banner back to previous version

Categories

(Webmaker Graveyard :: webmaker.org, defect)

Product:
Webmaker Graveyard
Component:
webmaker.org
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: matt, Assigned: hannah)

References

Details

(Whiteboard: [mozilla][aug8])

Attachments

(1 file)

GitHub Pull Request
44 bytes, text/x-github-pull-request
Details | Review 
* The new "Maker Party" mozilla.org banner is vastly under-performing the previous banner

* Let's just roll it back to the old Webmaker banner and link we were using previously
* Hannah: can you follow up to get this one done?
Assignee: nobody → hannah
Flags: needinfo?(hannah)
Whiteboard: [mozilla][july25]
Flags: needinfo?(hannah)
Adding Jen Bertsch and Erika Drushka to help and advise.

Jen, please see Matt's comment above re: low performance of the new Maker Party banner on the homepage. We're wondering if we can roll back to the previous banner and target page (which was promoting Webmaker), since that one was generating more click-throughs and conversions.

And then, in parallel, perhaps we can use data from the snippet test we've been conducting to inform a next iteration. Our goal is to *increase* the click-through and conversion rate from the Webmaker banner, not just maintain that level.

Jen, I'm not sure of the process for changing the banner. Is it a minor change, or is it difficult?
Flags: needinfo?(jbertsch)
Swapping out an already-existing banner is pretty low effort.  We're the tiniest bit resource constrained this week by PTO, but we'll see what we can do.
Flags: needinfo?(jbertsch)
Can we provide screenshots of both banners + post-click UX here for comparison? I'd like to analyze and learn from this if possible.
(In reply to Andrea Wood from comment #4)
> Can we provide screenshots of both banners + post-click UX here for
> comparison? I'd like to analyze and learn from this if possible.

+1 

I've added screenshots and notes into this (open) google doc. If anyone has a screenshot of the original banner, could you please drop that in too:

https://docs.google.com/a/mozillafoundation.org/document/d/1EkbZeO2foweXdx46BKHHU69ZzN-JFoROhKSQOr8q3cU/edit#
I added screenshots of the previous banner.
I believe the banner in closed state read "learn the web" and not "teach the web" - I'm updating the Google doc now.
I have a couple of questions for Jen before we determine this is truly reliable data:

-- What dates were both banners live? Why I ask:

a. Was the higher-performing banner live during a period where there were higher click thrus across all banners on the homepage? Meaning, has the drop in engagement on that page happened across all the clickable elements, or is it isolated to the Webmaker banner only?

b. I wonder if our web properties see a decrease in clicks and engagement during the summer months. This often happens, and perhaps explains some of the decrease? 

I don't expect a or b to explain away the discrepancy, but it's worth double checking that we're comparing apples-to-apples.

A word of caution about applying findings from one channel to another: the snippet and the homepage banner are not necessarily apples-to-apples. The homepage is generally sought out by people already curious or looking for something Mozilla-related. The CTR on our homepage will *always* be higher than the snippet because of that (e.g. avg. CTR on the snippet is 0.1% vs. our worst performing banner, at 0.3% - 3x better). It is possible what works on the snippet would also work better on the banner, but it's not a sure thing. Data will tell.

Some thoughts on why the performance is poorer on the Maker Party version (not backed by data, but based on other testing I've done and lots of experience at this):
-- I've tested using the word "join" before and found that people associate that word with a higher "lift", as if by "joining" you're committing to more, or taking on a "membership" role. I avoid it in our donation asks for this reason. "chipping in" or or "let's explore together" feels lighter to people.

-- The Maker Party version might make people tired because it says "let's teach, learn, and make" -- which is a LOT to put in front of people. It's never good to ask people to do more than 1 thing. 1 request is a lower barrier for most people. That headline is 3 big things in HUGE font in the PAGE HEADLINE. Most people will ONLY want to do one thing... maybe "learn" - which means they're overwhelmed by the presumed commitment to 2 other things they don't care about, and abandon the page altogether.

-- I've said it again and again... but adding photos or a video to a page where the #1 goal is to get a person to click a button is a bad idea. It almost always performs poorly because it's hugely distracting from the ONE task you want them to do: click "host or attend" an event. The Maker Party page has a video, the higher performing page does not. I realize it is excrutiatingly tempting to add a photo - i mean it's "smiling people!!!" - but in UX it means you're giving people a massively attractive temptation, and they'll abandon the one thing you want them to actually do on that page, which is click a button.
Although, there is scope for lots of fluctuation on homepage click-through rate, the current end-to-end drop is so dramatic we're better to roll-back quickly, and then plan a more structured route moving forward.

(In reply to Andrea Wood from comment #8) 
> I have a couple of questions for Jen before we determine this is truly
> reliable data:
> 
> -- What dates were both banners live? Why I ask:

* The original webmaker banner on the homepage has been live for most of the year, and since we've had goal tracking in GA has converted visitors into webmaker accounts at 2-3% consistently for several months.

* The new Maker Party banner has only been live a few days, but is converting at 0.3%.

Even if click-through rate was unaffected, the end-to-end performance hit from the conversion rate drop is 0.18% vs 2.5%. 

Or, for every 100 new accounts we get via the new banner, we would have expected ~1,300 accounts via the original version.

> A word of caution about applying findings from one channel to another: the
> snippet and the homepage banner are not necessarily apples-to-apples.
+1

> Some thoughts on why the performance is poorer on the Maker Party version
+1 to those suggestions too

> -- I've said it again and again... but adding photos or a video to a page
> where the #1 goal is to get a person to click a button is a bad idea. It
> almost always performs poorly because it's hugely distracting from the ONE
> task you want them to do: click "host or attend" an event. The Maker Party
> page has a video, the higher performing page does not.

+1, homepages are typically overwhelming experiences.

This is where some of our new single-ask, welcoming-to-new-users, action-focused landing page designs can really help with the UX.

Also, I think one of the main reasons people don't create accounts on party.webmaker.org, is because we don't have any login functionality on that site. (Makes it a bit challenging!).

# Moving forward

Where possible (given time and resource), we should roll-out changes to high-traffic high value promotion in a structured way. We have the tools in place to do this now. Before we point a significant amount of traffic to a new page, we can test this against say 10% of the existing traffic and measure the impact. If it's positive, we make the switch. If not, we test and iterate.

This was an unusual case because we're changing the brand, the ask, and the destination all in one go. And it's hard to do that in stages. But right now, we'd be better to get thousands more people onboard via the webmaker ask, and then tell them about maker party, than to engage a fraction of the people with the direct maker party ask.
Hey-

One quick question to make sure I understand the conversation:

Are you comparing conversion rates from a homepage promo (one which was a webmaker promo and the other a maker party promo) to setting up a webmaker account on a landing page with a prominent webmaker CTA vs a landing page without a webmaker CTA?

Also, I'm happy to put the webmaker promo back up while we have this conversation, if that is the consensus decision.  Just let us know.

Thx,
Jen

PS I'll round up the data you've asked for.
The webmaker promo was live January 2, 2014 - July 15, 2014.

The maker party promo went live July 15.
It sounds like the consensus might be that we changed too many things at once (both the banner and the destination) and we'd like to roll back to the previous state (both banner and destination), so that we can continue to get the higher number of new Wm accounts while we attempt to diagnose the problem. 

Perhaps we ought to clarify our goals as well. Does our goal continue to be to get new Wm accounts? 

Question: do we actually have the tooling to test iterations with smaller sample sizes, or is the banner an all-or-nothing type of thing?
Thanks, Hannah.  We will roll-back.  I'll see if we can do it tomorrow.  (We usually don't push to production on Fridays, but this is small.)  Jon - Could you please get that ready?

I'm 99% certain that we could test iterations using Optimizely, but will let Craig and/or Holly confirm.
Flags: needinfo?(jon)
Flags: needinfo?(hhabstritt.bugzilla)
Flags: needinfo?(craigcook.bugz)
Whiteboard: [mozilla][july25] → [mozilla][aug8]
As long as not too much has been merged since our last push to production, we can likely get this change out today.

Should we keep the images/CSS for the Maker Party promo around? Or are we scrapping that promo for good?
Flags: needinfo?(jon)
(In reply to Jon Petto [:jpetto] from comment #14)
> As long as not too much has been merged since our last push to production,
> we can likely get this change out today.

The previous Webmaker promo is actually still there so this should just be a case of removing the new one, or even just commenting it out.

> Should we keep the images/CSS for the Maker Party promo around? Or are we
> scrapping that promo for good?

I think we should keep everything for now because it sounds like we might want to do some testing with the two promos.
Flags: needinfo?(craigcook.bugz)
Attached file GitHub Pull Request — 
All assets and markup for the Webmaker Party promo have been left in the repo for possible/probable future use.
It would also be useful to know what to do with the strings (we have 26 locales with that tile translated), especially if you plan to reintroduce it at some point.
(In reply to Craig Cook (:craigcook) from comment #15)
> (In reply to Jon Petto [:jpetto] from comment #14)
> > As long as not too much has been merged since our last push to production,
> > we can likely get this change out today.
> 
> The previous Webmaker promo is actually still there so this should just be a
> case of removing the new one, or even just commenting it out.
> 
> > Should we keep the images/CSS for the Maker Party promo around? Or are we
> > scrapping that promo for good?
> 
> I think we should keep everything for now because it sounds like we might
> want to do some testing with the two promos.

+1 we generally keep promos that aren't live around in case we want to re-use them in the future.

I'm not sure what we do with the localized strings - Craig?
(In reply to Jennifer Bertsch [:jbertsch] from comment #18)
 
> +1 we generally keep promos that aren't live around in case we want to
> re-use them in the future.
> 
> I'm not sure what we do with the localized strings - Craig?

We keep some fallback promos in place for out-of-date locales but most promos do get permanently deleted after their run. Resurrecting them later means we have to dig them out of git history.

Normally when a promo is permanently deleted we also delete its strings from the lang file, so resurrecting an old promo effectively adds "new" strings. Of course, the lang files are also kept in version control, so it's possible to dig back through history and find those previous translations as well, rather than starting from scratch.

For now we've only commented out the Maker Party promo and we'll keep its strings in the lang file. That way we can easily reactivate the promo if we want to run some optimizely tests and it will still be localized.
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/f78427b07801132a51c73206bf6f301cef4d2819
Revert Webmaker promo. Bug 1042126.

- New Webmaker promo markup/CSS/images left in repo for
  probable future use.

https://github.com/mozilla/bedrock/commit/4773088f3904b606b472f0104cc2285afb01aefd
Merge pull request #2178 from jpetto/bug-1042126-revert-webmaker-promo

Revert Webmaker promo. Bug 1042126.
This has been said already, but I just wanted to add my thoughts / support to what Adam and Andrea said above:

- "Learn the Web" is an easy thing that provides value to me. "Join our community" is a huge ask that also doesn't define what's in it for me. We had similar performance issues when our fundraising campaign was cloaked as 'join' rather than 'donate'.

- "Build, teach, make" provide choice. Like Andrea and others, I firmly believe choice doesn't work in funnels. You have to make an educated guess at what most people will want and put that there. "Throw an event" is also a simple, grokkable thing vs. "make, teach, or build the web".

I realize resources are constrained, but I would like to test the old funnel with the new (MP) look.
Also:

- The images on our donate page during EOY also converted *way* poorer than the boring text form. Sometimes boring = better.
(In reply to Geoffrey MacDougall from comment #21)

> I realize resources are constrained, but I would like to test the old funnel
> with the new (MP) look.

Hi Geoffrey-

To clarify, you want to test the "Learn the Web/We teach the Web/Let's explore...  Become a webmaker" text on the yellow maker party creative against the text and creative on the homepage now?

Would Mavis or Andrea or someone else be available to run the test if Holly teaches them how/where to set it up in Optimizely?  I'm guessing Andrea might already have Optimizely experience if I remember correctly?

Thx,
Jen
I'm happy to defer to whatever works. Also happy to have that taken on by Studio MoFo and Andrea.

But also want to flag that it's not a priority at the moment, as we are pointing those people at the snippet funnel.

Was more that I'd like us to continue to innovate on how/whether the Maker Party flavour - when added to the existing banner - can increase traction.

Hannah can be on point.

Thanks!
(In reply to Geoffrey MacDougall from comment #24)
> I'm happy to defer to whatever works. Also happy to have that taken on by
> Studio MoFo and Andrea.
> 
> But also want to flag that it's not a priority at the moment, as we are
> pointing those people at the snippet funnel.
> 
> Was more that I'd like us to continue to innovate on how/whether the Maker
> Party flavour - when added to the existing banner - can increase traction.
> 
> Hannah can be on point.
> 
> Thanks!

Great, Geoffrey, thanks for the prioritization update.  Please let us know when you all are ready to prioritize testing and we'll get you started.

Just a note for everyone on this bug, we are redesigning the mozilla.org homepage for October 14 so we'll have new promo spots to test going forward!
Flags: needinfo?(hhabstritt.bugzilla)
We discussed this at our Snippet working group meeting today. I believe we decided we ought to redirect    some % of this traffic to webmaker.org/signup to see how that fares. And we're planning to build a sandstone landing page to direct another % to (see https://bugzilla.mozilla.org/show_bug.cgi?id=1048315)

@Aki, @Adam, @Andrea (Triple A!) - have I got that right, or was that conversation specifically about the non-banner mozilla.org traffic (i.e. the Webmaker link on the Products page)?
Flags: needinfo?(andrea)
Flags: needinfo?(aki)
Flags: needinfo?(adam)
(In reply to hannah from comment #26)
> @Aki, @Adam, @Andrea (Triple A!) - have I got that right

That's right. :)

For others on this thread, we're iterating quickly on the user journey for people coming into webmaker during Maker Party. We have a number of possible landing pages and calls to action for this audience and we're swapping them in and out to get stats on which ones work, and which don't.
Flags: needinfo?(andrea)
Flags: needinfo?(aki)
Flags: needinfo?(adam)
See Also: → 1048315
Hi Adam-

Could you please set up time to meet with Holly Habstritt and Gareth Cull, and they can orient you to the homepage in Optimizely?

We are actually in the process of redesigning the homepage, so iterating on this version isn't a huge priority right now.

That said, we don't want to stand in the way of optimizing the webmaker/maker party CTA.

Do you have bandwidth to manage the tests yourself after meeting with Holly and Gareth?  There's the tiniest chance we'll need to use GA rather than Optimizely just because of how the tiles are built.

Thx,
Jen

PS  Holly is in CA and Gareth is in CA, so if it works better to just meet with Gareth, go for it.
* Marking this one resolved -- Adam, not sure if there's any follow-up needed here from you re: comment 28
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(adam)
Resolution: --- → FIXED
Thanks Matt, good call.

And thanks Jen for the info, and sorry for my delayed reply.

We will pick up this idea in the future as we look at the various sources of traffic coming to webmaker, but we can file new bugs when we get to that.
Flags: needinfo?(adam)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: