1137020 - A/B test homepage changes; implement.

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: General, defect)

Description

[Stephanie Hobson [:shobson]]

Sean Martell has been looking at making some changes to the homepage in association with bug 1122793 and Justin asked if we could A/B test them. I think we should start the A/B testing before we make the changes to the content of blocks themselves so we can better compare to historical data.

There are two things to test 1) removing the heading 2) adding whitespace around the blocks. As pictured here: https://bugzilla.mozilla.org/attachment.cgi?id=8569324

Conditions for success:
> * search usage stays steady or improves
> * clicks on tiles stay the same or improve
> * bounce rate on this page does not increase

This is more up John's alley

 than mine but I am happy to do it too.

Comment 1

[John Karahalis [:openjck]]

Sounds awesome. Looks like this has everything I need to get started.

Is the creative final? Is there anything else we need from Sean before we get started?

Comment 2

[Stephanie Hobson [:shobson]]

Oops, I linked to the wrong screen shot >_< These are the changes Sean is proposing: http://cl.ly/image/1V3m213i0g3J I described it correctly though.

No copy changes, let's test the heading removal and white-space changes before changing the contents of the blocks.

Comment 3

[Stephanie Hobson [:shobson]]

Nothing else we need. Go fourth and A/B test!

Comment 4

[John Karahalis [:openjck]]

Attachment: A/B test variation

Just to verify before we start running the test, does this look good?

Comment 5

[John Karahalis [:openjck]]

Attachment: Mozilla_Developer_Network_-_2015-03-03_18.16.25.png

Comment 6

[John Karahalis [:openjck]]

I hate to be a nuisance, but how do we feel about foregoing bounce rate testing? Optimizely/GA integration has become pretty messy. I don't expect the number to change much, thankfully.

The good news is that Optimizely will have built-in bounce rate reporting before long. :-)

Comment 7

[Stephanie Hobson [:shobson]]

Test looks good!

I would be sad if we couldn't do bounce rate but I'm also happy to skip it since it sounds like you've spent more than a little time working on this already and it's more important to just do the test. Needsinfoing Justin to be sure.

Comment 8

[Justin Crawford [:hoosteeno] [:jcrawford]]

(In reply to John Karahalis [:openjck] from comment #6)
> I hate to be a nuisance, but how do we feel about foregoing bounce rate
> testing? Optimizely/GA integration has become pretty messy. I don't expect
> the number to change much, thankfully.

The context behind the request is this: We're talking about dropping the big h1 at the top of the front page. I think some users depend on h1s at the top of web pages to help them understand what the page is about. I think some users might leave if they don't have a big h1 at the top. 

My opinion is that the h1 we currently have does not do an amazing job of explaining what the site is about, and our audience probably doesn't need it. But I'm hoping to use something better than opinion to drive changes to the front page.

Can you think of another way to see if the header is an important part of keeping users around?

Comment 9

[John Karahalis [:openjck]]

That makes sense. Understanding bounce rate would be helpful when changing the heading. I wouldn't want to make a change like that based only on opinion, either.

Comment 0 talks about stylistic changes (spacing, color) to the callout boxes. Do we expect bounce rate to be impacted by those changes, too, or is bounce rate mostly needed for future tests, like the heading test?

Comment 10

[John Karahalis [:openjck]]

Another option to consider is measuring engagement, which Optimizely natively supports.

Engagement is basically the inverse of a bounce, in that engagement counts any click to the page. Navigation counts as engagement, and leaving the page right away doesn't.

Comment 11

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/05ee303c1ac765c3adbd43521750c1c8ba55716e
Bug 1137020: Track homepage search with Optimizely

https://github.com/mozilla/kuma/commit/ae6c3d233c3503994f0d30671108b6d0d756effd
Merge pull request #3095 from openjck/bug-1137020-optimizely-homepage-search-track-event

Bug 1137020: Track homepage search with Optimizely

Comment 12

[Justin Crawford [:hoosteeno] [:jcrawford]]

:openjck, thanks -- I think Engagement is a fine measure of this. We want Engagement to stay the same or increase after making changes to the page. If it decreases, we made the wrong change.

> Comment 0 talks about stylistic changes (spacing, color) to the callout
> boxes. Do we expect bounce rate to be impacted by those changes, too, or is
> bounce rate mostly needed for future tests, like the heading test?

I think engagement is really only concern with the heading change, but the changes may be easiest to implement all at once.

Comment 13

[Justin Crawford [:hoosteeno] [:jcrawford]]

Is there any update on this? The results I've seen suggest the changes are innocuous. In that case, success! We can launch them and put the new tiles in place.

Comment 14

[Stephanie Hobson [:shobson]]

I talked to John about this yesterday and he said, yes, the results are basically indistinguishable. He's planning on switching to testing the header text removal at the MDN hack weekend. 

Sounds like the new tiles have a little urgency now that we're getting more traffic to the home page. I'll add them to the MDN dev inbox and see if we can get some attention for them next week.

Comment 15

[John Karahalis [:openjck]]

Exactly. After running the experiment with more than 100,000 users, no statistically significant changes in behavior have been observed. (By comparison, most tests have significant results after 30,000 users.).

http://optimize.ly/~MNDadu?token=10f70f07b03bc3e3b440#view=2

If the change does affect behavior, the effect is very small. I can go ahead and make the changes permanent. \o/

Comment 16

[Justin Crawford [:hoosteeno] [:jcrawford]]

Great! Thanks for running the test.

Changing title to reflect the fact that this is actually a "test and implement" bug for designs suggested in bug 1122793 comment 14.

Comment 17

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/bd4b4ec057ac2a3f5bacb19cb4fdd62797d5ede6
Bug 1137020: Improve callout box design

https://github.com/mozilla/kuma/commit/86483ea4b29c260efd91a87e0e7e2f1d00260d95
Merge pull request #3132 from openjck/bug-1137020-homepage-callout-boxes

Bug 1137020: Improve callout box design

Comment 18

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/94ed071a9d508f40328a850f45cebc7db9f82580
Bug 1137020: Manual splitting for heading A/B test

https://github.com/mozilla/kuma/commit/b6ec8f7add846b605bcfe857a3e68cedc6316ce1
Merge pull request #3141 from openjck/bug-1137020-manual-heading-test-bucketing

Bug 1137020: Manual splitting for heading A/B test

Comment 19

[Justin Crawford [:hoosteeno] [:jcrawford]]

The changes look great -- thanks!

Comment 20

[John Karahalis [:openjck]]

There are no significant differences in user behavior when the headline is removed. If there is an effect, it's very small.

We should feel confident permanently removing the headline.

http://optimize.ly/~sQemB5?token=9f66d3146b5a8201bb65#view=2

Comment 21

[Justin Crawford [:hoosteeno] [:jcrawford]]

There are open questions about removing the headline.

1) Completely removing the headline has no effect on the "performance" of the page, where performance means visitors engaging with content.
2) The lack of a headline may have no effect on performance, but may impact "user comprehension, or user affinity, or something else along those lines". This was discussed at length in bug 1122793, where several were uneasy with this change. 

In the absence of any compelling data about its effect, a headline's value is subjective. We have people who are experts in designing websites and writing punchy headlines for them. If we are going to make subjective decisions, I'd like them to be expert decisions. 

Sean, Matej, do you think the argument about affinity and comprehension is valid?

Comment 22

[Matej Novak [:matej]]

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #21)
> There are open questions about removing the headline.
> 
> 1) Completely removing the headline has no effect on the "performance" of
> the page, where performance means visitors engaging with content.
> 2) The lack of a headline may have no effect on performance, but may impact
> "user comprehension, or user affinity, or something else along those lines".
> This was discussed at length in bug 1122793, where several were uneasy with
> this change. 
> 
> In the absence of any compelling data about its effect, a headline's value
> is subjective. We have people who are experts in designing websites and
> writing punchy headlines for them. If we are going to make subjective
> decisions, I'd like them to be expert decisions. 
> 
> Sean, Matej, do you think the argument about affinity and comprehension is
> valid?

I think Sean has a stronger design POV than I do a copy POV. Given the nature of this site, I don't think a headline is required, but if we can agree on one that helps a visitor to the page understand what it's about or what to do next, I'm in favour. Just not sure about next steps to get to that.

Comment 23

[Sean Martell]

So, to start, I'm pretty sure I made up that headline and it wasn't even in the wireframes for this page: http://cl.ly/image/1P3R0g1d3l1m

:)

I'm really just trying to give the searchbar a stronger presence and hierarchy on the page by removing the headline. That entire top section seems to be overly distracting with too many funnel options. 

If we do keep a headline, I'd prefer it not be staggered on two lines as it is in it's current form. That was me being fancy and it just adds unnecessary waste. 

If this can wait a few days, I'd like to explore a few solutions.

Comment 24

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/3c50317a2a0bb7ab3fce31401ffe5fe39c4df97b
Bug 1137020: Remove unused Optimizely bucketing

https://github.com/mozilla/kuma/commit/d8145242ab6a05dc02729c42874c9ed3ab243ddd
Merge pull request #3591 from openjck/bug-1137020-remove-optimizely-bucketing

Bug 1137020: Remove unused Optimizely bucketing

Comment 25

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/f12e27b7f5e51b486d2f24f155c206dab43903f1
Bug 1137020: Remove unused Waffle flag

https://github.com/mozilla/kuma/commit/65828bfe6381f6ae1345e0ca23f4d0d7e82109e3
Merge pull request #3645 from openjck/bug-1137020-remove-unused-experiment-code

Bug 1137020: Remove unused Waffle flag

Comment 26

[Stephanie Hobson [:shobson]]

This bug does not apply to the new homepage redesign.