945053 - Create GA test for exposed search field on Mozilla Homepage

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Holly Habstritt Gaal [:Habber]]

For this test we will add a search field outside of Tabzilla on the Mozilla.org homepage. We decided to use the search field version with the red search icon.

Field without search icon -  http://cl.ly/image/0Y1E0r36430f
Field with red search icon -  http://cl.ly/image/2R0E0t45370i


Primary Goal: 
Testing search outside of Tabzilla on the Mozilla homepage will allow us to see what users are looking for and determine if we are providing the right resources both on the homepage and within Tabzilla.
 
Secondary Goals: 
Determine if users are using search more for frequently for content discovery than when it was hidden in Tabzilla - and if so, do they click through to a destination page via results?


Additional questions:
- Does this feature end up distracting users from engaging with other pieces of content?
- Since moving the download button position on the homepage, will more users be searching for terms like "firefox" or "firefox download" from search field?



Testing implementation notes:
- We are not currently tracking the search results page. Can't see CTR. 
- Need to investigate exact flow 
- en-us only for now
- We currently receive btwn 75k-100k views per day for homepage
- send 10% to test (6-8k per day) 
- Run test for 2 weeks
- Use GA Content Experiments 
- Craig Cook has latest .psd that includes search field style with red icon
- This new search field placement is applied to Homepage only
- Track this field separately from search field in Tabzilla. Add custom tracking for this new search field.

Comment 1

[Craig Cook (:craigcook)]

Attachment: Mozilla_Homepage_search.psd.zip

PSD attached. The search box options are in the "MENU" group.

Comment 2

[Holly Habstritt Gaal [:Habber]]

As discussed in #www, the query string appended for the 10% of users viewing the homepage version with search field can be ?s=1. I'd rather keep it generic and simple than calling out the change explicitly.

Comment 3

[Jonathan Petto [:jpetto]]

Should we focus the search box on page load? Do we want to call that much attention to it?

Comment 4

[Holly Habstritt Gaal [:Habber]]

Nope. No need to focus the search box. We want users to use it if we don't provide what they are looking for with the content on the homepage or other links in Tabzilla, footer, etc. 


(In reply to Jon Petto [:jpetto] from comment #3)
> Should we focus the search box on page load? Do we want to call that much
> attention to it?

Comment 5

[Gareth Cull [:garethc]]

jpetto: Below the GA Experiments code. Please paste this experiment code immediately after the opening head tag at the top of the original page: www.mozilla.org/en-US/

habber: I've allocated 25% of traffic to flow into the experiment and the test will run for 2 weeks and we can stop it any time. Let me know if you are ok with this. 

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='71153379-33',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->

Let me know when this is up on a demo server and i'll make sure the new search event 'outside tabz search' is working correctly. Thanks.

Gareth

Comment 6

[Holly Habstritt Gaal [:Habber]]

Gareth: Thanks! 25% for 2 weeks sounds good to me. The test will end during holiday break. I don't see any issues with the test running while we are on break, as long as we have a couple of days this week to see how it's doing. 

Jpetto: Ping me as well once this is on demo and I'll take a look.

Comment 7

[Jonathan Petto [:jpetto]]

The search field branch is up on demo1:

https://www-demo1.allizom.org/en-US/
https://www-demo1.allizom.org/en-US/?s=1

Comment 8

[Holly Habstritt Gaal [:Habber]]

Hi Jon, I have a couple of small adjustments to suggest. 

Could you slightly reduce the height of the field so that it is a little slimmer? I realize this means the icon would also be smaller, but the field width is fine.  

Is there a highlight state that you could recommend that isn't blue? The blue looks a bit out of place. If the highlight is not there, but the cursor indicates the field focus, do you think this will be enough? 



(In reply to Jon Petto [:jpetto] from comment #7)
> The search field branch is up on demo1:
> 
> https://www-demo1.allizom.org/en-US/
> https://www-demo1.allizom.org/en-US/?s=1

Comment 9

[Jonathan Petto [:jpetto]]

Reduced the height by 4px and made the focus style much more subtle:

https://www-demo1.allizom.org/en-US/?s=1

Comment 10

[Holly Habstritt Gaal [:Habber]]

Awesome Jon. I like the size and focus style. Thanks!

(In reply to Jon Petto [:jpetto] from comment #9)
> Reduced the height by 4px and made the focus style much more subtle:
> 
> https://www-demo1.allizom.org/en-US/?s=1

Comment 11

[Jonathan Petto [:jpetto]]

Great!

Gareth - Can you verify GA is working as expected?

Comment 12

[Gareth Cull [:garethc]]

Hey Jon,

It doesn't look like the search term is being captured in the Event Label. Can you please check this? 

Here's a screencap of what i'm seeing in GA upon searching: http://cl.ly/image/0i360g0Q0V2V

Thanks.

Gareth

Comment 13

[Jonathan Petto [:jpetto]]

Hey Gareth - 

Ack, had a typo in the JavaScript. Should be fixed now on demo1. Please test again when you have a minute.

Comment 14

[Gareth Cull [:garethc]]

Thanks for the fix Jon. The data looks like its coming in correctly now: http://cl.ly/image/2e0s26051S3T

Comment 15

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/5da16510aee92a8f062919092755e8b6202e1c21
Add home nav search box for GA experiment. Bug 945053.

https://github.com/mozilla/bedrock/commit/ef245915965ac642f1728483aaacea8c4a6e6325
Merge pull request #1536 from jpetto/bug-945053-home-search-field

Add home nav search box for GA experiment. Bug 945053.

Comment 16

[Chris More [:cmore]]

Looks to be on production!

Comment 17

[Gareth Cull [:garethc]]

Search test is now live and 25% of traffic is now being sent into the test. RT data confirms that we are seeing traffic flow to the ?s=1 page.

Gareth

Comment 18

[Gareth Cull [:garethc]]

Closing this bug as the test has run its course.