Closed Bug 945053 Opened 7 years ago Closed 7 years ago

Create GA test for exposed search field on Mozilla Homepage

Categories

(www.mozilla.org :: Pages & Content, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Habber, Assigned: jpetto)

References

()

Details

(Whiteboard: [kb=1202037] )

Attachments

(1 file)

For this test we will add a search field outside of Tabzilla on the Mozilla.org homepage. We decided to use the search field version with the red search icon.

Field without search icon -  http://cl.ly/image/0Y1E0r36430f
Field with red search icon -  http://cl.ly/image/2R0E0t45370i


Primary Goal: 
Testing search outside of Tabzilla on the Mozilla homepage will allow us to see what users are looking for and determine if we are providing the right resources both on the homepage and within Tabzilla.
 
Secondary Goals: 
Determine if users are using search more for frequently for content discovery than when it was hidden in Tabzilla - and if so, do they click through to a destination page via results?


Additional questions:
- Does this feature end up distracting users from engaging with other pieces of content?
- Since moving the download button position on the homepage, will more users be searching for terms like "firefox" or "firefox download" from search field?



Testing implementation notes:
- We are not currently tracking the search results page. Can't see CTR. 
- Need to investigate exact flow 
- en-us only for now
- We currently receive btwn 75k-100k views per day for homepage
- send 10% to test (6-8k per day) 
- Run test for 2 weeks
- Use GA Content Experiments 
- Craig Cook has latest .psd that includes search field style with red icon
- This new search field placement is applied to Homepage only
- Track this field separately from search field in Tabzilla. Add custom tracking for this new search field.
Whiteboard: [kb=1202037]
Assignee: nobody → jon
PSD attached. The search box options are in the "MENU" group.
As discussed in #www, the query string appended for the 10% of users viewing the homepage version with search field can be ?s=1. I'd rather keep it generic and simple than calling out the change explicitly.
Should we focus the search box on page load? Do we want to call that much attention to it?
Nope. No need to focus the search box. We want users to use it if we don't provide what they are looking for with the content on the homepage or other links in Tabzilla, footer, etc. 


(In reply to Jon Petto [:jpetto] from comment #3)
> Should we focus the search box on page load? Do we want to call that much
> attention to it?
jpetto: Below the GA Experiments code. Please paste this experiment code immediately after the opening head tag at the top of the original page: www.mozilla.org/en-US/

habber: I've allocated 25% of traffic to flow into the experiment and the test will run for 2 weeks and we can stop it any time. Let me know if you are ok with this. 

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='71153379-33',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->

Let me know when this is up on a demo server and i'll make sure the new search event 'outside tabz search' is working correctly. Thanks.

Gareth
Gareth: Thanks! 25% for 2 weeks sounds good to me. The test will end during holiday break. I don't see any issues with the test running while we are on break, as long as we have a couple of days this week to see how it's doing. 

Jpetto: Ping me as well once this is on demo and I'll take a look.
Hi Jon, I have a couple of small adjustments to suggest. 

Could you slightly reduce the height of the field so that it is a little slimmer? I realize this means the icon would also be smaller, but the field width is fine.  

Is there a highlight state that you could recommend that isn't blue? The blue looks a bit out of place. If the highlight is not there, but the cursor indicates the field focus, do you think this will be enough? 



(In reply to Jon Petto [:jpetto] from comment #7)
> The search field branch is up on demo1:
> 
> https://www-demo1.allizom.org/en-US/
> https://www-demo1.allizom.org/en-US/?s=1
Reduced the height by 4px and made the focus style much more subtle:

https://www-demo1.allizom.org/en-US/?s=1
Awesome Jon. I like the size and focus style. Thanks!

(In reply to Jon Petto [:jpetto] from comment #9)
> Reduced the height by 4px and made the focus style much more subtle:
> 
> https://www-demo1.allizom.org/en-US/?s=1
Great!

Gareth - Can you verify GA is working as expected?
Flags: needinfo?(garethcull.bugs)
Hey Jon,

It doesn't look like the search term is being captured in the Event Label. Can you please check this? 

Here's a screencap of what i'm seeing in GA upon searching: http://cl.ly/image/0i360g0Q0V2V

Thanks.

Gareth
Flags: needinfo?(garethcull.bugs)
Hey Gareth - 

Ack, had a typo in the JavaScript. Should be fixed now on demo1. Please test again when you have a minute.
Thanks for the fix Jon. The data looks like its coming in correctly now: http://cl.ly/image/2e0s26051S3T
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/5da16510aee92a8f062919092755e8b6202e1c21
Add home nav search box for GA experiment. Bug 945053.

https://github.com/mozilla/bedrock/commit/ef245915965ac642f1728483aaacea8c4a6e6325
Merge pull request #1536 from jpetto/bug-945053-home-search-field

Add home nav search box for GA experiment. Bug 945053.
Looks to be on production!
Search test is now live and 25% of traffic is now being sent into the test. RT data confirms that we are seeing traffic flow to the ?s=1 page.

Gareth
Closing this bug as the test has run its course.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.