Closed Bug 979496 Opened 10 years ago Closed 10 years ago

[Australis Onboarding] GA for tour copy test for Beta

Categories

(www.mozilla.org :: Analytics, defect)

Production
x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: malexis, Assigned: garethc)

References

Details

(Whiteboard: [kb=1293093] )

From Bug#944098

We will move forward with 2 versions of the tour copy:
1.  feature driven (here are the new features and here is how you use them) vs
2.  benefit driven (here is why they are new and what they do for you)

With the goal of seeing which version has a higher tour completion rate, hypothesizing that a better educated user will generate more usage hours (despite our inability to fully test this).

We are aiming for final copy on March 5, to be ready for Beta on March 18, analyze results, and begin localization of final text April 1 for General Audience release on April 29.
Whiteboard: [kb=1293093]
Hi Alex-

Could you please give Gareth the URLs to test up the update test for Beta when they are ready?

Thx,
Jen
Flags: needinfo?(agibson)
Hey Alex,

As a follow-up to our conversation on vidyo, I noticed that when someone clicks on a bottom link within a step (version b), we fire an event for the link click, and then we trigger another event for when the next step loads. Unfortunately, this will make it difficult to determine whether someone is using link more vs. the arrows to navigate through the steps because the step event always fires. 

Instead, is it possible to track the link clicks and arrow clicks separately? 

For example, I just clicked 'Lets go' and am now on step 1. Can we set the link and arrow clicks to step 2 as:

link:  _gaq.push(['_trackEvent', 'Tour Interaction', 'link click to', 'Step 2']);
arrow: _gaq.push(['_trackEvent', 'Tour Interaction', 'arrow click to', 'Step 2']);  

This way, we'll only need to fire one event, and we can keep our aggregate reporting the same because the event labels stay consistent. Then i'll be able to segment the steps by the event action to see what's getting used more. Does that work? 


Also, i noticed that we don't have any event tracking on the landing page after the user completes the tour. 

Can we set the following events on click of:

Sync Button: link:  _gaq.push(['_trackEvent', 'whatsnew Page Interactions - New Firefox Tour', 'button click', 'Get Started with Sync']);

Learn more about the new Firefox links: _gaq.push(['_trackEvent', 'whatsnew Page Interactions - New Firefox Tour', 'link click', {url}]);

Any questions, please feel free to ping me. Thanks for all your help!

Gareth
Thanks, Gareth!

Makes sense, I'll update on the demo server and you can test further.
Great!

Here's the GA Experiments code for the demo3 test before we roll out to prod. Please paste this experiment code immediately after the opening head tag at the top of the original page only: https://www-demo3.allizom.org/en-US/firefox/29.0/whatsnew/  

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='66844022-36',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->

I've set up the conversion goal as the 'Step 4' event. Let me know when its live and i'll turn on the experiment on demo3. Then we can test it.

Thanks!
Thanks, Gareth!

I've updated demo3 with the new tracking labels in comment 3 and the GA experiment snippet in comment 5.

Also, the code for bug 981081 is in there and available for testing.
Looks like the demo experiment is populating with test data. I'm seeing conversion data flow into both variations based on our goal we set up.

Test data: http://cl.ly/image/3c3t3p3s0A0y

So, now I'll need to build another experiment for production. I'll post the updated snippet here shortly.
Alex,

Here's the updated experiments snippet for when we push to prod. Please paste this experiment code immediately after the opening head tag at the top of the original page only: http://www.mozilla.org/en-US/firefox/29.0/whatsnew/

Thanks.

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='71153379-35',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->
Hey Alex,

As a follow-up our conversation, the one thing I don't think we are capturing is the first time a user is seeing the tour. We need this so we can calculate the % that take the tour for the first time.

So, after the page loads, can you please fire the following event:

_gaq.push(['_trackEvent', 'Tour Interaction', 'First Time Seeing Tour', '[True OR False]',0,true]);

The 0 is an event value that i need to assign, and the last value of true sets this event as being non-interactive.

Please set the [TRUE or False] based on whether or not its their first time seeing the tour. Thanks.

Gareth
We tested GA events and the A/B test on demo3 prior to prod push and Gareth confirmed all data was coming through ok.

Marking this as fixed by bug 975099
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.