1102610 - Analytics and testing for new tour pages

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, defect)

Description

[Chris More [:cmore]]

Scope:

* Web analytics events on all major in-page events (most already exists in current tours)
* Add event tracking to new search/opt-in CTAs
* Add key/value cohort pair to FHR
* Ensure that optimizely is able to vary both page and TourUI content.

Test plan coming.

Comment 1

[Chris More [:cmore]]

I've contacted optimizely and if we go over our max amount of unique users tested in 30 days, we will just get charged per 1,000 unique users. We may stay under the max and if not, we'll have a one-time bill to pay. I have verified that it won't shut off.

Comment 2

[Chris More [:cmore]]

Gareth and I ran an optimizely test on this page:

https://www.mozilla.org/en-US/firefox/33.1.1/firstrun/

We changed the "Take a quick tour" in the door hanger by using this optimizely code:

$("#firstrun > #strings").attr({"data-title":"Take a quick tour!"});

As long as the strings that we need to vary on are somewhere in the page, we should be able to create experiments that optimize whatever goal we need to test for. It doesn't matter if the page are data attributes in a div vs. content within the body. If the content is accessible via JavaScript, we can change it on the fly.

Comment 3

[Chris More [:cmore]]

Update: We are going to try to do this without using optimizely and build in the sampling and variations directly within the whatsnew page.

A few metrics notes:

* The first dice roll (opt in vs forced) will need to be changed often after launch from 0 to 100% probability to either of the paths.

* The second dice roll (even split between the 3 variations) will probably have to be modified at some point to not be even. Options: 33,33,33; 0,50,50; 0,0,100, etc. By default they will be even split among all 3 variations, at some point we may have to eliminate one or two variations thus evenly splitting the probability among the remaining variations.

* All of the same custom events used in onboarding tours should be used.

* On top of the existing GA events, we will need to add new events/custom variables to takes into account the dice rolls and the 3 variations per path.

* We will also need to set the paths (treatments) and actions taken by the user to the UITour to evaluate long-term impact on choices.

The GA and UITour tagging is being documented here:

https://docs.google.com/a/mozilla.com/spreadsheets/d/1F7-mz7cgSLTpD6uahMezU0Oe_qs9UxYtsf_iEXFTXxQ/edit#gid=0

Comment 4

[Chris More [:cmore]]

Gareth: Can you review the GA events and custom variables in the comment above?

bsmedberg/dzeber: Can you review the UITour tagging for the path and the action the user has taken?

Comment 5

[Chris More [:cmore]]

Alex: Can you review comment 3 and reply if you have any specific concerns now with these metrics?

Comment 6

[Alex Gibson [:agibson]]

Looks good to me Chris, 

- I don't think we'll need an "non-en-us" event since other locales won't hit this page.

- I also wonder if we should apply some more descriptive labeling for the events to make things easier to follow? There are three designs "Ravioli", "Flare", "Independence", so I'm guessing we could just use things like "whatsnew_forced_independence" instead of "whatsnew_forced_1C" (I'm assuming this is how things break down). Thoughts?

- Also, do the custom GA variables that do not accompany a GA event (non-usa, non-google) still need a dummy event to trigger them? I think Gareth has used this in the past.

Comment 7

[Benjamin Smedberg]

In the opt-in cases, it appears that we're not measuring anything until the user explicitly accepts or declines, is that correct?

I'm a little worried about that because if the user just closes the firstrun tab (without either accepting or declining) we wouldn't get any event at all. Could we add a "Shown" event or replace "Close" with "Shown":

three-way:

_gaq.push(['_trackEvent', 'whatsnew srch-chg page Interactions', 'Door Hanger', 'Shown']);
_gaq.push(['_trackEvent', 'whatsnew srch-chg page Interactions', 'Door Hanger', 'Switch']);
_gaq.push(['_trackEvent', 'whatsnew srch-chg page Interactions', 'Door Hanger', 'Close']);

or two-way:

_gaq.push(['_trackEvent', 'whatsnew srch-chg page Interactions', 'Door Hanger', 'Shown']);
_gaq.push(['_trackEvent', 'whatsnew srch-chg page Interactions', 'Door Hanger', 'Switch']);

Since the ratio we really care about optimizing is "switch/shown".

Comment 8

[Dave Zeber [:dzeber]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #7)
> Could we add a "Shown" event or replace "Close" with "Shown":
> 
I agree that we need to know the overall number of users who were shown the page, and I think recording the three events is most robust. What does the "Close" event mean for the opt-in pages - explicitly declined?

For the forced cases, will it be the case that the total pages shown is = "Try" + "Close"?

Comment 9

[Chris More [:cmore]]

I would have assumed we were going to add all of the show events and other in-page actions like we do in most tours when I created that doc, but I talked to Alex and he is starting from scratch. The doc above was mainly for people interacting with the door hanger assuming we had show events. We'll spec out a complete one now.

Comment 10

[Chris More [:cmore]]

We will already have all of the basic show data on the page because the Google Analytics tag is on all of www.mozilla.org by default. We just need to add some more events to capture it all. We're on it.

Comment 11

[Chris More [:cmore]]

bsmedberg dzeber: Gareth and I met and we made a bunch of changes to the UITour and GA event tagging. Check it out now and see if you have any concerns before we move forward.

Comment 12

[Benjamin Smedberg]

LGTM

Comment 13

[Dave Zeber [:dzeber]]

Yes, looks fine. 

Questions to double-check my understanding of the terminology: for each branch, should the count of ShowHanger equate to the # users who updated and saw the What's New page? Should we expect that # Try/Switch + # Close = # ShowHanger?

Comment 14

[Chris More [:cmore]]

(In reply to dzeber from comment #13)
> Yes, looks fine. 
> 
> Questions to double-check my understanding of the terminology: for each
> branch, should the count of ShowHanger equate to the # users who updated and
> saw the What's New page? Should we expect that # Try/Switch + # Close = #
> ShowHanger?

Good question. For un-modified Firefox who see these whatsnew pages, try/switch + close = showHanger. Though, if the user has removed their search bar, they won't be counted in the showHanger. This happened to me this morning and we don't want to assume every person successfully had the door appear just because they hit the page with the correct version. The numbers should line up in probably the major of use cases and not all, which is why we explicitly are setting an event when the door hanger appears.

Comment 15

[Chris More [:cmore]]

Alex: can you review line 2 & 3 for the firstrun page and line 23 for the whatsnew page?

https://docs.google.com/a/mozilla.com/spreadsheets/d/1F7-mz7cgSLTpD6uahMezU0Oe_qs9UxYtsf_iEXFTXxQ/edit#gid=0

All cool?

Comment 16

[Alex Gibson [:agibson]]

(In reply to Chris More [:cmore] from comment #15)
> Alex: can you review line 2 & 3 for the firstrun page and line 23 for the
> whatsnew page?
> 
> https://docs.google.com/a/mozilla.com/spreadsheets/d/1F7-
> mz7cgSLTpD6uahMezU0Oe_qs9UxYtsf_iEXFTXxQ/edit#gid=0
> 
> All cool?

I think this should be fine Chris, thanks

Comment 17

[Alex Gibson [:agibson]]

Analytics as per the doc in Comment 15 have been implemented and are now up on demo5 for testing.

For testing setup see instructions in bug 1102404.

Comment 18

[Gareth Cull [:garethc]]

Am testing now...will update the bug with results shortly.

Gareth

Comment 19

[Gareth Cull [:garethc]]

Here's a screencap of the test results showing all of the different whatsnew variation events being captured correctly in GA. Still need to test /firstrun/.

http://cl.ly/image/3B2P2j3F2s15

cmore...do you also want to verify? It looks good to me.

I have one other question, for users that have removed the search bar from their browser and are presented with the post tour page with no doorhanger, do we want to want to have an event trigger for them? 

Something like: _gaq.push(['_trackEvent', 'whatsnew srch-chg interactions','No Search Bar','ViewPage']);

Comment 20

[Chris More [:cmore]]

(In reply to Gareth Cull [:garethc] from comment #19)
> Here's a screencap of the test results showing all of the different whatsnew
> variation events being captured correctly in GA. Still need to test
> /firstrun/.
> 
> http://cl.ly/image/3B2P2j3F2s15
> 
> cmore...do you also want to verify? It looks good to me.
> 
> I have one other question, for users that have removed the search bar from
> their browser and are presented with the post tour page with no doorhanger,
> do we want to want to have an event trigger for them? 
> 
> Something like: _gaq.push(['_trackEvent', 'whatsnew srch-chg
> interactions','No Search Bar','ViewPage']);

Gareth:

Can you fill in column H that you have verified every line in this doc? Just put yes/no.

https://docs.google.com/a/mozilla.com/spreadsheets/d/1F7-mz7cgSLTpD6uahMezU0Oe_qs9UxYtsf_iEXFTXxQ/edit#gid=0

We could add an event for when the door hanger is not displayed.

Alex: For the conditional on when to display the door hanger or not, are you able to know exactly when the search box is not active to be able to send a GA event?

Comment 21

[Alex Gibson [:agibson]]

(In reply to Chris More [:cmore] from comment #20)
> We could add an event for when the door hanger is not displayed.
> 
> Alex: For the conditional on when to display the door hanger or not, are you
> able to know exactly when the search box is not active to be able to send a
> GA event?

Yes I can do this

Comment 22

[Gareth Cull [:garethc]]

cmore: i'm updating the doc now. Also, included a column for a screencap of the verified data coming into GA.

agibson: As per conversations in IRC, here's the event to be triggered when 'yahoo' is set as the default search provider.

_gaq.push(['_trackEvent', 'whatsnew srch-chg interactions','yahooDefault','ViewPage']);

Comment 23

[Chris More [:cmore]]

agibson: Check out last 4 lines of the tagging doc:

https://docs.google.com/a/mozilla.com/spreadsheets/d/1F7-mz7cgSLTpD6uahMezU0Oe_qs9UxYtsf_iEXFTXxQ/edit#gid=0

We've added a few exception event to handle edge cases.

Let us know when those are ready to verify.

Thanks!

Comment 24

[Gareth Cull [:garethc]]

Verified the following events are working:

YahooDefault and OtherDefault on /firstrun/: http://cl.ly/image/0P0y241e0g2g
Default Button Click: http://cl.ly/image/11033H2b241L

I've updated the spreadsheet. Just waiting on the remaining events to be implemented as per comment 23.

Gareth

Comment 25

[Alex Gibson [:agibson]]

(In reply to Chris More [:cmore] from comment #23)
> agibson: Check out last 4 lines of the tagging doc:
> 
> https://docs.google.com/a/mozilla.com/spreadsheets/d/1F7-
> mz7cgSLTpD6uahMezU0Oe_qs9UxYtsf_iEXFTXxQ/edit#gid=0
> 
> We've added a few exception event to handle edge cases.
> 
> Let us know when those are ready to verify.
> 
> Thanks!

Added the extra events and pushed to demo5 for verification, thanks!

Comment 26

[Chris More [:cmore]]

Gareth: After you get all of the line items verified, we need to create a spreadsheet of the key metrics that we will want to report on during the first few weeks of December. We need to get that rolling this week since your time off is right around the corner. If you don't do automated pulls, I will probably just export unsampled data on a regular basis and do pivot table conversion rates.

Comment 27

[Gareth Cull [:garethc]]

Updated the spreadsheet. Remaining Events verified here: http://cl.ly/image/3H1J170t3W2s

I'll start putting together a spreadsheet now based on demo-5 data then you should be able to adjust the filter to pull production data from GA.

Gareth

Comment 28

[Alex Gibson [:agibson]]

Links to dice roll code for reference:

opt-in / forced doorhanger: https://github.com/mozilla/bedrock/commit/9ca90fc28123c48e804d7eb40b613aef718da1e0#diff-1d93649c46a3e53687ec98af24c7deb2R177

page variation: https://github.com/mozilla/bedrock/commit/9ca90fc28123c48e804d7eb40b613aef718da1e0#diff-1d93649c46a3e53687ec98af24c7deb2R167

Comment 29

[Dave Zeber [:dzeber]]

Am I reading this right that users who already have Yahoo as default get entered in the experiment as "Forced"?

https://github.com/mozilla/bedrock/commit/9ca90fc28123c48e804d7eb40b613aef718da1e0#diff-1d93649c46a3e53687ec98af24c7deb2R316

I'm presuming that Yahoo would be default either because it was set by the user prior to updating or because it was reset on updating to 34 and then a paveover was done.

Comment 30

[Alex Gibson [:agibson]]

(In reply to dzeber from comment #29)
> Am I reading this right that users who already have Yahoo as default get
> entered in the experiment as "Forced"?
> 
> https://github.com/mozilla/bedrock/commit/
> 9ca90fc28123c48e804d7eb40b613aef718da1e0#diff-
> 1d93649c46a3e53687ec98af24c7deb2R316
> 
> I'm presuming that Yahoo would be default either because it was set by the
> user prior to updating or because it was reset on updating to 34 and then a
> paveover was done.

Yes we're currently always showing those who have Yahoo set as default the forced door hanger. Can you say for certain this is not what we should be doing? It's easy enough to update and show them the default view, but I need to know for certain what we should be doing here.

Comment 31

[Dave Zeber [:dzeber]]

We don't want to count them in the GA testing branches (1[ABC]). 

I'm not sure about which Whatsnew view they should get though. Should users who already have Y! as search default see the default Whatsnew page?

Comment 32

[Alex Gibson [:agibson]]

(In reply to dzeber from comment #31)
> We don't want to count them in the GA testing branches (1[ABC]). 
> 
> I'm not sure about which Whatsnew view they should get though. Should users
> who already have Y! as search default see the default Whatsnew page?

I can change it to still make sure they don't get counted in the GA testing branches (1[ABC]). Although Chris / Gareth might already be able to differentiate the data seeing as we are also adding a separate FA event for users with Yahoo already as default.

Need a definitive answer to this quick if we are going to push to prod today.

Comment 33

[Alex Gibson [:agibson]]

For now I've gone with the assumption that users who already have Yahoo as their default will see the default whatsnew page (Sync promo). They will not see the page variations or doorhanger.

for those users we will still track the following GA event:

['_trackEvent', 'whatsnew srch-chg interactions', 'All', 'yahooDefault']

If this needs to change, please let me know ASAP. In the mean time, I am going to proceed with a pull request so we can get this into code review.

Comment 34

[Benjamin Smedberg]

If they have y! as default, they are likely a paveover install. It's up to cpetit which message he would like to show them, but I think that the default sync-promo page is at least acceptable for first-ship.

Comment 35

[Alex Gibson [:agibson]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #34)
> If they have y! as default, they are likely a paveover install. It's up to
> cpetit which message he would like to show them, but I think that the
> default sync-promo page is at least acceptable for first-ship.

Thanks - this is an easy last minute change so let's go with this for Dec 1. If we would like to show pave-over users some other messaging we can always do this post-launch.

Comment 36

[Chris More [:cmore]]

rebecca: Check out this tagging document:

https://docs.google.com/a/mozilla.com/spreadsheets/d/1F7-mz7cgSLTpD6uahMezU0Oe_qs9UxYtsf_iEXFTXxQ/edit#gid=0

Column E & F are the treatment tags that are being set onload and on specific in-page interactions. The treatment tags are being saved in Firefox. The treatment tags are name and value pairs and you can see all of the names/values in those two columns.

1) open about:healthreport

2) click raw data

3) search for the string treatment

4) Write Y or No for column J in the spreadsheet if you are able to find the name/value in about:healthreport.

Not all name/values will be there as they are only set depending on which variation you see and what page you are looking at and what the dice roll was. The flow can be viewed here:

https://drive.google.com/file/d/0B5fr2uXG8zVqS3E1YjJoU3d2SkU/view

Comment 37

[Rebecca Billings [:rbillings]]

Verified all variations except lines 23-26. The only missing FHR value is on the whatsnew_Default page.

Comment 38

[Rebecca Billings [:rbillings]]

Lines 23-26 generate GA events, but leave no record in FHR. For example I can see the Declare button on the page, but cannot find any declareButton [or variations thereof] in the raw FHR data.

Comment 39

[Chris More [:cmore]]

(In reply to Rebecca Billings [:rbillings] from comment #38)
> Lines 23-26 generate GA events, but leave no record in FHR. For example I
> can see the Declare button on the page, but cannot find any declareButton
> [or variations thereof] in the raw FHR data.

For the declare button to show up in the data, you have to click the declare button. 

Gareth, agibson: can you help finish verifying comment 36?

Comment 40

[Alex Gibson [:agibson]]

Attachment: whatsnew_default.jpg

Lines 23-26 don't have any FHR events associated with them, so I think we're fine there.

I see `whatsnew_Default` in about:healthReport OK (see attached screenshot), which is being recorded if the user is not in US timezone, or has neither google or yahoo set as their default search engine choice.

It's worth noting that We're not recording any FHR data if 'yahoo' is already the default search engine (as per line 25 in the doc).

Comment 41

[Alex Gibson [:agibson]]

Gareth, do we want to record FHR data for users with yahoo set as their default (see Comment 40)? If so, please let me know and I can make a quick follow up PR.

Comment 42

[Gareth Cull [:garethc]]

Alex, let's record FHR data for users with Yahoo if it is easy to quickly add this in. Also, i'm seeing declare button events come through in GA: http://cl.ly/image/11033H2b241L

Let me know if there is anything else you need.

Comment 43

[Alex Gibson [:agibson]]

Talked to Gareth on Vidyo and we're going to set "whatsnew_Default" FHR data for those with Yahoo as their default, as well as those without the searchbar in their UI.

Opened a pull request here: https://github.com/mozilla/bedrock/pull/2551

Comment 44

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/b882e8d4aa385e8815bfa0ac16d84a12ac666e63
[fix bug 1102610] Update FHR analytics for whatsnew

https://github.com/mozilla/bedrock/commit/939bd19cfa45b20e79d7101e76cbb783f3d4829d
Merge pull request #2551 from alexgibson/bug-1102610-update-fhr-whatsnew-analytics

[fix bug 1102610] Update FHR analytics for whatsnew

Comment 45

[Chris More [:cmore]]

We are seeing good data in GA for all variations! All good here. Thanks everyone.