1245230 - measure clicks on "update now" action button plug in check

Status: RESOLVED FIXED

(Plugin Check Graveyard :: UI, defect)

Description

[Jennifer Bertsch [:jbertsch]]

Hi Gareth-

We are going to run some tests on plug in check to see if we can increase email conversions on this page, however we also want to make sure we aren't negatively impacting "update now" actions by page visitors.

Could you please provide the event tracking we should use to measure that by EOD Friday, Feb 5?

Thx,
Jen

Comment 1

[Gareth Cull [:garethc]]

Hey Jen,

I just took a look at this and we have event tracking on the 'Update Now' Firefox Download button. 

We don't have any event tracking on any of the plugins which should be updated. 

On click of a plugin's update button, could we send:

window.dataLayer.push({
  'event': 'plugincheck-interactions',
  'interaction': 'button click',
  'plugin-action': buttonCopy,        // Eg. Up to Date, Research, Update, etc.
  'plugin-name': pluginName,          // Eg. Adobe Flash Player
});

One other thing that would be useful (but we can address later), would be to understand what plugins are being shown on this page (impressions), what states they are in and how that could potentially influence their next update (visit to whatsnew). I have some thoughts on how we could measure this, and will file a follow-up bug.

Comment 2

[Jennifer Bertsch [:jbertsch]]

Hi Schalk-

Could you please add these parameters to measure the plug in update clicks?

Thx,
Jen

Comment 3

[Schalk Neethling [:espressive]]

(In reply to Jennifer Bertsch [:jbertsch] from comment #2)
> Hi Schalk-
> 
> Could you please add these parameters to measure the plug in update clicks?
> 
> Thx,
> Jen

Yes, it was added in the initial pull request

Comment 4

[Jennifer Bertsch [:jbertsch]]

Gareth isn't seeing it recording in GA, could you please double check?  Thanks!

Comment 5

[Gareth Cull [:garethc]]

(In reply to Schalk Neethling [:espressive] from comment #3)

> Yes, it was added in the initial pull request

Will take a look.

Comment 6

[Gareth Cull [:garethc]]

I was thinking about this a bit more, and we also need to determine which sessions to the /plugincheck/ page actually have plugins that need updating. 

If we could capture plugin counts that would be great. We'll use sessions with plugins that need updating as the denominator when we calculate the conversion rate for users interacting with these buttons.

On load of the page, can we send:

window.dataLayer.push({
  'event': 'plugincheck-load',
  'interaction': 'page load',
  'total-plugins': totalNum,              // total number of plugins on page
  'plugin-up-to-date-count': num1,       // Number of plugins that are up to date
  'plugin-unknown-count': num2,          
  'plugin-not-up-to-date-count': num3 
});

Comment 7

[Schalk Neethling [:espressive]]

(In reply to Gareth Cull [:garethc] from comment #6)
> I was thinking about this a bit more, and we also need to determine which
> sessions to the /plugincheck/ page actually have plugins that need updating. 
> 
> If we could capture plugin counts that would be great. We'll use sessions
> with plugins that need updating as the denominator when we calculate the
> conversion rate for users interacting with these buttons.
> 
> On load of the page, can we send:
> 
> window.dataLayer.push({
>   'event': 'plugincheck-load',
>   'interaction': 'page load',
>   'total-plugins': totalNum,              // total number of plugins on page
>   'plugin-up-to-date-count': num1,       // Number of plugins that are up to
> date
>   'plugin-unknown-count': num2,          
>   'plugin-not-up-to-date-count': num3 
> });

Will add this. Are you seeing the current interactions come through?

Comment 8

[Schalk Neethling [:espressive]]

Attachment: Link to Github pull-request: https://github.com/mozilla/bedrock/pull/3855

Comment 9

[Schalk Neethling [:espressive]]

Hey Gareth,

This is what I am sending onload:

window.dataLayer.push({
    'event': 'plugincheck-load',
    'interaction': 'page load',
    'total-plugins': response.length,
    'plugin-vulnerable-count': pluginTotals.vulnerableCount,
    'plugin-outdated-count': pluginTotals.outdatedCount,
    'plugin-up-to-date-count': pluginTotals.upToDateCount,
    'plugin-unknown-count': pluginTotals.unknownCount,
});

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/31d22018a31da1a3aeb917f007207ac6548116e5
Fix Bug 1245230, add GA pings of plugin totals on load

https://github.com/mozilla/bedrock/commit/d70eadf38b667bfc109274e73de72c874d0e4046
Merge pull request #3855 from schalkneethling/bug1245230-add-onload-ga-for-plugincheck

Fix Bug 1245230, add GA pings of plugin totals on load

Comment 11

[Gareth Cull [:garethc]]

(In reply to Schalk Neethling [:espressive] from comment #9)
> Hey Gareth,
> 
> This is what I am sending onload:
> 
> window.dataLayer.push({
>     'event': 'plugincheck-load',
>     'interaction': 'page load',
>     'total-plugins': response.length,
>     'plugin-vulnerable-count': pluginTotals.vulnerableCount,
>     'plugin-outdated-count': pluginTotals.outdatedCount,
>     'plugin-up-to-date-count': pluginTotals.upToDateCount,
>     'plugin-unknown-count': pluginTotals.unknownCount,
> });

Thanks. This is great. 

Is this on a demo-server? Or in prod now? Same question for the dataLayer.push in Comment #1.

Comment 12

[Schalk Neethling [:espressive]]

The first set is definitely in production but, as we did not have a prod push yesterday, the on page load pings, are not. It is however on https://www-dev.allizom.org/en-US/plugincheck

Comment 13

[Jennifer Bertsch [:jbertsch]]

Hi Gareth and Schalk-

Is this one live on prod?

Thanks,
Jen

Comment 14

[Schalk Neethling [:espressive]]

(In reply to Jennifer Bertsch [:jbertsch] from comment #13)
> Hi Gareth and Schalk-
> 
> Is this one live on prod?
> 
> Thanks,
> Jen

Yes it is.

Comment 15

[Jennifer Bertsch [:jbertsch]]

Thanks!  Gareth - are you seeing what you need?

Comment 16

[Gareth Cull [:garethc]]

Schalk,

I'm seeing significant differences in users engaging with the 'vulnerable' button between our control (/plugincheck/) and the variations (/plugincheck/?v=(b|c|d). Engagement rate with these buttons in the control is nearly 3x higher than the variation.

Interactions with other plugin buttons (up to date, outdated, research) did not exhibit such a difference. 

Can you please check the data layer events associated with these 'vulnerable' button clicks? I'm wondering what could be causing this? Thoughts?

Comment 17

[Schalk Neethling [:espressive]]

(In reply to Gareth Cull [:garethc] from comment #16)
> Schalk,
> 
> I'm seeing significant differences in users engaging with the 'vulnerable'
> button between our control (/plugincheck/) and the variations
> (/plugincheck/?v=(b|c|d). Engagement rate with these buttons in the control
> is nearly 3x higher than the variation.
> 
> Interactions with other plugin buttons (up to date, outdated, research) did
> not exhibit such a difference. 
> 
> Can you please check the data layer events associated with these
> 'vulnerable' button clicks? I'm wondering what could be causing this?
> Thoughts?

Thanks for the feedback Gareth. It is odd indeed. The same code is used by all templates including the control. Is there a specific variation that exhibit this, or is it all variations? Thx

Comment 18

[Gareth Cull [:garethc]]

It seems to be associated with all variations, except the control.

Comment 19

[Schalk Neethling [:espressive]]

(In reply to Gareth Cull [:garethc] from comment #18)
> It seems to be associated with all variations, except the control.

Thanks Gareth. It's odd as they all use the same function. Is the interaction just lower?

Also, how do I test thing locally? It is one of the challenges here. I can prevent the button's default action and log out what will be sent to GA but, that is not ideal.