972930 - Clicks (raw number) for tiles

Status: VERIFIED FIXED

(Tracking Graveyard :: Firefox Operations, defect)

Description

[Marco Mucci [:MarcoM]]

1.  Goal
* As a product owner I need to know the click rate for the directory tiles so that we can understand if and which tiles are useful to users and tile sponsors.

2.  Acceptance Criteria (AC)
* Click rate data for each directory tile is sent back to Mozilla through a channel such as FHR

3.  Notes/Supporting Documentation
* Initially we could use mz.la links for the static tile list but because of the manual work required we won’t be able to scale that system
** “Passing additional parameters after a bitly link is a feature that's available to [our] enterprise customers”
** https://groups.google.com/forum/#!topic/bitly-api/90dBzTzcTbk
* This data is necessary to understand if these tile suggestions are useful for our users and if certain versions work better than others
* This data is also required to understand pricing of sponsored tiles

Comment 1

[John Jensen]

I assume that "Click Rate" is:

Total number of clicks on an individual tile
--------------------------------------------
    Total number of New Tab page views

... and not ...

Total number of clicks on an individual tile
--------------------------------------------
Total number of views of an individual tile

The two denominators could conceivably be different (frecency effect, etc).

Comment 2

[Marco Mucci [:MarcoM]]

Converting to a work item following discussion with Bryan.

Comment 3

[Benjamin Smedberg]

Does click rate actually mean a ratio as in comment 1, or just a raw count of the number of times a tile was clicked?

Using a redirector (mz.la or something else) would give us this data in aggregate but not per user. It would be great if that's sufficient, but it's not clear whether we need to also correlate this count against other data.

Comment 4

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #3)
> Does click rate actually mean a ratio as in comment 1, or just a raw count
> of the number of times a tile was clicked?
> 
> Using a redirector (mz.la or something else) would give us this data in
> aggregate but not per user. It would be great if that's sufficient, but it's
> not clear whether we need to also correlate this count against other data.

Yes we need to correlate that against the other metrics to build the report ( bug 972929 ).  As pointed out in comment 1 (clicks per tile / views per tile) is the ratio we need.

Comment 5

[Chad Weiner [:chadw]]

(In reply to Bryan Clark (Firefox Search PM) [:clarkbw] from comment #4)

> Yes we need to correlate that against the other metrics to build the report
> ( bug 972929 ).  As pointed out in comment 1 (clicks per tile / views per
> tile) is the ratio we need.

Bryan, just to be clear, you're saying that that "click rate" is defined, using Jensen's ratio as:

Total number of clicks on an individual tile
--------------------------------------------
Total number of views of an individual tile

?

Comment 6

[Bryan Clark (DevTools PM) [:clarkbw]]

For this bug we're looking for the raw number of times during the measurement period that a tile has been clicked.  The ratio of clicks / views can be determined elsewhere; this is specifically to measure the number of clicks, not perform the calculation.

Comment 7

[Bryan Clark (DevTools PM) [:clarkbw]]

noting that bug 975570 returns a general interaction rate though it is not targeted for Directory Tiles.

Comment 8

[Ed Lee :Mardak]

Attachment: v1

Comment 9

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 8392475 [details] [diff] [review]
v1

I don't know about |this.link.type|, but the rest looks reasonable.
Would maybe a boolean histogram (whether the clicked tile was sponsored) be sufficient?

I'm not a peer and ttaubert is currently away, hg history suggests you could try gavin or makh.

Comment 10

[Ed Lee :Mardak]

(In reply to Georg Fritzsche [:gfritzsche] from comment #9)
> Would maybe a boolean histogram (whether the clicked tile was sponsored) be
> sufficient?
The number of sponsored tiles will change based on how many type=history tiles are visible, so I believe we want the enumerated type just like regular newtab clicks.

clarkbw, do we want a count of sponsored tiles clicked or a distribution of which positions sponsored tiles were clicked? If the former, we could simplify to the boolean histogram as gfritzsche suggests.


There are potentially other types of tiles that we want to have insights into such as type={history,organic,affiliate,sponsored} and that would seem to point to enumerated buckets again, but this then loses out on the position data. Just to check, if we were to do this, we would need to stably map those types into {0,1,2,3}?

Comment 11

[Georg Fritzsche [:gfritzsche]]

(In reply to Ed Lee :Mardak from comment #10)
> Just to check, if we were to do this, we would need to stably
> map those types into {0,1,2,3}?

Exactly.

Comment 12

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to Ed Lee :Mardak from comment #10)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #9)
> clarkbw, do we want a count of sponsored tiles clicked or a distribution of
> which positions sponsored tiles were clicked? If the former, we could
> simplify to the boolean histogram as gfritzsche suggests.

A distribution of Directory Tiles in general would be most useful, we want to know that the organic tiles etc have real user value and to be able to match that against any sponsors.  If we can't get the position data having all the types would be of more value.

Comment 13

[Georg Fritzsche [:gfritzsche]]

From what i understand sponsored tiles should come only after the tiles populated from user history?
So you probably want to measure which of the *sponsored* tiles was clicked, as the tile index itself wouldn't provide a stable mapping to the sponsor.

Will the sponsored tiles actually have a stable order so you can match them from "sponsored tile number" and time frame to a specific sponsor?

Comment 14

[Ed Lee :Mardak]

(In reply to Georg Fritzsche [:gfritzsche] from comment #13)
> So you probably want to measure which of the *sponsored* tiles was clicked,
> as the tile index itself wouldn't provide a stable mapping to the sponsor.
That is true, but we need to be careful about how we send that data via Telemetry as we cannot collect browsing history. Do you think it'll be okay to have an enumerated histogram of which directory tile numbers were clicked as opposed to a histogram of which positions a directory tile was clicked in?

E.g., if tiles were showing 4 history and 5 directory:

[h1] [h2] [h3]
[h4] [d1] [d2]
[d3] [d4] [d7]

The user clicking on h3 would record [NEWTAB_PAGE_SITE_CLICKED, 2].

A click on d1 would record [NEWTAB_PAGE_SITE_CLICKED, 4] and [NEWTAB_PAGE_SPONSORED_SITE_CLICKED, 0]

A click on d9 (assume user removed 5,6 or were for some other reason not shown): [NEWTAB_PAGE_SITE_CLICKED, 8] and [NEWTAB_PAGE_SPONSORED_SITE_CLICKED, 6]

Comment 15

[Ed Lee :Mardak]

Attachment: v2

Use the directory index instead of tile index for the enumerated histogram

Comment 16

[Georg Fritzsche [:gfritzsche]]

(In reply to Ed Lee :Mardak from comment #14)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #13)
> > So you probably want to measure which of the *sponsored* tiles was clicked,
> > as the tile index itself wouldn't provide a stable mapping to the sponsor.
> That is true, but we need to be careful about how we send that data via
> Telemetry as we cannot collect browsing history.

Right, this might need privacy review? Maybe ping curtisk on it?

> Do you think it'll be okay
> to have an enumerated histogram of which directory tile numbers were clicked
> as opposed to a histogram of which positions a directory tile was clicked in?
[...]

This sounds good to me and covers my understanding of the goal in comment 0.

Comment 17

[Ed Lee :Mardak]

curtisk, can we get a spot check privacy review for a specific telemetry probe?

The functionality that we want to get metrics on is a new set of tiles data shown in the newtab page. Firefox already probes which position tiles are clicked, e.g., does the user only click tile position #1 and #2. The new probe is similar except recording which index of the new set of tiles are clicked. See comment 14 for some examples.

We want to be able to answer how many times do users click on directory tile in index #1 (which is different from position #1 because directory tiles are shifted down in position as the user browses).

The privacy concern is that we don't want to collect browsing history -- we just want interaction click counts with the interface presented that happen to load a page. We don't care if the user successfully navigates/browses to the associated page -- just the fact that a tile at some index was clicked.

The tricky aspect is that we know what page is associated with which index. I think this is similar to a theoretical probe that we could add for users who launch outdated versions of Firefox where the probe would help report how many users are launching which outdated versions, but we would happen to also know that Firefox most likely tried to load a tab with "upgrade Firefox to the latest version" page which is now technically part of the browsing history.

Comment 18

[:Gavin Sharp [email: [e-mail]]]

I think you should probably file a Product Review::Privacy bug to track getting an answer to that question.

Comment 19

[Drew Willcoxon :adw]

Comment on attachment 8393309 [details] [diff] [review]
v2

Review of attachment 8393309 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/base/content/newtab/sites.js
@@ +190,5 @@
>                        .add(aIndex);
> +
> +    // Specially count clicks on directory tiles
> +    let {telemetryId} = this.link;
> +    if (telemetryId != null) {

if ("telemetryID" in this.link)

::: toolkit/modules/NewTabUtils.jsm
@@ +831,5 @@
>        // reached by a history tile, the last tile is pushed out
>        aCallback(rawLinks.map((link, position) => {
>          link.frecency = DIRECTORY_FREECENCY;
>          link.lastVisitDate = rawLinks.length - position;
> +        link.telemetryId = Math.min(position, 9);

Please capitalize it telemetryID (capital ID).

Comment 20

[Ed Lee :Mardak]

Attachment: for checkin

Comment 21

[Ed Lee :Mardak]

Attachment: for checkin

Comment 22

[Ed Lee :Mardak]

Attachment: v3

r? for changes to *not* track directory index but only counting overall how often each type of the 3 types were clicked.

Comment 23

[Drew Willcoxon :adw]

Comment on attachment 8396696 [details] [diff] [review]
v3

Review of attachment 8396696 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/base/content/newtab/sites.js
@@ +190,5 @@
>                        .add(aIndex);
> +
> +    // Specially count clicks on directory tiles
> +    let typeIndex = DirectoryLinksProvider.linkTypes.indexOf(this.link.type);
> +    if (typeIndex != -1) {

Nit: This would be a little nicer if linkTypes were a Set instead of an array.  Up to you.

Comment 24

[Ed Lee :Mardak]

(In reply to Drew Willcoxon :adw from comment #23)
> > +    let typeIndex = DirectoryLinksProvider.linkTypes.indexOf(this.link.type);
> > +    if (typeIndex != -1) {
> Nit: This would be a little nicer if linkTypes were a Set instead of an
> array.  Up to you.
We use typeIndex as the value to store in telemetry, so indexOf is a convenient way to get that. Unless you were suggesting the Set({
  "sponsored": 0,
  "affiliate": 1,
  "organic": 2,
}) ?

Comment 25

[Drew Willcoxon :adw]

You can tell I did a thorough job reviewing this. :-/

Comment 26

[Ed Lee :Mardak]

Attachment: for checkin

Comment 27

[Ed Lee :Mardak]

https://hg.mozilla.org/integration/fx-team/rev/8524260ce49a

Comment 28

[Phil Ringnalda (:philor)]

Backed out in https://hg.mozilla.org/integration/fx-team/rev/4caa44af6d1b, see bug 975228 comment 30.

Comment 29

[Ed Lee :Mardak]

https://hg.mozilla.org/integration/fx-team/rev/f72dc82b5ee0

For testing, the BBC and WIRED tiles are sponsored and result in "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "0" (sponsored)

Wikipedia and Mozilla Foundation result in "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "1" (affiliate)

The other directory links result in "2" (organic)

Comment 30

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/f72dc82b5ee0

Comment 31

[Paul Silaghi, QA [:pauly]]

(In reply to Ed Lee :Mardak from comment #29)
> https://hg.mozilla.org/integration/fx-team/rev/f72dc82b5ee0
> 
> For testing, the BBC and WIRED tiles are sponsored and result in
> "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "0" (sponsored)
> 
> Wikipedia and Mozilla Foundation result in
> "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "1" (affiliate)
> 
> The other directory links result in "2" (organic)
Verified fixed 31.0a1 (2014-04-02), Win 7, Ubuntu 12.10 and Mac OS X 10.9.
Filed bug 991111 for a related issue.