Closed Bug 972930 Opened 6 years ago Closed 6 years ago

Clicks (raw number) for tiles

Categories

(Tracking Graveyard :: Firefox Operations, defect)

x86_64
All
defect
Not set

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: MarcoM, Assigned: Mardak)

References

()

Details

(Whiteboard: [tiles] p=8 s=it-31c-30a-29b.2 [qa!])

Attachments

(1 file, 5 obsolete files)

1.  Goal
* As a product owner I need to know the click rate for the directory tiles so that we can understand if and which tiles are useful to users and tile sponsors.

2.  Acceptance Criteria (AC)
* Click rate data for each directory tile is sent back to Mozilla through a channel such as FHR

3.  Notes/Supporting Documentation
* Initially we could use mz.la links for the static tile list but because of the manual work required we won’t be able to scale that system
** “Passing additional parameters after a bitly link is a feature that's available to [our] enterprise customers”
** https://groups.google.com/forum/#!topic/bitly-api/90dBzTzcTbk
* This data is necessary to understand if these tile suggestions are useful for our users and if certain versions work better than others
* This data is also required to understand pricing of sponsored tiles
Blocks: 973273
Depends on: 973427
I assume that "Click Rate" is:

Total number of clicks on an individual tile
--------------------------------------------
    Total number of New Tab page views

... and not ...

Total number of clicks on an individual tile
--------------------------------------------
Total number of views of an individual tile

The two denominators could conceivably be different (frecency effect, etc).
Depends on: 974474
No longer depends on: 974474
No longer depends on: 973427
Converting to a work item following discussion with Bryan.
No longer blocks: 973273
Whiteboard: [story] [tiles] → p=0
Does click rate actually mean a ratio as in comment 1, or just a raw count of the number of times a tile was clicked?

Using a redirector (mz.la or something else) would give us this data in aggregate but not per user. It would be great if that's sufficient, but it's not clear whether we need to also correlate this count against other data.
Flags: needinfo?(clarkbw)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #3)
> Does click rate actually mean a ratio as in comment 1, or just a raw count
> of the number of times a tile was clicked?
> 
> Using a redirector (mz.la or something else) would give us this data in
> aggregate but not per user. It would be great if that's sufficient, but it's
> not clear whether we need to also correlate this count against other data.

Yes we need to correlate that against the other metrics to build the report ( bug 972929 ).  As pointed out in comment 1 (clicks per tile / views per tile) is the ratio we need.
Flags: needinfo?(clarkbw)
(In reply to Bryan Clark (Firefox Search PM) [:clarkbw] from comment #4)

> Yes we need to correlate that against the other metrics to build the report
> ( bug 972929 ).  As pointed out in comment 1 (clicks per tile / views per
> tile) is the ratio we need.

Bryan, just to be clear, you're saying that that "click rate" is defined, using Jensen's ratio as:

Total number of clicks on an individual tile
--------------------------------------------
Total number of views of an individual tile

?
Whiteboard: p=0 → [tiles] p=0
For this bug we're looking for the raw number of times during the measurement period that a tile has been clicked.  The ratio of clicks / views can be determined elsewhere; this is specifically to measure the number of clicks, not perform the calculation.
Summary: Click rate metrics for tiles → Clicks (raw number) for tiles
Status: NEW → ASSIGNED
Assignee: nobody → edilee
Whiteboard: [tiles] p=0 → [tiles] p=8 s=it-30c-29a-28b.3
Whiteboard: [tiles] p=8 s=it-30c-29a-28b.3 → [tiles] p=8 s=it-30c-29a-28b.3 [qa+]
noting that bug 975570 returns a general interaction rate though it is not targeted for Directory Tiles.
Attached patch v1 (obsolete) — Splinter Review
Attachment #8392475 - Flags: review?(georg.fritzsche)
Depends on: 975228
Blocks: tiles-dev
Comment on attachment 8392475 [details] [diff] [review]
v1

I don't know about |this.link.type|, but the rest looks reasonable.
Would maybe a boolean histogram (whether the clicked tile was sponsored) be sufficient?

I'm not a peer and ttaubert is currently away, hg history suggests you could try gavin or makh.
Attachment #8392475 - Flags: review?(georg.fritzsche) → feedback+
(In reply to Georg Fritzsche [:gfritzsche] from comment #9)
> Would maybe a boolean histogram (whether the clicked tile was sponsored) be
> sufficient?
The number of sponsored tiles will change based on how many type=history tiles are visible, so I believe we want the enumerated type just like regular newtab clicks.

clarkbw, do we want a count of sponsored tiles clicked or a distribution of which positions sponsored tiles were clicked? If the former, we could simplify to the boolean histogram as gfritzsche suggests.


There are potentially other types of tiles that we want to have insights into such as type={history,organic,affiliate,sponsored} and that would seem to point to enumerated buckets again, but this then loses out on the position data. Just to check, if we were to do this, we would need to stably map those types into {0,1,2,3}?
Flags: needinfo?(clarkbw)
(In reply to Ed Lee :Mardak from comment #10)
> Just to check, if we were to do this, we would need to stably
> map those types into {0,1,2,3}?

Exactly.
(In reply to Ed Lee :Mardak from comment #10)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #9)
> clarkbw, do we want a count of sponsored tiles clicked or a distribution of
> which positions sponsored tiles were clicked? If the former, we could
> simplify to the boolean histogram as gfritzsche suggests.

A distribution of Directory Tiles in general would be most useful, we want to know that the organic tiles etc have real user value and to be able to match that against any sponsors.  If we can't get the position data having all the types would be of more value.
Flags: needinfo?(clarkbw)
From what i understand sponsored tiles should come only after the tiles populated from user history?
So you probably want to measure which of the *sponsored* tiles was clicked, as the tile index itself wouldn't provide a stable mapping to the sponsor.

Will the sponsored tiles actually have a stable order so you can match them from "sponsored tile number" and time frame to a specific sponsor?
Whiteboard: [tiles] p=8 s=it-30c-29a-28b.3 [qa+] → [tiles] p=8 s=it-31c-30a-29b.1 [qa+]
(In reply to Georg Fritzsche [:gfritzsche] from comment #13)
> So you probably want to measure which of the *sponsored* tiles was clicked,
> as the tile index itself wouldn't provide a stable mapping to the sponsor.
That is true, but we need to be careful about how we send that data via Telemetry as we cannot collect browsing history. Do you think it'll be okay to have an enumerated histogram of which directory tile numbers were clicked as opposed to a histogram of which positions a directory tile was clicked in?

E.g., if tiles were showing 4 history and 5 directory:

[h1] [h2] [h3]
[h4] [d1] [d2]
[d3] [d4] [d7]

The user clicking on h3 would record [NEWTAB_PAGE_SITE_CLICKED, 2].

A click on d1 would record [NEWTAB_PAGE_SITE_CLICKED, 4] and [NEWTAB_PAGE_SPONSORED_SITE_CLICKED, 0]

A click on d9 (assume user removed 5,6 or were for some other reason not shown): [NEWTAB_PAGE_SITE_CLICKED, 8] and [NEWTAB_PAGE_SPONSORED_SITE_CLICKED, 6]
Blocks: 972936
Attached patch v2 (obsolete) — Splinter Review
Use the directory index instead of tile index for the enumerated histogram
Attachment #8392475 - Attachment is obsolete: true
Attachment #8393309 - Flags: review?(adw)
(In reply to Ed Lee :Mardak from comment #14)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #13)
> > So you probably want to measure which of the *sponsored* tiles was clicked,
> > as the tile index itself wouldn't provide a stable mapping to the sponsor.
> That is true, but we need to be careful about how we send that data via
> Telemetry as we cannot collect browsing history.

Right, this might need privacy review? Maybe ping curtisk on it?

> Do you think it'll be okay
> to have an enumerated histogram of which directory tile numbers were clicked
> as opposed to a histogram of which positions a directory tile was clicked in?
[...]

This sounds good to me and covers my understanding of the goal in comment 0.
curtisk, can we get a spot check privacy review for a specific telemetry probe?

The functionality that we want to get metrics on is a new set of tiles data shown in the newtab page. Firefox already probes which position tiles are clicked, e.g., does the user only click tile position #1 and #2. The new probe is similar except recording which index of the new set of tiles are clicked. See comment 14 for some examples.

We want to be able to answer how many times do users click on directory tile in index #1 (which is different from position #1 because directory tiles are shifted down in position as the user browses).

The privacy concern is that we don't want to collect browsing history -- we just want interaction click counts with the interface presented that happen to load a page. We don't care if the user successfully navigates/browses to the associated page -- just the fact that a tile at some index was clicked.

The tricky aspect is that we know what page is associated with which index. I think this is similar to a theoretical probe that we could add for users who launch outdated versions of Firefox where the probe would help report how many users are launching which outdated versions, but we would happen to also know that Firefox most likely tried to load a tab with "upgrade Firefox to the latest version" page which is now technically part of the browsing history.
Flags: needinfo?(curtisk)
I think you should probably file a Product Review::Privacy bug to track getting an answer to that question.
Flags: needinfo?(curtisk)
Comment on attachment 8393309 [details] [diff] [review]
v2

Review of attachment 8393309 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/base/content/newtab/sites.js
@@ +190,5 @@
>                        .add(aIndex);
> +
> +    // Specially count clicks on directory tiles
> +    let {telemetryId} = this.link;
> +    if (telemetryId != null) {

if ("telemetryID" in this.link)

::: toolkit/modules/NewTabUtils.jsm
@@ +831,5 @@
>        // reached by a history tile, the last tile is pushed out
>        aCallback(rawLinks.map((link, position) => {
>          link.frecency = DIRECTORY_FREECENCY;
>          link.lastVisitDate = rawLinks.length - position;
> +        link.telemetryId = Math.min(position, 9);

Please capitalize it telemetryID (capital ID).
Attachment #8393309 - Flags: review?(adw) → review+
Depends on: 986319
Attached patch for checkin (obsolete) — Splinter Review
Attachment #8393309 - Attachment is obsolete: true
QA Contact: paul.silaghi
Attached patch for checkin (obsolete) — Splinter Review
Attachment #8394610 - Attachment is obsolete: true
Attached patch v3 (obsolete) — Splinter Review
r? for changes to *not* track directory index but only counting overall how often each type of the 3 types were clicked.
Attachment #8394615 - Attachment is obsolete: true
Attachment #8396696 - Flags: review?(adw)
Comment on attachment 8396696 [details] [diff] [review]
v3

Review of attachment 8396696 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/base/content/newtab/sites.js
@@ +190,5 @@
>                        .add(aIndex);
> +
> +    // Specially count clicks on directory tiles
> +    let typeIndex = DirectoryLinksProvider.linkTypes.indexOf(this.link.type);
> +    if (typeIndex != -1) {

Nit: This would be a little nicer if linkTypes were a Set instead of an array.  Up to you.
Attachment #8396696 - Flags: review?(adw) → review+
(In reply to Drew Willcoxon :adw from comment #23)
> > +    let typeIndex = DirectoryLinksProvider.linkTypes.indexOf(this.link.type);
> > +    if (typeIndex != -1) {
> Nit: This would be a little nicer if linkTypes were a Set instead of an
> array.  Up to you.
We use typeIndex as the value to store in telemetry, so indexOf is a convenient way to get that. Unless you were suggesting the Set({
  "sponsored": 0,
  "affiliate": 1,
  "organic": 2,
}) ?
You can tell I did a thorough job reviewing this. :-/
Attached patch for checkinSplinter Review
Attachment #8396696 - Attachment is obsolete: true
https://hg.mozilla.org/integration/fx-team/rev/f72dc82b5ee0

For testing, the BBC and WIRED tiles are sponsored and result in "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "0" (sponsored)

Wikipedia and Mozilla Foundation result in "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "1" (affiliate)

The other directory links result in "2" (organic)
Whiteboard: [tiles] p=8 s=it-31c-30a-29b.1 [qa+] → [tiles] p=8 s=it-31c-30a-29b.2 [qa+]
https://hg.mozilla.org/mozilla-central/rev/f72dc82b5ee0
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Depends on: 991111
(In reply to Ed Lee :Mardak from comment #29)
> https://hg.mozilla.org/integration/fx-team/rev/f72dc82b5ee0
> 
> For testing, the BBC and WIRED tiles are sponsored and result in
> "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "0" (sponsored)
> 
> Wikipedia and Mozilla Foundation result in
> "NEWTAB_PAGE_DIRECTORY_TYPE_CLICKED" counting "1" (affiliate)
> 
> The other directory links result in "2" (organic)
Verified fixed 31.0a1 (2014-04-02), Win 7, Ubuntu 12.10 and Mac OS X 10.9.
Filed bug 991111 for a related issue.
Status: RESOLVED → VERIFIED
Whiteboard: [tiles] p=8 s=it-31c-30a-29b.2 [qa+] → [tiles] p=8 s=it-31c-30a-29b.2 [qa!]
No longer blocks: fxdesktopbacklog
Flags: firefox-backlog+
Blocks: 974474
Product: Tracking → Tracking Graveyard
You need to log in before you can comment on or make changes to this bug.