1316074 - get telemetry on started and completed extension/theme installs

Status: RESOLVED FIXED

(Toolkit :: Add-ons Manager, defect, P2)

Description

[Markus Jaritz [:designakt] (UX)]

We are working on (improving) the add-on install process.
To know if we succeed with our work, one relevant measure is how successful people navigate that flow.

Therefor we need data on 
- number of started installs
- number of successful installs
- number of user aborted installs
- time to success / abortion (with/without duration of download)
- duration of download
- number of permissions listed

With that we can establish a baseline, learn where we can improve, and see how each change to the install-flow affects intentional completion (both abort or install).

Comment 1

[:shell escalante]

work with dzeber, kev, and privacy on making a full spec for implementation

Comment 2

[Markus Jaritz [:designakt] (UX)]

Another relevant data point is what initiated the install: DiscoPane, AMO, ThirdParty, from File, Sideloading. (I would expect the other data points to vary a lot depending on that.)

Comment 3

[krupa raj [:krupa--use this to needinfo]]

With the permissions prompt, it is safe to assume that will be a drop in the # of installs (at least initially) given that we are adding one more click before the user can install an add-on they want.

What is the acceptable drop in # of installs after which we will all collectively freak out? :)

Comment 4

[Markus Jaritz [:designakt] (UX)]

(In reply to kraj from comment #3)
> With the permissions prompt, it is safe to assume that will be a drop in the
> # of installs (at least initially) given that we are adding one more click
> before the user can install an add-on they want.

For disco pane we are adding a step. For every other install the number of steps is the same. Users had to click install in a doorhanger prior to permissions as well.

> What is the acceptable drop in # of installs after which we will all
> collectively freak out? :)

We currently can not measure this in Firefox as this is not implemented. So I guess we will not freak out. ;-) In our analytics for disco pane we can however monitor for any impact, even if this is only a small percentage of all installs. And in addition, we are currently preparing for a user test of the new flow.

Comment 5

[Kev Needham [:kev]]

(In reply to Markus Jaritz [:designakt prev :maritz] (UX) from comment #4)
> We currently can not measure this in Firefox as this is not implemented. So
> I guess we will not freak out. ;-) In our analytics for disco pane we can
> however monitor for any impact, even if this is only a small percentage of
> all installs. And in addition, we are currently preparing for a user test of
> the new flow.

It would be valuable to understand how many times the permissions dialog is shown in addition to the number of times installation from the perms dialog is initiated. If they can be instrumented, it will help address questions like that in comment #3 and set baseline measures for what we should expect on conversion rates for addons that require permissions (and which permissions are presented).

Comment 6

[:shell escalante]

updated User Story with any metrics missing that were mentioned in comments and from OKR discussions.  

Kev, Markus, Jorge: can you sign off on the list under "user story" (and if there are any missing) and we'll put the request to engineering for adding?

Comment 7

[Jorge Villalobos [:jorgev] (he/him)]

Looks good to me.

Comment 8

[Markus Jaritz [:designakt] (UX)]

Thanks you. Looks good. I made some small changes.

And I did not understand what this refers to:
> Permissions:
> - # of times shown
> - # of times installation from the perms dialog is initiated

In the install flow one always has to click install on the permission dialog to continue, but install is initiated earlier, on Disco Pane or AMO etc.


One more interesting data-point would knowing how much each permission affects abortion? (Which one is the scariest?) Wasn't sure how to add that...

Comment 9

[emanuela [ux]]

I believe we need to like to collect data on the number of successful permissions update.

We can use this metric to understand if the notification is too vanilla or not.

Comment 10

[:shell escalante]

Adding need info for BDS and Rebecca for the URL of the "Request for new data and review" document: https://docs.google.com/document/d/1SCkIUwaQYQ_QQGIraMQG6qlL3XsE_goKpBawzt5z6Es/edit#

I didn't think I should attach a stagnant version over putting this in the URL.

From talking with Rebecca - I think there will be a discussion with folks about what data we collect and also a discussion on how. Let me know if I should set that up - and the scope.  If it's more on the what - then we'd include ben and dzeber probably (with kev)... if it includes the how, likely need mark reid too.

Comment 11

[Benjamin Smedberg]

This might be better in a meeting, but I'm going to write down a *preliminary* data review. The final data review happens on the patch which changes the in-tree documentation.

Most of this review is against the sample payload. That payload is not actually technically correct because pings have "payload" as a substructure, but that doesn't really matter for the early review.

    "date": 14567 OR 2017-01-01 //not sure which format is better here - UT uses numeric.
    "timestamp": 13198139587398 //client-side timestamp - useful for chaining events

I'm not sure why you need both of these, but I'm concerned about correlation risk here. When there is a precise timestamp like this, it means that you can correlate this with other data (such as AMO logs/GA) to identify a particular person who installed addon X at this same time. I would like you to consider other ways of accomplishing the same general goal. e.g. can you use a counter to make sure events are ordered without doing a timestamp? In general we ask people to keep timestamps as granular as possible, and prefer local-day (YYYY-MM-DD) as much as possible.

    "action": "install", //Install, enable, disable, update, remove
    "reason": "manual", //update, blocklist, manual, etc.
    "origin": "AMO", //AMO, Disco Pane, Search AMO, Search add-ons, 3rd party. We don't actually want the URLs.

What will the "origin" be for a system addon? Will this distinguish between system addons that ship with Firefox and which are delivered separately via balrog?

    "fx_source": "File", //File, foreign_install, Third party (general), Search about add-ons
    "result": "success", //did the action succeed? success/user_abort/error

For all of these enumerated listing, i will be necessary to list all possible values for these precisely. It might be more efficient and preferable to store these as numeric enumerations rather than strings, but that is mainly an implementation detail and not a privacy/data issue.

    "events": //array of actions in order of occurrence [timestamp, action]
    [
        [1000, "installDiag"],
        [2000, "permissions"],
        [3000, "download"]
    ]

This needs much more precise definition/documentation. Presumably timestamp 0 is some triggering event? And these are timestamps in millisecs?

Comment 12

[Dave Zeber [:dzeber]]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #11)
>     "date": 14567 OR 2017-01-01 //not sure which format is better here - UT
> uses numeric.
>     "timestamp": 13198139587398 //client-side timestamp - useful for
> chaining events
> 
> I'm not sure why you need both of these, but I'm concerned about correlation
> risk here. When there is a precise timestamp like this, it means that you
> can correlate this with other data (such as AMO logs/GA) to identify a
> particular person who installed addon X at this same time. I would like you
> to consider other ways of accomplishing the same general goal. e.g. can you
> use a counter to make sure events are ordered without doing a timestamp?

Good point - I think a counter would be fine for this. My main concern is that we're collecting event data, and I want to make sure we're able to recreate the sequence of events. Date can be yyyy-mm-dd.

> What will the "origin" be for a system addon? Will this distinguish between
> system addons that ship with Firefox and which are delivered separately via
> balrog?

We should distinguish.

> For all of these enumerated listing, i will be necessary to list all
> possible values for these precisely.

Agreed - I think we need a further iteration on the payload to pin down all the enumerated values.

Comment 13

[Julien Cristau [:jcristau] (back April 22)]

Mass wontfix for bugs affecting firefox 52.

Comment 14

[:shell escalante]

Hi Rebecca, Is this approved for us to break down and move forward with?

Comment 15

[:shell escalante]

approved with comment 12 changes, untriaging to get work assigned

Comment 16

[Markus Jaritz [:designakt] (UX)]

Hi Shell, will we be able to land this prior to 57 so that we can watch if 57 changes any behaviors.
Especially interesting during the transition with 57 would be if overall number of installs change and if the source of installs shift.

Comment 17

[Markus Jaritz [:designakt] (UX)]

We should also instrument the update-flow on added permissions, to see how much engagement we get with that current flow, and to know how many people are left on older versions of an extension. (WebExtensions can expand the permissions they require with an update. To get such an update, the user has to confirm the additional permission first.)
(I just received such notification as Ghostry added a permission to their extensions.)

Comment 18

[Andy McKay]

I'm going to break this down into a series of smaller bugs so we can prioritize these in the right components.

Comment 19

[:shell escalante]

Hi Andy,

Is there a plan to break this bug down into smaller work items?  The Data approval happened as one big chunk https://docs.google.com/document/d/1SCkIUwaQYQ_QQGIraMQG6qlL3XsE_goKpBawzt5z6Es/edit# - but implementation details is likely a few bits.

Comment 21

[Andy McKay]

I've file bugs and written a doc for this and attached as blockers to this bug.

Comment 22

[David Durst [:ddurst]]

I'm assuming that we have this covered by the telemetry implementation in the dependencies? If so, then we should close this.

Comment 23

[Ben Miroglio [:bmiroglio]]

Yep, ok to close!