1201304 - Report the status of add-on installs to telemetry

Status: RESOLVED DUPLICATE of bug 1433335

(Toolkit :: Add-ons Manager, defect)

Description

[Dave Townsend [:mossop]]

As part of the signed add-ons work I'd like to understand how many users are attempting to install unsigned add-ons so I'm tinkering with adding a telemetry ping at the end of an install attempt which reports whether it succeeded or failed and what add-on was being installed.

Comment 1

[Dave Townsend [:mossop]]

Attachment: wip

This is neither finished, tested or has docs written but I wanted to see whether this seemed like a good idea. It would apply on top of bug 1177960.

Basically this sends a ping whenever an add-on install ends in some way, either success or failure. It includes information about what started the install, the add-on being installed and the url installing from. That latter seems privacy sensitive so I'm not sure it should be included.

Comment 2

[Dave Townsend [:mossop]]

Comment on attachment 8656279 [details] [diff] [review]
wip

Basically I want to be able to see how common various results of installation are, an example being how often do users attempt to install unsigned add-ons.

Comment 3

[Vladan Djeric (:vladan)]

Comment on attachment 8656279 [details] [diff] [review]
wip

Review of attachment 8656279 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
@@ +5417,5 @@
> +  sendTelemetryPing: function AI_sendTelemetryPing() {
> +    Cu.import("resource://gre/modules/TelemetryController.jsm");
> +    Cu.import("resource://gre/modules/TelemetryUtils.jsm");
> +
> +    TelemetryController.submitExternalPing("addon-install-ping", {

why create a new external ping? why not report this as a JSON field inside ping.addonDetails?

@@ +5424,5 @@
> +      updateReason: this.updateReason,
> +      state: this.state,
> +      error: this.error,
> +      existingAddon: !!this.existingAddon,
> +      sourceURI: this.sourceURI ? this.sourceURI.spec : undefined,

is this a privacy risk?

@@ +5429,5 @@
> +      addon: this.addon ? TelemetryUtils.getAddonData(this.addon) : undefined,
> +    }, {
> +      addClientId: true,
> +      addEnvironment: true,
> +    });

when would we stop collecting this data?

Comment 4

[Dave Townsend [:mossop]]

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #3)
> Comment on attachment 8656279 [details] [diff] [review]
> wip
> 
> Review of attachment 8656279 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
> @@ +5417,5 @@
> > +  sendTelemetryPing: function AI_sendTelemetryPing() {
> > +    Cu.import("resource://gre/modules/TelemetryController.jsm");
> > +    Cu.import("resource://gre/modules/TelemetryUtils.jsm");
> > +
> > +    TelemetryController.submitExternalPing("addon-install-ping", {
> 
> why create a new external ping? why not report this as a JSON field inside
> ping.addonDetails?

I don't know how I would go about doing that or what the difference is, can you give me some guidance?

> @@ +5424,5 @@
> > +      updateReason: this.updateReason,
> > +      state: this.state,
> > +      error: this.error,
> > +      existingAddon: !!this.existingAddon,
> > +      sourceURI: this.sourceURI ? this.sourceURI.spec : undefined,
> 
> is this a privacy risk?

As I said above, perhaps, I'm not sure. Would just reporting the hostname be less of a risk or should we avoid it altogether?

> @@ +5429,5 @@
> > +      addon: this.addon ? TelemetryUtils.getAddonData(this.addon) : undefined,
> > +    }, {
> > +      addClientId: true,
> > +      addEnvironment: true,
> > +    });
> 
> when would we stop collecting this data?

We probably wouldn't. This data can help us understand how healthy add-on installation is and how much impact various tools like the blocklist and signing have over time.

Comment 5

[Benjamin Smedberg]

Is there a clear question we're trying to answer? That will help determine whether this is the minimum set of data required to answer that question. This is particularly important for me in terms of collecting the source URI, which is not something that I'd prefer to avoid collecting unless there's a clear and compelling user benefit.

I encouraged Dave (and other people) to use separate pings for data such as this. Small targeted pings are better for event data, and this is clearly event data.

Who will be responsible for monitoring this data?
Who is going to develop and maintain the dashboards so that Kev can monitor this long-term?
Why do you want the clientID for this ping? Are you hoping to correlate this against other client data?

Comment 6

[Dave Townsend [:mossop]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> Is there a clear question we're trying to answer? That will help determine
> whether this is the minimum set of data required to answer that question.
> This is particularly important for me in terms of collecting the source URI,
> which is not something that I'd prefer to avoid collecting unless there's a
> clear and compelling user benefit.

The question I'm trying to answer right now is what proportion of add-on installs involve unsigned add-ons. Knowing where the unsigned add-ons are coming from might help us understand who we need to coax into getting their stuff signed, maybe we get that from the add-on ID though.

I'm also trying to include more data because I find it frustrating that questions keep coming up that we can't answer for our release population because we aren't collecting the data yet. If that's something we have to live with for our privacy policy then ok.

Some other questions:

How many installs involve multiple add-ons (either by InstallTrigger or by a multipackage add-on). The code and UI for these cases is sort of nasty and I'd like to just drop support for it but I don't know how often it happens.

We can track AMOs performance by seeing how many installs from AMO fail letting us spot if the signing service is broken or a CDN is down. We wouldn't beed the full source URI here, just an "is from AMO" flag.

How often do users install an add-on from the web that they already have installed? Should we add better UI for this case?

> Who will be responsible for monitoring this data?
> Who is going to develop and maintain the dashboards so that Kev can monitor
> this long-term?

For the initial question I was just going to write a spark dashboard to track it.

> Why do you want the clientID for this ping? Are you hoping to correlate this
> against other client data?

It could tell us if there are particular clients that are never able to install add-ons, we have had a single report of a user who sees all add-ons as broken, knowing if that is more prevalent but even with that data it probably wouldn't help us understand what the problem is so we could drop the clientID.

Comment 7

[Benjamin Smedberg]

We don't have a mechanism right now to expire custom pings. But I think this makes perfect sense for as long as you're actively working on this project and are signing up to monitor the data.

We should plan on removing or revising this data collection after you aren't looking at it in detail. If we need it long-term, we need to identify who is monitoring (Kev) and the minimal data to drive the dashboards he's monitoring.

+1 for changing the domain to an "is from AMO" flag.

Comment 8

[Vladan Djeric (:vladan)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> I encouraged Dave (and other people) to use separate pings for data such as
> this. Small targeted pings are better for event data, and this is clearly
> event data.

I agree external pings make sense for pure event data, but I'm still not convinced there's value to making addon-info an external ping: https://bugzilla.mozilla.org/show_bug.cgi?id=1177960#c11

Btw, how do we deal with "expiring" ad-hoc pings? The easiest way to do this seems to be with a shared Google Calendar

Comment 9

[Dave Townsend [:mossop]]

Kev and Andy are on point for figuring out any further details here. Happy to do patch work once we have nailed down what it is we want.

Comment 10

[Benjamin Smedberg]

I'd like to eventually have a manifest of ad-hoc pings with their status (opt-in, opt-out) and expiration, just like histograms. For now yeah we should keep this on a shared spreadsheet or calendar.

Comment 11

[Andrew Swan [:aswan]]

The original motivator (attempted installs of unsigned addons) is pretty much obsolete at this point, but the duplicate covers the main subject.