1653020 - Collect UTM parameters when installing an add-on

Status: VERIFIED FIXED

(Toolkit :: Add-ons Manager, enhancement, P2)

Description

[William Durand [:willdurand]]

We'd like to use Telemetry data to propel the (new) download stats on AMO. We have a proof of concept using main pings but this isn't ideal because we don't have data for all add-on types (see also Bug 1645347) and we.

We think that event pings would work to retrieve the number of installs (downloads) per add-on, given that we send event pings when add-ons get installed. We also store the sourceURL in the add-on db (which was fixed in Bug 1648036), which is the URL of the page where the install process has been initiated. This could be somehow used for attribution.

The proposal would be to parse the query string of the sourceURL and extract 4 UTM parameters (utm_source, utm_medium, utm_content, and utm_campaign) that we can then send in the event ping once the install is successful. Of course, this requires a data review.

WDYT?

Comment 1

[Shane Caraveo (:mixedpuppy)]

We do collect those for firefox installs[1]. Do we have those params on all addon installs?

[1] https://searchfox.org/mozilla-central/rev/82c04b9cad5b98bdf682bd477f2b1e3071b004ad/toolkit/components/telemetry/app/TelemetryEnvironment.jsm#1695

Comment 2

[William Durand [:willdurand]]

Do we have those params on all addon installs?

No, they are optional. I also think that we should only collect those params when host is AMO because otherwise we could end up collecting UTM params from various websites (unlisted add-ons).

Comment 3

[William Durand [:willdurand]]

We decided to only rely on Telemetry events (addonsManager/install) when the step is completed and the source is either amo or disco in order to only have download stats for "listed" add-ons (disco is for recommended extensions, which should be listed). Therefore, the plan would be to only record the 4 UTM params mentioned above for those specific events (in extra payload probably).

Comment 4

[William Durand [:willdurand]]

Attachment: Data review request

Comment 5

[Chris H-C :chutten]

Comment on attachment 9165450 [details]
Data review request

Load-balancing to :phire.

Comment 6

[Jenny Zhang]

Comment on attachment 9165450 [details]
Data review request

DATA COLLECTION REVIEW RESPONSE:

> Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is part of Telemetry and will be documented in the events definition file: https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml

> Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

> If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, William Durand will be responsible.

> Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

> Is the data collection request for default-on or default-off?

Default on for all channels if Telemetry is enabled. 

> Does the instrumentation include the addition of any new identifiers?

No, UTM parameters are not linked to individual users.

> Is the data collection covered by the existing Firefox privacy notice?

Yes.

> Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 7

[William Durand [:willdurand]]

Attachment: Bug 1653020 - Collect UTM params when installing an add-on from AMO. r=rpl

Comment 8

[William Durand [:willdurand]]

Attachment: Bug 1653020 - Collect UTM params when installing an add-on from AMO. r=rpl

Comment 9

[William Durand [:willdurand]]

Hi :phire, I'd like to draw your attention to the implementation of this data collection: the existing "install" event has too many extra keys already, so we cannot add the 4 UTM parameters there, and the size limit of a single extra value prevents us from serializing the 4 UTM parameters in a single extra key.

Therefore, we introduced a new "install_stats" event, documented in Events.yaml, which collects the 4 UTM parameters as described in the original data request. Nothing has changed in this regard, we collect the 4 UTM parameters only when the installation of an add-on has completed and only when the "source" is "amo".

The event itself is only sent when the installation has completed and the source is "amo" or "disco". The existing "install" event is sent more often with more sources but we don't need events for other sources so we thought reducing the number of events being sent was a good thing.

Comment 10

[Jenny Zhang]

Thanks for following up here :willdurand, that sounds fine and doesn't have any impact on the data review. I appreciate the clarification.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bea7434c12af
Collect UTM params when installing an add-on from AMO. r=rpl

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/bea7434c12af

Comment 13

[William Durand [:willdurand]]

So, we'd like to uplift this patch to Beta (FF 80) because it would be very useful to collect the new event sooner rather than later. This would allow us to enable the new download stats feature on AMO sooner. The patch is covered by unit tests, it applies cleanly and you can see it on Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fac3e1f4982db3995e12a085a561ce8cee301da

Will we need the patch for Bug 1657513 too?

:rpl could you take care of the uplift request while I am away for parental leave? Thanks!

Comment 14

[William Durand [:willdurand]]

Comment on attachment 9166983 [details]
Bug 1653020 - Collect UTM params when installing an add-on from AMO. r=rpl

Beta/Release Uplift Approval Request

• User impact if declined: Without this patch, we wouldn't be able to collect add-on installation statistics for AMO. Unfortunately, the existing Telemetry install events cannot be used for AMO so that's why this patch adds a new event (install_stats).

End users wouldn't be impacted but add-on developers would be, because there wouldn't be any stats data available to them on AMO. Having this patch in Beta would allow us to collect more data sooner.

• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: To verify this patch, please install a web-extension from AMO. The add-on is expected to be successfully installed. The "Events" tab in about:telemetry should eventually contain an event for category = addonsManager and method = install_stats.
• List of other uplifts needed: Bug 1657513
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch isn't risky because it only adds a new Telemetry event that is collected when an add-on is fully installed (fairly small change to the AddonManager internals). In addition, the patch includes a comprehensive set of tests.

The patch for Bug 1657513 only sets a pref in the test file to make sure the data is collected for Thunderbird when running the test file.

• String changes made/needed:

Comment 15

[Julien Cristau [:jcristau]]

Comment on attachment 9166983 [details]
Bug 1653020 - Collect UTM params when installing an add-on from AMO. r=rpl

approved for 80.0b6

Comment 16

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/18aa6d1d7c02

Comment 17

[Alin Ilea, Desktop Test Engineering]

Reproduced the initial behavior using an old Nightly 80.0a1 (build id: 20200715215205). The Events tab does not contain the method = install_stats (it contains just the install method) after installing an add-on from AMO.
Verified - Fixed in latest Nightly 81.0a1 (build id: 20200809213940) and latest Beta 80.0b6 (build id: 20200807195315). The Events tab in about:telemetry contains the method = install_stats and category = addonsManager.