1699225 - Evaluate addition to addonsManager telemetry events for comparison between TAAR based vs. manual discovery list

Status: RESOLVED FIXED

(Toolkit :: Add-ons Manager, task, P2)

Description

[Luca Greco [:rpl] [:luca] [:lgreco]]

The goal of this issue is to evaluate the additions needed to the addonsManager telemetry events to allows us to better compare TAAR based vs. manually managed discovery list, tracked by the meta issue Bug 1697699 (following up the preliminary analysis from Bug 1697699 comment 1).

Comment 1

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 1699225 - Add taar_enabled/taar_based extra vars to addonsManager telemetry. r?robwu!,willdurand

TODO:

• [ ] confirm with Will the proposed approach for the addition to the install_stats telemetry events
• [ ] confirm with Rob the proposed approach for the addition to the view and action telemetry events
• [ ] attach a data review request on bugzilla to get the data team perspecitve on the addition (through the usual data review request form format)
• [ ] add test to cover the need collection with proper tests (the changes to the tests included in the initial patch do only prevent
  breaking existing tests assertions, I'm going to add some more specific test as soon as I got the confirmation that we are proceeding with
  the collection and an approach that we have reached an agreement on)

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: request_bug_1699225_taar_enabled_extra_var.md

Comment 3

[mlopatka]

Comment on attachment 9210055 [details]
request_bug_1699225_taar_enabled_extra_var.md

Some comments for the data review.

1. What questions will you answer with this data?

For the questions being answered, are we measuring browser engagement? or engagement with the addons ecosystem? In which case, is the surrogate for engagement WebExtension installation rate? Please be specific in terms of how we define engagement as an outcome measure here.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

Likewise, it is not obvious here what engagement is here and whether a causal relationship can be determined from this data collection.

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

It is helpful in the interest of transparency, completeness, and accuracy, to link to the specific probes as opposed to the top level of the probes dictionary page.

Other than those clarification this looks good to me.
If I recall correctly the data stewardship steering committee agreed that data stewards should not be performing the canonical review of products that they worked on. So Im going to kick this back to triage with the comments above as (hopefully) helpful context.

@nshadowen, are you the right person to reassign this to an unbiased steward?

Comment 4

[Nicole Shadowen]

Hi there, happy to jump in. I've been out on PTO and am catching up. Will have this review completed by tomorrow EOD. Thanks.

Comment 5

[Nicole Shadowen]

Hi Luca, can you please make the changes as suggested by Martin, and then I will review? Thanks

Comment 6

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: request_bug_1699225_taar_enabled_extra_var-rev2.md

Hi Luca, can you please make the changes as suggested by Martin, and then I will review? Thanks

Hi Nicole, thanks for the ping.

I tweaked point 1 and 2 to clarify that we are referring to the engagement with the about:addons page (and effect on the recommended extensions install rates and disco and list extensions views impressions).

I linked the comment from Scott in the review request because it does contain some more details and I wasn't sure of the value of just copying and pasting that in the data review request itself (vs. linking it as I did).

If we need more details for those two points, I'll talk with Scott again (because I'm not going to cover the analysis and I may not have much more details than the ones I did mention in the data review request).

About point 6 I wasn't completely sure how to link the collection (this is also how I referred to changes to the data collection in some previous data review request) and so it it currently unmodified:

• I can't link the additional extra var until the patch is actually landed
• I could link the telemetry events that I did mention by name in the point 6, but
  • a permalink wouldn't include the new extra var because they are not yet landed
  • a non-permalink may not point to those telemetry events anymore in the future

and so I'm mentioning the telemetry events and the new extra var by their name (which should make it possible to verify in the future if the probe is already there, also the telemetry event will also mention the bugzilla issue and that would also allow to identify the collection in the future on searchfox).

Let me know wdty.

Comment 7

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: request_bug_1699225_taar_enabled_extra_var-rev3.md

Hi Nicole,
I've just replaced the data review request attached to this bug with a new revision.

While waiting for confirmation from your side on the proposed additions for this data collection, I was finalizing the attached patch and I did notice that we may be actually able to improve further our ability to compare the recommended addon installations between TAAR-based and statically curated ones (by using a property that we are already getting from addons.mozilla.org which wasn't yet used anywhere but it seems that it has been added exactly for this kind of use case).

The new revision of the data review request is exactly like the previous one, beside the answer to the question (5), where there is now a new entry for the taar_based extra var which we would like to use for the addonsManager.install_stats, and addonsManager.action telemetry events.

Would you mind to take a look to this new revision?
(if that is ok from a data review perspective I'll proceed with updating the patch accordingly)

Comment 8

[Nicole Shadowen]

Comment on attachment 9212255 [details]
request_bug_1699225_taar_enabled_extra_var-rev3.md

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
Yes, once landed, the probes will be documented publicly here once patch is landed.
addonsManager.install_stats: https://searchfox.org/mozilla-central/rev/9043e515e9608cc55b252a40cb2dfb6f767bcffd/toolkit/components/telemetry/Events.yaml#192-217 (edited)
11:31
addonsManager.action: https://searchfox.org/mozilla-central/rev/9043e515e9608cc55b252a40cb2dfb6f767bcffd/toolkit/components/telemetry/Events.yaml#285-313 (edited)
11:31
addonsManager.view: https://searchfox.org/mozilla-central/rev/9043e515e9608cc55b252a40cb2dfb6f767bcffd/toolkit/components/telemetry/Events.yaml#264-284 (edited)

Is there a control mechanism that allows the user to turn the data collection on and off?
Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?
'addonsManager.install_stats' is permanent and will be monitored by [e-mail]
'addonsManager.view' and 'addonsManager.action' will expire with version 92

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
Category 2, Interaction.

Is the data collection request for default-on or default-off?
Default-on.

Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
No.

Is the data collection covered by the existing Firefox privacy notice?
Yes.

Does there need to be a check-in in the future to determine whether to renew the data?
Yes. [e-mail] is responsible for making sure 'addonsManager.view' and 'addonsManager.action' expire in version 92.

Does the data collection use a third-party collection tool?
No.

---

Result: datareview+

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d187ac759948
Add taar_enabled/taar_based extra vars to addonsManager telemetry. r=willdurand,robwu

Comment 10

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/d187ac759948