1459318 - Enable Telemetry for Activity Stream Router

Status: RESOLVED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Nan Jiang [:nanj]]

As a followup for bug 1433214 and bug 1456127, let's enable telemetry for a-s router, document all the new pings, and kick off the data review in this thread.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4142

Comment 2

[Nan Jiang [:nanj]]

1) What questions will you answer with this data?

To measure the user engagement for Activity Stream Router Experiment, including the new Snippets and Firefox Onboarding Tour implementation.

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

This helps us understand how often the user interacts with this experiment. Further, to evaluate the experiment hypothesis of whether or not users are seeing snippets at a frequency close to what is expected.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

There is no current instrumentation of this code.

4) Can current instrumentation answer these questions?

There is no current instrumentation of this code.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.   

* Impression (Category 3, bugzilla #: 1459318), reports an impression ping when Activity Stream Router serves a UI component to a user, such as a Snippet message.
* User interactions (Category 3, bugzilla #: 1459318), reports an event ping when a user interacts with a UI component, such as click a button or block a message.

It's worth noting that, like Activity Stream impression telemetry, all the pings above inherit the use of impression_id other than client_id to maintain the anonymity of the telemetry.

6) How long will this data be collected?  Choose one of the following:

This is scoped to a time-limited experiment.

7) What populations will you measure?

The channel where the experiment will be conducted only in English-speaking locales.

8) If this data collection is default on, what is the opt-out mechanism for users?

The user can opt-out the data collection by either disabling the telemetry of Activity Stream or disabling the Firefox telemetry as a whole.

9) Please provide a general description of how you will analyze this data.

General reporting on Redash.

10) Where do you intend to share the results of your analysis?

This will be shared over Mozilla LDAP via Redash.

Comment 3

[François Marier [:francois]]

(In reply to Nan Jiang [:nanj] from comment #2)
> 5) List all proposed measurements and indicate the category of data
> collection for each measurement, using the Firefox [data collection
> categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla
> wiki.   
> 
> * Impression (Category 3, bugzilla #: 1459318), reports an impression ping
> when Activity Stream Router serves a UI component to a user, such as a
> Snippet message.
> * User interactions (Category 3, bugzilla #: 1459318), reports an event ping
> when a user interacts with a UI component, such as click a button or block a
> message.

This falls into the category of data collection requests that need to be escalated to Legal:

- The data falls within the Web activity category (3) AND is default-on.

Marshall, from what I can gather here, the Category 3 data essentially boils down to which snippet/pocket story the user has clicked on (and therefore visited).

It's time-limited but default-on.

Comment 4

[Nan Jiang [:nanj]]

(In reply to François Marier [:francois] from comment #3)

> This falls into the category of data collection requests that need to be
> escalated to Legal:
> 
> - The data falls within the Web activity category (3) AND is default-on.

Thanks :francois for the feedback! As we're still targeting 61 for this experiment, would it be easier for you to sign off the data review if we adjust the proposed telemetry. For instance, if we take out the "impression_id", does that make it a cat-3 without legal's review request?

In the long run, we'd still like to be able to collect the telemetry as the original proposal, and we're open to have followup conversations to discuss it with you and legal folks.

Comment 5

[Merwin]

Nan,

I'm not familiar with the "Activity Stream Router Experiment."  Can you give me more context on what this is?

Marshall

Comment 6

[Kate Hudson :k88hudson]

The Activity Stream Router experiment is testing a technical refactor of our current snippets and onboarding systems on the new tab page / about:home. We would like to collect to data through our data pipeline (Ping Centre) rather than the current snippets sytem (GA) as part of this refactor. 

There are no visible UI changes for snippets [1], however, there is new onboarding UI [2]. 

The whole experiment scope for 61/62 is here: https://docs.google.com/document/d/1E_1hxrU4GNMwKz6rwN4Iw8LoTefUmUl6wneyJ9-7U84. Happy to provide more clarification if needed

[1] Snippet design spec is here: https://mozilla.invisionapp.com/share/UVFOR9UY3X6#/screens
[2] Onboarding design spec https://mozilla.invisionapp.com/share/2EHWPLNSPQ6#/screens/294845569

Comment 7

[François Marier [:francois]]

After talking to Nan on IRC, it became clear that I was confused about what the impression_id was. It's actually a kind of user_id.

So, here are the two reasons I can't complete this data review without Marshall's help:

1. The impression_id is a unique identifier that is stable for the lifetime of a browser profile and could uniquely identify a user.

2. The message_id tells us which exact message (including landing URL) a user clicked on which makes this "default ON Category 3 data".

Comment 8

[Merwin]

(In reply to François Marier [:francois] from comment #7)
> After talking to Nan on IRC, it became clear that I was confused about what
> the impression_id was. It's actually a kind of user_id.
> 
> So, here are the two reasons I can't complete this data review without
> Marshall's help:
> 
> 1. The impression_id is a unique identifier that is stable for the lifetime
> of a browser profile and could uniquely identify a user.

What is the purpose of the impression ID?  Is it distinct from the standard client ID?  Can it in any way be associated with the client ID?

Ideally this should NOT be associated with the main client ID and should reset when the user turns on and off the telemetry settings. Assuming those things are the case (and that this honors the telemetry setting in general), I think it is okay. 

Nan, can you confirm? 

> 
> 2. The message_id tells us which exact message (including landing URL) a
> user clicked on which makes this "default ON Category 3 data".

This is an exception we've made in the past for category 3 data.  It is the same risk that exists within the current snippets system and similar to the risk of impression and click collection for Pocket content.  Compared to Pocket, the risk is more limited because the content is much more limited and typically focused on Mozilla landing pages.

Will we be doing any personalized snippet content.  Is that snippet content still largely focused on Mozilla?

Comment 9

[Nan Jiang [:nanj]]

(In reply to Merwin from comment #8)

> What is the purpose of the impression ID?  Is it distinct from the standard
> client ID?  Can it in any way be associated with the client ID?
> 
> Ideally this should NOT be associated with the main client ID and should
> reset when the user turns on and off the telemetry settings. Assuming those
> things are the case (and that this honors the telemetry setting in general),
> I think it is okay. 
> 
> Nan, can you confirm?

Impression ID will be used as the unique ID that allows us to calculate the unique blocks/clicks of a specific snippets message. Actually it's already being used by Pocket in Activity Stream, therefore it has NO association with the client ID. Also, all the telemetry in this experiment honors the user's telemetry settings (both at the Activity Stream level and Firefox level).

> Will we be doing any personalized snippet content.  Is that snippet content
> still largely focused on Mozilla?

There is no personalization in this experiment, and the snippets content will be largely focused on Mozilla. Although we might want to do some targeting in the following iterations, it's not in the scope of this experiment.

If you have any other questions, we can schedule a meeting with A-S team and Francois to clarify it in more detail.

Comment 10

[Nan Jiang [:nanj]]

Hi Marshall, any updates on this? Let us know if you still have concerns or questions so that we can schedule a meeting to clarify them.

Thanks!

Comment 11

[Merwin]

No concerns.  While this will be using telemetry, it seems to maintain the same privacy properties as our current snippet service and has roughly the same properties (or better) than Activity Steam data collection, so it is within policy.

Comment 12

[Nan Jiang [:nanj]]

Thanks a lot for confirming this, Marshall!

:francois, could you resume the data review based on Marshall's comments above please?

Comment 13

[François Marier [:francois]]

datareview+

(In reply to Nan Jiang [:nanj] from comment #2)

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes: https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md

2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Not permanent.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 2 and 3.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

Yes, the impression_id is a kind of user_id (see comment 7).

7) Is the data collection covered by the existing Firefox privacy notice?

Yes, according to comment 11.

8) Does there need to be a check-in in the future to determine whether to renew the data?

Yes. Nan, is there a follow-up bug to remove this telemetry once the time-limited experiment is over?

Comment 14

[Nan Jiang [:nanj]]

(In reply to François Marier [:francois] from comment #13)

> 8) Does there need to be a check-in in the future to determine whether to
> renew the data?
> 
> Yes. Nan, is there a follow-up bug to remove this telemetry once the
> time-limited experiment is over?

Good idea! Let's track this in bug 1461466

Comment 15

[[github robot]]

Commits pushed to master at https://github.com/mozilla/activity-stream

https://github.com/mozilla/activity-stream/commit/d1f54414fd0e5c1cc831ce7780e8446e70dbfd19
Fix Bug 1459318 - Enable telemetry for AS Router

https://github.com/mozilla/activity-stream/commit/90ce2ea229d1dc661d2476c4910078d5bd530e64
Merge pull request #4142 from ncloudioj/bz1459318

Fix Bug 1459318 - Enable telemetry for AS Router

Comment 16

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/79ab9d325994