1485069 - Add client_id for snippets reporting

Status: RESOLVED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Lisa Wright]

With the newly established organization wide KPI of increasing aDAU, we need client id available to make better decisions, provide more relevant communication, better serve our user needs and most importantly - be able to measure the impact of our experimentation on product usage and retention.

Without client id in place, we won't be able to tie any of the snippet-related efforts to aDAU, which is essential for us to meet our company goals.

As part of the tiger team initiatives to increase aDAU, Porfirio will work with Marshall on the review needed to get this new attribute made available for in-product communications.

Comment 2

[Merwin]

Lisa, can you help me with the questions below:

1) Is there a specific snippet experiment that is driving this or is this a more general ask to collect the client ID for snippets?

2) Does snippet's current collect an ID in the snippet's ping?  If so, why is that not sufficient to measure retention using that ID?  Why do you need the client ID for this?  

Generally the approach we encourage various teams to take is to avoid collecting the client ID and instead establish a separate ID and then instrument the ping such that you can get the information you need for aDAU.  That approach should allow us to answer any of the aDAU questions we have without correlating to the main client ID, although it does take more engineering effort. So before we decide to use the client ID, I'd like to understand why that approach won't work here.

Comment 3

[Lisa Wright]

@Merwin - There are a few aDAU experiments in the queue. Plans are to get started with increase of usage via specific add-ons, adding a Firefox Account, and viewing Pocket/Mozilla content. 

Because we need client_id available in the snippet service in order to run these experiments, it looks like a more generic request. 

With regards to instrumenting a new id instead, we'd need Tim/Kate and George to answer whether that's an option to do that and still be able to understand the impact of the experiment on user/usage behavior.

Thanks!

Comment 4

[Tim Spurway [:tspurway]]

Hi Lisa,

Could we characterize the types of experiments that require client_id are around retention?  The example would be:  "we are going to run a campaign with Snippets X, Y, and Z and compare it to control with just X and Y and compare weekly retention for the two cohorts" with the premise being Snippet Z creates a lift in retention.  

If this is the case, then it would be necessary to collect the client_id, to make sure we account for people who remain active but (for a variety of reasons) don't send a Snippets ping.

If you don't care about retention and are more interested in DAU/aDAU/MAU, then we could include an 'impression_id' that has much better privacy characteristics, but still allows for accurately calculating Snippet's aDAU/DAU/MAU over time.

Comment 5

[George Kaberere [:gkaberere]]

The aDAU number we are trying to lift is not a specific snippet aDAU number but rather aDAU for Firefox overall. Given this, I don't think a snippet ID would work. Some of the limitations I foresee are:
1. If we use a snippet ID, we would have no idea of knowing whether the user assigned that snippet ID is considered aDAU in the overall fx metrics or not
2. If they interact with a snippet encouraging e.g. to install addons or make fx their default browser, I won't be able to tell if they actually completed the action / converted. 

Regarding experimentation. We are currently in the process of setting up our continuous experimentation process. One of the experiments being planned is a 100 day snippet journey for new acquisitions which aims to see if we can improve retention rates as well as create more active users with 5 or more pageviews for clients who go through the journey vs clients who do not.

Comment 6

[Merwin]

I spoke Spurway about this and understand the properties of this a bit better.  It does seem like the client ID is necessary for this purpose. Let's provisionally move forward with that.  However, I would like to know exactly what experiments we are considering to know exactly what data will be associated with the client ID. The ones mentioned in this bug seem ok but it sounds like we have a long list we are planning to set through.

Lisa, can you iterate through the "aDAU experiments in the queue." Here is the one's mentioned in this bug so far:
1. usage via specific add-ons;
2. adding a Firefox Account;
3. viewing Pocket/Mozilla content
4. 100 day snippet journey

Comment 7

[Merwin]

I'm headed out on PTO but added Alicia Gray to this to handle while I'm out. You are good to move forward with using the client ID but assume that means that if there are particular experiments that would associated the ID with sensitive data, you might have to nix those.

Comment 8

[Michele Warther]

That's great - thank you @merwin. Lisa is out on PTO, I'll get the additional details on the full list of experiments in here for you/Alicia to review. The 100 day journey has a subset of experiments with in.

FYI - We have updated the email experiment so it's just based if they clicked/converted and not associated to a specific email address, that was the only one that might have been considered sensitive in the mix.

Comment 9

[Lauren Niolet]

I have a NI here, but am unsure what it's for. :mwarther:?

Comment 10

[Michele Warther]

@lniolet - it was for snippet experiment details, I've clear it and added them below.

Marshall - FYI, the details on some of the initial experiments. The hypothesis for all the experiments are if a user interacts with them, there will be a increase in their active days and positive impact on retention.

FxAccounts - example copy
Browse boldly. Sign up for a Firefox Account to sync your protected passwords, open tabs and bookmarks on all your devices.

Addons (general) - copy for tests
Power up Firefox with add-ons. Choose from dozens of customizations for privacy, social, gaming, and a whole lot more.
Personalize Firefox with add-ons for privacy, social, gaming, and a whole lot more. Make Firefox yours.
Powerful privacy tools, media enhancers and more: our essential extensions collection is loaded with awesome add-ons. Check them out.

Addons (specific) - copy for tests
Tired of using multiple browsers to view your 2nd email? Now you can do it all in Firefox with Multi-Account Containers!
Using private browsing mode to juggle multiple accounts? Now there's a better way. Try Multi-Account Containers!

Test Pilot - copy for tests
Want to save a tree or two? Jot down your thoughts directly in Firefox with Notes, a new Test Pilot experiment.

Be among the first to try out new features in Firefox. Sign up to be a Test Pilot today.

Mission/Internet Health - copy for tests
Hey, nice work choosing Firefox for your webby journeys. Not all heroes wear capes and by browsing with us, you're helping build a better internet.

Firefox Mobile - copy for tests
Browse without boundaries. Send Firefox to your mobile device for better browsing on the go.
Browse unbound. Send Firefox to your phone and take a powerful independent browser with you.
Full-featured. Customizable. Lightning fast. Browse without compromise with Firefox Mobile.

We are currently working with the analytics team on the measurement strategy (how we will define success as a group in addition to each individual snippet) and plan to run the test over a 12 week period to ensure we are at the retention tipping point, as well as to have statistical significance. The start date is TBD, based on the date that the the client id could be uplifted. order

Comment 11

[Michele Warther]

Attachment: Snippets ASR Telemetry Policy Rationale

Marshall - we meet last week to review with Michael/legal and created this document to provide some answers to outstanding questions. If you could please reconfirm your approval, the engineering team will get this wrapped up and we can confirm in beta before the launch into release for 63 (10/23). Thank you.

Comment 12

[Merwin]

Michele, apologies for the delay.  This is good to go from my perspective.

Comment 13

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4454

Comment 14

[Nan Jiang [:nanj]]

Attachment: data_review_request_snippets.txt

Hey Francois, could you take a data review on this, please? The legal signoff is already granted above.

Thanks!

Comment 15

[François Marier [:francois]]

From https://docs.google.com/document/d/1K57lJT-LJlJlMIOwHl0QQH81R7VCBOghtmoR3BpwsiI/edit?ts=5baea2ea, I see that the URLs are not restricted to just Mozilla channels, but there is also an exception for Wikipedia:

> Are there any restrictions on message content or targeting when this additional data is collected?
> Restrictions on URLS:
> Mozilla owned or managed channels only, with the exception of Wikipedia. With regards to Wikipedia specifically, they
> only understand the referring traffic came from Mozilla but absolutely no data is passed to them beyond.

To which Michelle added:

> these are mostly the delight snippets so they link to "what is a bathysphere" https://en.wikipedia.org/wiki/Bathysphere
> or "we are celebrating Ada's birthday https://en.wikipedia.org/wiki/Ada_Lovelace if we don't have an internal blog post
> to link to.

This is default ON Category 3 data, so I do need to ask Marshall to confirm that he's aware of this and that it is covered by our privacy policy.

I'm not personally very comfortable with a blanket exception for Wikipedia. While it's unlikely that such material would make it to a snippet, there is a lot of sensitive material on that site that could harm some users if it were linked to them. Perhaps this exception needs to be restricted in some way.

Comment 16

[Michele Warther]

@francois @merwin - we can kept to managed locations for this if preferred or only specific pages, although it sounds like because they might navigate to another page that might not be ideal.

Comment 17

[Merwin]

Michele, let's stick to the managed locations for the moment.  I'm not as concerned as Francois, given that in all likelihood we will not be linking to that sensitive material, but historically in recommendation services we have considered, we have tried to avoid, outside of Mozilla's own properties, being able to infer the site the user can visit.

Comment 18

[François Marier [:francois]]

Comment on attachment 9012901 [details]
data_review_request_snippets.txt

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_events.md.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting (or Activity Steam telemetry setting in about:config).

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Yes, George Kaberere.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Technically, category 3, but since all of the outbound links are Firefox/Mozilla related (comment 17), we have in the past considered that as if it were category 2. Given that Marshall has given his approval in comment 12, I assume that's the case here too.

5) Is the data collection request for default-on or default-off?

Default ON.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

Yes, the client_id. It's not a new identifier since it's used in telemetry, but the point of this change is to link it to new data.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes, as per comment 12.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, permanent.

Comment 19

[[github robot]]

Commit pushed to master at https://github.com/mozilla/activity-stream

https://github.com/mozilla/activity-stream/commit/ec0d77add087fc4772891cb723fb6213bea6ae49
Fix Bug 1485069 - Add client_id for Snippets reporting (#4454)

Comment 20

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4463

Comment 21

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/60fed71d173f

Comment 22

[Nan Jiang [:nanj]]

Attachment: Bug 1485069 - Add client_id for Snippets reporting

Comment 23

[Nan Jiang [:nanj]]

Comment on attachment 9014427 [details]
Bug 1485069 - Add client_id for Snippets reporting

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: Bug 1485069

User impact if declined: We will not be able to measure the user engagement impact of the new Snippets feature in Activity Stream.

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): It just changes a field in the current telemetry payload for Snippets.

String changes made/needed: No

Comment 24

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9014427 [details]
Bug 1485069 - Add client_id for Snippets reporting

Low risk, uplift approved for 63 beta 13, thanks.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/241c336e1fd0