1457226 - [Shield] Launch initial exploratory study to understand relationship between product analytics and user retention (Project Savant + Project Critical Event)

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Chris More [:cmore]]

==Hypothesis ==

Collecting anonymized event and usage data on the Firefox category 2 UI [1] for a representative sample of release users will enable us to:

* Understand frequency of actions on primary UI elements
* Understand how an action or series of actions leads to a retention improvement and the "aha moment" [2]
* Understand most frequent series of actions

Firefox critical event definition (aha moment):

An user action or series of user actions within the product within the first week of use of Firefox that aligns to one of our value propositions and leads to a retention improvement higher than the aggregate week one retention mean.

== Study Overview == 

This experiment is not an A/B test and is exploratory in nature. If one or more critical events are discovered, real A/B tests will be created in the future to understand the difference between correlation and causation of the critical event.

For this project, we will create a Firefox system add-on (deployed via Shield native study) that will use desktop event telemetry to capture the most common user experience events in the browser and send them back through a pipeline for later analysis. 

== Timeline == 

The goal is to launch this study by the end of Q2 2018 so that insights can be used in Q3 2018.

== Supporting Documentation ==

This project initial started out as two separate projects called the Critical Event and Savant.

Original critical event PHD: https://docs.google.com/document/d/1JZoiGw8WiZRU-OtHekAvhJ3GafKM80g18ABMoe7Get8/edit

Original Savant proposal: https://docs.google.com/document/d/1D2DoSWoNcGQg9jesL-jbYaf3Eis7gmy0HjsgvXecNPE/edit 

We combined the two project into one given the similar user event instrumentation and analysis outcomes.

Combined Savant + Critical Event PHD: https://docs.google.com/document/d/1emT5bdFQoMh5Zl0bLZS5fHKWKhXRpLrptDesKdfQYCw/edit

[1] https://wiki.mozilla.org/Firefox/Data_Collection#Data_Collection_Categories

[2] https://amplitude.com/blog/2016/09/15/user-retention-app-critical-event/

Comment 1

[Bianca Danforth [:bdanforth]]

cmore: I know this study has a designated analyst (:shong), but is there a designated survey writer/analyst?

Comment 2

[Chris More [:cmore]]

(In reply to Bianca Danforth [:bdanforth] from comment #1)
> cmore: I know this study has a designated analyst (:shong), but is there a
> designated survey writer/analyst?

Good question. The last time we ran a survey, I worked with tdowner and Francis Djabri. Let's needinfo them here to see if they could help with writing a post-experiment survey. The survey will just ask general questions if they are new to Firefox and what features or functionality did they find valuable. 

Tyler:Francis: can you let us know if either of you could help with a post exploratory analysis survey?

Comment 3

[Tyler Downer [He/Him]]

I can definitely help with that. Please follow the directions at https://docs.google.com/document/d/1RKQYoM4wbWn4MYt_v3xLXo2vxzpig2mUWxaVFFI1tFI/edit to submit a survey request doc (Like a PHD, but much lighter)

Comment 4

[Bianca Danforth [:bdanforth]]

pdol: In cmore's absence, would you be able to secure a survey writer/analyst and obtain a data review by a Data Steward for this study? Please see Tyler's previous comment for how to request a survey through S&I.

Comment 5

[Peter Dolanjski [:pdol]]

(In reply to Bianca Danforth [:bdanforth] from comment #4)
> pdol: In cmore's absence, would you be able to secure a survey
> writer/analyst and obtain a data review by a Data Steward for this study?
> Please see Tyler's previous comment for how to request a survey through S&I.

I'll defer this to :RT

Comment 6

[Romain Testard [:RT]]

(In reply to Peter Dolanjski [:pdol] from comment #5)
> (In reply to Bianca Danforth [:bdanforth] from comment #4)
> > pdol: In cmore's absence, would you be able to secure a survey
> > writer/analyst and obtain a data review by a Data Steward for this study?
> > Please see Tyler's previous comment for how to request a survey through S&I.
> 
> I'll defer this to :RT
The request was sent on May 14th to surveys-review FYI

Comment 7

[Romain Testard [:RT]]

Hi Rob, Bianca pointed me your way for data review on this project. Can you please help review and confirm if this is approved?
Happy to meet to review the details if helpful, please ping me on slack and I can arrange this.

Comment 8

[Romain Testard [:RT]]

(In reply to Romain Testard [:RT] from comment #7)
> Hi Rob, Bianca pointed me your way for data review on this project. Can you
> please help review and confirm if this is approved?
> Happy to meet to review the details if helpful, please ping me on slack and
> I can arrange this.

Apologies, wrong data steward pinged here, clearing NI for Rob and asking chutten for review instead since apparently he's been working on the patch himself.

Comment 9

[Bianca Danforth [:bdanforth]]

This observational study will take the form of a preference experiment (http://normandy.readthedocs.io/en/latest/user/actions/preference-experiment.html) with code landing in the tree. Due to the cross-cutting nature of the probes, my approach will be to group probes by Firefox component and file separate bugs for each group. The first bug, bug 1462725, lays the groundwork for this.

Comment 10

[Chris H-C :chutten]

I should be available to review the data collection review requests that arise from adding the various events to Firefox.

Comment 11

[Peter Dolanjski [:pdol]]

Here is the data review form for these new probes:

1. Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
Yes, a telemetry.md document will be produced and published publicly describing the schema.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
Yes, these probes will simply use the existing telemetry opt-out.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
Temporary for now (if it proves useful, then it may be made permanent)

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
Category 2: Interaction Data

5. Is the data collection request for default-on or default-off?
Default on (for 1% of users in 2 cohorts)

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
No

7. Is the data collection covered by the existing Firefox privacy notice?
Yes

8. Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**
No, since it is temporary (and will be disabled after the shield study)

Comment 12

[Bianca Danforth [:bdanforth]]

(In reply to Peter Dolanjski [:pdol] from comment #11)
> Here is the data review form for these new probes:
> 
> 1. Is there or will there be documentation that describes the schema for the
> ultimate data set available publicly, complete and accurate?
> Yes, a telemetry.md document will be produced and published publicly
> describing the schema.
> 
> 2. Is there a control mechanism that allows the user to turn the data
> collection on and off?
> Yes, these probes will simply use the existing telemetry opt-out.
> 
> 3. If the request is for permanent data collection, is there someone who
> will monitor the data over time?
> Temporary for now (if it proves useful, then it may be made permanent)
> 
> 4. Using the category system of data types on the Mozilla wiki, what
> collection type of data do the requested measurements fall under?
> Category 2: Interaction Data
> 
> 5. Is the data collection request for default-on or default-off?
> Default on (for 1% of users in 2 cohorts)
> 
> 6. Does the instrumentation include the addition of any new identifiers
> (whether anonymous or otherwise; e.g., username, random IDs, etc. See the
> appendix for more details)?
> No
> 
> 7. Is the data collection covered by the existing Firefox privacy notice?
> Yes
> 
> 8. Does there need to be a check-in in the future to determine whether to
> renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if
> appropriate)**
> No, since it is temporary (and will be disabled after the shield study)

Thanks pdol; chutten: Here is the Data Collection form for your review. This form is for all 13 of the probes planned for this study. Please let me know if there is anything more you need before you can begin review of bug 1462725.

Comment 13

[Chris H-C :chutten]

The answer to question 4 should be a table of the measures being collected, the categories under which they fall, and the bug numbers in which they will be implemented. Could you please add that here (a comment will be fine)

Comment 14

[Chris H-C :chutten]

Actually... are you looking at request.md or review.md?

review.md is the one -I'm- supposed to fill out :)

Comment 15

[Bianca Danforth [:bdanforth]]

(In reply to Chris H-C :chutten from comment #14)
> Actually... are you looking at request.md or review.md?
> 
> review.md is the one -I'm- supposed to fill out :)

chutten: Here is the correct form. Please let me know if you need anything else for the Data Collection Review.

---

# Request for data collection review form

**All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.**

1) What questions will you answer with this data?

* Are there critical events that cause a user to continue or stop using Firefox?

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

* Understand what the most important tasks are that users attempt in the browser
* Design product solutions to make these actions easier and more robust to keep users in the product longer.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

* We have already launched a series of surveys to find out what users are doing in the first session. Even though that work was very informative, it does not provide the full story. We need the behavioral data as well.

4) Can current instrumentation answer these questions?
* No. We collect histogram and scalar data for some of the actions we hypothesize to be important, but we are interested in time series data to learn if there are early actions or series of actions that impact retention – particularly for new users in their first session.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.   

**Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.**

<table>
  <tr>
    <td>Measurement Description</td>
    <td>Data Collection Category</td>
    <td>Tracking Bug #</td>
  </tr>
  <tr>
    <td>"search": Client makes a search</td>
    <td>2</td>
    <td>[1462725](https://bugzilla.mozilla.org/show_bug.cgi?id=1462725)</td>
  </tr>
  <tr>
    <td>"end_study": Client ends the study</td>
    <td>2</td>
    <td>[1462725](https://bugzilla.mozilla.org/show_bug.cgi?id=1462725)</td>
  </tr>
  <tr>
    <td>"pw_manager": Password Manager prompts user to save or update a password</td>
    <td>2</td>
    <td>[1465685](https://bugzilla.mozilla.org/show_bug.cgi?id=1465685)</td>
  </tr>
    <tr>
    <td>"pw_manager_use": Client uses a password from the Password Manager</td>
    <td>2</td>
    <td>[1465685](https://bugzilla.mozilla.org/show_bug.cgi?id=1465685)</td>
  </tr>
    <tr>
    <td>"login_form": Client loads or submits a login form</td>
    <td>2</td>
    <td>[1465685](https://bugzilla.mozilla.org/show_bug.cgi?id=1465685)</td>
  </tr>
  <tr>
  <td>"tab": Client loads or submits a login form</td>
    <td>2</td>
    <td>[1465694](https://bugzilla.mozilla.org/show_bug.cgi?id=1465694)</td>
  </tr>
    <tr>
    <td>"library_open": Client opens the library menu</td>
    <td>2</td>
    <td>[1465697](https://bugzilla.mozilla.org/show_bug.cgi?id=1465697)</td>
  </tr>
  <tr>
    <td>"hamburger_open": Client opens the hamburger menu</td>
    <td>2</td>
    <td>[1465697](https://bugzilla.mozilla.org/show_bug.cgi?id=1465697)</td>
  </tr>
  <tr>
    <td>"dotdotdot_open": Client opens the dotdotdot menu</td>
    <td>2</td>
    <td>[1465697](https://bugzilla.mozilla.org/show_bug.cgi?id=1465697)</td>
  </tr>
  <tr>
    <td>"readermode": Client turns on or turns off reader mode feature</td>
    <td>2</td>
    <td>[1465698](https://bugzilla.mozilla.org/show_bug.cgi?id=1465698)</td>
  </tr>
  <tr>
    <td>"bookmark": Client saves or removes a bookmark</td>
    <td>2</td>
    <td>[1465703](https://bugzilla.mozilla.org/show_bug.cgi?id=1465703)</td>
  </tr>
  <tr>
    <td>"follow_bookmark": Client opens a bookmark</td>
    <td>2</td>
    <td>[1465703](https://bugzilla.mozilla.org/show_bug.cgi?id=1465703)</td>
  </tr>
  <tr>
    <td>"follow_awesomebar_link": Client clicks on a link displayed by the awesomebar</td>
    <td>2</td>
    <td>[1465704](https://bugzilla.mozilla.org/show_bug.cgi?id=1465704)</td>
  </tr>
  <tr>
    <td>"addon": Client installs, enables, disables, or removes an addon</td>
    <td>2</td>
    <td>[1465707](https://bugzilla.mozilla.org/show_bug.cgi?id=1465707)</td>
  </tr>
</table>

6) How long will this data be collected?  Choose one of the following:

* This is scoped to a time-limited experiment/project until (on or around) date 07-30-2018.

7) What populations will you measure?

1% of each cohort in Release 61:
* Cohort 1: en-US and geo United States
* Cohort 2: de-DE and geo Germany

8) If this data collection is default on, what is the opt-out mechanism for users?

* Opting out can be done through the default telemetry mechanism.

9) Please provide a general description of how you will analyze this data.

* The data will be sorted by new and existing users in each of the two cohorts described above. We will aggregate first session new user actions and examine the relationship, if any, between those actions and retention. We will also look at the frequency of actions or series of actions to understand which features are most and least used.

10) Where do you intend to share the results of your analysis?

* The analysis will be shared with the Product Management team and likely at a few internal meetings initially.

Comment 16

[Chris H-C :chutten]

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Standard Telemetry mechanisms apply. (Events.yaml covers the collections, the probe dictionary should make them searchable, and they will appear in the events dataset as documented on docs.tmo)

    Is there a control mechanism that allows the user to turn the data collection on and off?

Standard Telemetry mechanisms apply.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 2, Interaction.

    Is the data collection request for default-on or default-off?

Default-on for study participants.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

It does not appear so, no. The search events will contain the search engine name, if I'm not mistaken.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data? 

Yes, as part of a study.

---
Result: datareview+, pending answers.

 :bdanforth, will any of the events contain string identifiers? Things like addon ids, the actual text or URL address of links, or anything of that sort? I need a list.

Comment 17

[Bianca Danforth [:bdanforth]]

(In reply to Chris H-C :chutten from comment #16)
> DATA COLLECTION REVIEW RESPONSE:
> 
>     Is there or will there be documentation that describes the schema for
> the ultimate data set available publicly, complete and accurate?
> 
> Standard Telemetry mechanisms apply. (Events.yaml covers the collections,
> the probe dictionary should make them searchable, and they will appear in
> the events dataset as documented on docs.tmo)
> 
>     Is there a control mechanism that allows the user to turn the data
> collection on and off?
> 
> Standard Telemetry mechanisms apply.
> 
>     If the request is for permanent data collection, is there someone who
> will monitor the data over time?
> 
> N/A
> 
>     Using the category system of data types on the Mozilla wiki, what
> collection type of data do the requested measurements fall under?
> 
> Type 2, Interaction.
> 
>     Is the data collection request for default-on or default-off?
> 
> Default-on for study participants.
> 
>     Does the instrumentation include the addition of any new identifiers
> (whether anonymous or otherwise; e.g., username, random IDs, etc. See the
> appendix for more details)?
> 
> It does not appear so, no. The search events will contain the search engine
> name, if I'm not mistaken.
> 
>     Is the data collection covered by the existing Firefox privacy notice?
> 
> Yes.
> 
>     Does there need to be a check-in in the future to determine whether to
> renew the data? 
> 
> Yes, as part of a study.
> 
> ---
> Result: datareview+, pending answers.
> 
>  :bdanforth, will any of the events contain string identifiers? Things like
> addon ids, the actual text or URL address of links, or anything of that
> sort? I need a list.

:chutten, short answer: No.
Earlier proposals for this study did include string identifiers, but those probes were ultimately excluded. If you would like to see the schema for this study's probes, please NI the study's analyst, Su-Young Hong (shong@).

Comment 18

[Bianca Danforth [:bdanforth]]

Forgot to NI you on my reply above ^^ (Comment 17).

Comment 19

[Chris H-C :chutten]

ni?shong - Are we including string identifiers in the preliminary set of Savant events? If so, could you please list them. They could be things like addon ids, search engine names, or other strings.

Looking at the spreadsheet (https://docs.google.com/spreadsheets/d/1zCNCdEHz83uK9i5ozQEfJdV7NZBGS6RcOwrxnfkOtz0/edit#gid=0) I see "info about engine", "name of addon", and "flow_id".

"info about engine" what info, exactly?
"name of addon" do you mean addon id? 
"flow_id" it says it is built as a hash. A hash of what? 

(am I missing anything?)

Comment 20

[Bianca Danforth [:bdanforth]]

(In reply to Chris H-C :chutten from comment #19)
> ni?shong - Are we including string identifiers in the preliminary set of
> Savant events? If so, could you please list them. They could be things like
> addon ids, search engine names, or other strings.
> 
> Looking at the spreadsheet
> (https://docs.google.com/spreadsheets/d/
> 1zCNCdEHz83uK9i5ozQEfJdV7NZBGS6RcOwrxnfkOtz0/edit#gid=0) I see "info about
> engine", "name of addon", and "flow_id".
> 
> "info about engine" what info, exactly?
> "name of addon" do you mean addon id? 
> "flow_id" it says it is built as a hash. A hash of what? 
> 
> (am I missing anything?)

Note: The spreadsheet you linked is a modified version that I created for engineering annotations. Here is the official spec sheet linked in the PHD (Comment 1): https://docs.google.com/spreadsheets/d/192unlFYdQ_nc5UVaG79YhY2bp6bV7XZm1G4BW-etOLE/edit?usp=sharing. Sorry for the confusion, though the probe information is the same; refer to the blue rows in sheet 2.

---

:chutten: Here's a partial answer and a question for you:

Partial answer:
* "flow_id" is a hash. For the add-on probe, it's a hash of the add-on ID and the telemetry client ID. For the Password Manager and login form probes, it's a hash of the site URI prepath (example: "https://github.com/") and the telemetry client ID. Would either of these be problematic?
* "info about engine" is the same information we currently collect with the existing "navigation" category "search" event, as this probe is a duplicate of that one under the "savant" category. Per their description in Events.yaml, it's the id of the search engine used.

Question:
Blanket collection of add-on IDs is considered Category 3 data collection right? What would our options be for re-scoping collection so that it is Category 2? For example, can we collect add-on IDs for a subset of add-ons? If so, what would the constraints on that subset be? For example, what if we only record the add-on ID if it's one of the top 20 add-ons on AMO?

Comment 21

[Bianca Danforth [:bdanforth]]

After meeting with cmore, shong and discussions with chutten, the plan going forward is:
* Remove the hash method; there will be no hashing.
* Remove the flow_id from the addon probe and send the addon ID as the `value` key.
* Time permitting, replace the value of the flow_id for the pw_manager, pw_manager_use and login_form probes to use the tab ID, otherwise remove the flow_id for these probes as well.

List of string identifiers by probe:
* “search” probe:
  - `value`: one of “enter”, “suggestion”, “oneoff” or “alias”
  - `extra`:
    - “engine”: the search engine ID
    - “subcategory”: “navigation”
* “end_study”:
  - `extra`:
    - “subcategory”: “shield”
* “pw_manager”:
- `extra`:
    - “subcategory”: “prompt”
    - “flow_id”: tab ID
* “pw_manager_use”:
  - `extra`:
    - “subcategory”: “feature”
    - “flow_id”: tab ID
* “login_form”:
- `extra`:
    - “subcategory”: “encounter”
    - “flow_id”: tab ID
* “tab”:
- `extra`:
    - “subcategory”: “frame”
Note: The following 3 “menu” probes may not end up going into the study due to time constraints.
* “library_open”:
- `extra`:
    - “subcategory”: “menu”
* “hamburger_open”:
- `extra`:
    - “subcategory”: “menu”
* “dotdotdot_open”:
- `extra`:
    - “subcategory”: “menu”
* “readermode”:
- `extra`:
    - “subcategory”: “feature”
* “bookmark”:
- `extra`:
    - “subcategory”: “feature”
* “follow_bookmark”:
  - `extra`:
    - “subcategory”: “navigation”
* “follow_urlbar_link” (previously known as “follow_awesomebar_link”):
  - `extra`:
    - “subcategory”: “navigation”
* “addon”:
- `value`: addon ID
- `extra`:
    - “subcategory”: “customize”

Comment 22

[Chris H-C :chutten]

These identifiers are mostly category labels. The two of specific note are "tab ID" which is category 1 technical (though the number of distinct tab ids is category 2 interaction, since they can collectively be used to infer the number of open tabs), and "addon id" which is category 2 interaction (and isn't a new identifier, as we use it to key some histograms (ADDON_SHIM_USAGE), at least one scalar (extensions.updates.rdf), and to key the entire activeAddons section of the Telemetry Environment).

datareview+

Comment 23

[Bianca Danforth [:bdanforth]]

Bug 1433335 (added under "See Also") is not directly related to this study, but is an independent, parallel effort by the add-ons team to instrument add-on events. In general, I think a good pattern to follow for adding probes permanently in various components of Firefox would be to work with the engineering teams for that component.

My plan is to revert the study patches (temporary probes) once the study is over unless instructed otherwise.

Comment 24

[Robert (rrayborn, he/him)]

We launched yesterday evening.  It looks like Bianca is working on coordinating some early data validation meetings.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

I realized today that I hadn't actually left a comment anywhere officially giving a RelMan blessing on this study even though I'd already done so verbally. So, a bit belated, but consider this the RelMan signoff for posterity's sake :)

Comment 26

[Su-Young Hong]

Added data monitoring dashboard here: 

https://sql.telemetry.mozilla.org/dashboard/savant-data-monitoring?fullscreen

Comment 27

[Matt Grimes [:Matt_G]]

We've paused enrollment for this project per Chris More. Users currently enrolled will stay enrolled. We'll end the study on Aug 20th.

Comment 28

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Disabled all 4 recipes, 489,490,491,492.

Comment 29

[Bianca Danforth [:bdanforth]]

Attachment: Bug 1457226 - Revert Savant Shield study commits; r=rhelmer

The Savant pref flip Shield study was ended on 08-20-18 (https://bugzilla.mozilla.org/show_bug.cgi?id=1457226#c28), so this patch removes all temporary study code from Firefox.

Comment 30

[Robert Helmer [:rhelmer]]

Comment on attachment 9004021 [details]
Bug 1457226 - Revert Savant Shield study commits; r=rhelmer

Robert Helmer [:rhelmer] has approved the revision.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/febddb5a4dc2
Revert Savant Shield study commits; r=rhelmer

Comment 32

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/febddb5a4dc2

Comment 33

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Su, is there a writeup, slide deck, and/or analysis repo that we can leave breadcrumbs to here?

Comment 34

[Su-Young Hong]

Hi Tim,

I have a repo with work around this for documentation/monitoring/implementation notes:

https://github.com/mozilla/SAVANT-study-analysis

For analysis, I don't have any materials. I was just involved in the planning and data collection / migration part of the project (i.e. get the data into Amplitude so end users could run analysis independently).

:cmore, are there any SAVANT slide decks / dashboards / writeups / analysis that we could link to here?

Comment 35

[Chris More [:cmore]]

Summary slide deck on experiment, insights and next steps:

https://docs.google.com/presentation/d/1kmmHE4sUnVOZx_g9uUf_gOd72QBVeYxBoDIHClC7mhY/edit?usp=sharing

Next steps is bug 1513025 being driven by :gxu