1382270 - (shield-study-containers) [Shield] Opt-in Study Containers

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Luke Crouch [:groovecoder]]

+++ This bug was initially created as a clone of Bug #1379251 +++

> Basic description of experiment/add-on and link to code.

Copy of the Containers Test Pilot experimental add-on to deploy to a broader Shield study audience.

https://github.com/mozilla/testpilot-containers

> What are the branches of the study? You can have multiple branches, but every study must include a control.

Our control branch is the existing on-boarding experience. We have a 2nd branch with a more privacy/security-focused onboarding experience.

> What percentage of users do you want in each branch? We can do branches of uneven proportions. Example: 20% overall with 5% in control and 15% in treatment 1. 

50% in each branch

> What Channels/locales do you intend to ship to?

Release; English

> What is your intended go live date and how long will the study run? All experiments have expiration dates. 

July 20 or ASAP

> Are there specific criteria for participants? We can include/exclude based on anything available in telemetry.

They must have the default privacy.userContextEnabled = false pref so we don't trash their existing containers

> What is the main effect you are looking for?  

Do users who receive Containers retain better than users who do not? Do users who receive privacy+security on-boarding engage more?

> Who is the owner of the data analysis for this study?

Luke Crouch, Ryan Harter

> Do you plan on surveying users at the end of the study? If so, please link to the survey.

https://www.surveygizmo.com/s3/3621810/shield-txp-containers

> Link to any relevant google docs / Drive files that describe the project.

https://docs.google.com/document/d/1WQdHTVXROk7dYkSFluc6_hS44tqZjIrG9I-uPyzevE8/edit?ts=5824ba12#heading=h.teb6zxd69yfy

> Paste a real telemetry packet generated by your addon for the Data Steward to review

{
  "test": "@shield-study-containers",
  "version": "2.4.0",
  "timestamp": 43,
  "variants": null,
  "payload": {
    "uuid": "{f7b24c29-2f84-d341-b0b8-b68958c78038}",
    "event": "page-requests-completed-per-tab",
    "userContextId": "firefox-default",
    "pageRequestCount": 0,
    "object": null,
    "category": "interactions",
    "variant": null
  }
}

> Links to prior art if it exists:

https://testpilot.firefox.com/experiments/containers

> If there are previous results (particularly if they allow for the removal of experimental branches), review them here.

https://sql.telemetry.mozilla.org/dashboard/txp-containers-executive-summary

Comment 1

[Luke Crouch [:groovecoder]]

glind, rweiss - can I get the appropriate science & data approvals?

Comment 2

[[e-mail]]

I'm still a little uneasy with the phrasing of the main effect that you're looking for, specifically:

> Do users who receive Containers retain better than users who do not?

"receive Containers" seems like a pretty bland claim to make for a retention story.  It seems more like you should be claiming "receive *and use* Containers" and establish what a minimum definition of "use" is here.  This is a pedantic nitpick, rather than a substantive part of the data review. 

As for your data review, we're testing a new approach to data review.  I am an r+ assuming that I can get answers to (2) and (5) below.

1) Is there documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
Yes (https://github.com/mozilla/testpilot-containers/blob/master/docs/metrics.md)

2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism)   
Not sure.  :groovecoder is this intended to be an opt in or opt out Shield study?

3) What type of data do the requested instruments or measurements fall under?
No assertion of data types has been provided.  Looking at the documentation, these all appear to be collection type 1 or 2.

4) Is the data collection default-on or default-off?
Not default-on in release, but not sure if this is an opt-out or opt-in study.  Regardless, if this is an opt-in study, only people who have consented to participating in the study will have measurement. 

5) Does the instrumentation include the addition of any new identifiers?
I don't believe so.  :groovecoder can you confirm?

6) Is the data collection covered by the existing Firefox privacy notice? If unsure: escalate to legal.
If the study obeys the assumptions I outline above, yes.

7) Does there need to be a check-in in the future to determine whether to renew the data?
If the results of this study result in a decision to ship the feature to release, yes.  Otherwise, no.

Comment 3

[[e-mail]]

And actually ni'ing :groovecoder.

Comment 4

[Luke Crouch [:groovecoder]]

2. This is an opt-in study. Our consent page content is here: https://github.com/mozilla/addons-server/pull/5884/files#diff-bfabcfe17efd48f5b434bfd8315e8e7a (and should be live on AMO sometime today)

3. Yes - only type 1 & type 2 data are collected.

5. We *do* generate a UUID value for each add-on install.

Comment 5

[Kamil Jozwiak [:kjozwiak]]

Attachment: Screen Shot 2017-07-25 at 2.55.37 PM.png

Prerequisite Test Cases:
========================

* ensured that the shield study isn't being installed when privacy.userContext.enabled;true
* ensured that the shield study is only being installed when a user has an english locale
** went through zh-TW, zh-CN, ja-JP-mac
* ensured that the shield study is only being installed on the release channel
** went through m-b and m-c

Test Cases:
===========

* ensured that the shield study is being installed without any issues
* ensured that the user is being placed in either "branch": "control", or "branch": "securityOnboarding" under about:healthreport -> Raw Data
* ensured that the correct on-boarding experience is being used when the user is placed under "branch": "control"
* ensured that the correct on-boarding experience is being used when the user is placed under "securityOnboarding"
* ensured that the shield study is removed correctly via about:addons (made sure the survey appears)
* ensured that the shield study is removed correctly after 28 days (made sure the survey appears)
* ensured that the prefs under about:config are reverted once the shield experiment expires or is manually removed
* ensured that all the container access points are removed once the shield study expires or has been removed
* ensured that the cookies.sqlite is correctly being cleared once the shield study expires or has been removed

Suggestions/Possible Issues:

Possible Issue #1:
------------------

We need to double check and ensure that the Shield/Normandy server only recruits users with privacy.userContext.enabled;false. Currently, the XPI from https://addons.mozilla.org/en-US/firefox/shield_study_15 doesn't validate the pref and installs even if the user has containers enabled via privacy.userContext.enabled;true. This causes some issues when the user decides to remove the shield study and already had containers enabled previously.

If the Shield/Normandy server doesn't validate the users pref, this needs to be fixed via the XPI.

Possible Issue #2:
------------------

We need to double check and ensure that the Shield/Normandy server only recruits users that are using english locales. Currently, the XPI from https://addons.mozilla.org/en-US/firefox/shield_study_15 doesn't validate locales and installs the shield study either way.

If the Shield/Normandy server doesn't validate users locales, this needs to be fixed via the XPI.

Possible Issue #3:
------------------

We need to double check and ensure that the Shield/Normandy server only recruits users that are on the m-r channel and not m-b or m-c. Currently, the XPI from https://addons.mozilla.org/en-US/firefox/shield_study_15 doesn't validate channels and installs the shield study on all the channels.

If the Shield/Normandy server doesn't validate the users channel, this needs to be fixed via the XPI.

Suggestion #1:
--------------

When you click on the "Learn more..." link under the add-on install modal window, you'll be forwarded to the following SUMO page:
* https://support.mozilla.org/en-US/kb/find-and-install-add-ons-add-features-to-firefox?as=u&utm_source=inproduct

Should we be opening the containers SUMO page instead? The way it's currently laid out, I would assume users would want more information about the container feature rather than getting a SUMO page about add-ons. If it's the same link that's on all XPI modals/windows, than I guess there's nothing we can really do. Attached a screenshot for more context.

Comment 6

[Luke Crouch [:groovecoder]]

Gregg - can Normandy restrict to recruiting English locale Release users with privacy.userContext.enabled = false?

Comment 7

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Yes we can!  And I have a tool for checking on these sorts of things now (using the normandy API)

https://github.com/gregglind/x-shield-inflight-dashboard

And for reference:
http://normandy.readthedocs.io/en/latest/user/filter_expressions.html#preference-filters

Comment 8

[Luke Crouch [:groovecoder]]

Note: S&I has started to recruit users into the study now. We should start to see their data in the dashboard:

https://sql.telemetry.mozilla.org/dashboard/shield-containers-executive-summary

Comment 9

[Matt Grimes [:Matt_G]]

This study shipped and has now ended. Closing this bug.