1463301 - [Shield] Opt-out Study: Firefox Monitor

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Peter Dolanjski [:pdol]]

Basic description of experiment: 
Since we want to see which design will have the best outcomes with respect to subscriptions, scans and trustworthiness, we will test various approaches against each other

What are the branches of the study?
UX Spec of variations: https://mozilla.invisionapp.com/share/KEIV66OTS23#/screens/297625868 
Branch 1: Site specific with [Go to Fx Monitor] and [Dismiss] buttons
Account compromise warning shown on first site visited that is known to have been breached.
User can click through to Firefox Monitor webpage or Dismiss
Clicking Go to Firefox Monitor will go to a landing page describing that we’re testing the functionality and thank you for your interest.

Branch 2: Site specific with email field and [Scan] and [Dismiss] buttons
Same as branch 1, except user is asked for their email upfront to start scan
User can click through to Firefox Monitor scan results or Dismiss
Clicking scan will go to a landing page describing that we’re testing the functionality and thank you for your interest.

Branch 3: As per 2) but with option to subscribe
Same as branch 2, except that a subscribe box is selected by default (meaning that user will scan and subscribe in one step)
User can click through to Firefox Monitor scan results/subscribe or Dismiss
Clicking scan will go to a landing page describing that we’re testing the functionality and thank you for your interest.

Branch 4: Non-site specific with [Go to Fx Monitor] and [Dismiss] buttons
Account compromise warning shown on next about:newtab load after 48 hours of getting study
User can click through to Firefox Monitor webpage or Dismiss
Clicking Go to Firefox Monitor will go to a landing page describing that we’re testing the functionality and thank you for your interest.

Branch 5: As per 2) but with some information about the site breach
Same as branch 2, except detailed information presented about website breach being described
User can click through to Firefox Monitor scan results or Dismiss
Clicking scan will go to a landing page describing that we’re testing the functionality and thank you for your interest.

All branches: 
If user Dismisses, they will be asked why they dismissed the offer for a scan
What percentage of users do you want in each branch? 20% of users in each branch, target enrollment = 2500 per branch (who actually see the initial doorhanger - we may need to boost enrollment to hit this number in a week)

What Channels and locales do you intend to ship to? 
Release 60
En locales

What is your intended go live date and how long will the study run? 
June 4th
Study will be live for 1 week

Are there specific criteria for participants? 
Two targets:
New users
Existing users

What is the main effect you are looking for and what data will you use to make these decisions? 
Breach scans
% of users that click through to actually do a scan
% of users that enter an email address to see if they’ve been in a breach

Subscriptions
% of users that attempt to subscribe to future notifications of breaches (note that depending on the branch this takes different forms - subscribe is opt-out in some cases and requires a click on a specific button in other cases)

User reported trustworthiness (to be measured via survey)

We will ship the version that scores the best based on a combination of the above factors

Who is the owner of the data analysis for this study? TBD

Who will have access to the data? Others with data access

Do you plan on surveying users at the end of the study? Yes

User facing title of the experiment: Firefox Monitor

User facing description of the experiment: Firefox Monitor is a service that allows users to check if their email address has been in found in known data breaches on third party websites.

Code Review performed by: Johann Hoffman  

QA Status of your add-on: To be requested.

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
Original security advisor shield study: https://docs.google.com/document/d/1jchhTRx0sRSsAK7MsHdSNvQEYo1veXTv8dJc7dLLry4/edit

Comment 1

[Peter Dolanjski [:pdol]]

NI's shield owner, mgrimes

Comment 2

[Peter Dolanjski [:pdol]]

Saptarshi, can you look at the data being collected in this study to validate permissibility of collected data fields?

Comment 3

[Peter Dolanjski [:pdol]]

NI'ing peer for review, Johann

Comment 4

[Peter Dolanjski [:pdol]]

Full design specs: https://e-pang.github.io/breach-alerts/

Comment 5

[Johann Hofmann [:johannh]]

Nihanth or Luke, it would be ideal if I had a place to leave review comments, could you make a pull request onto an empty commit, (like this: https://github.com/jonathanKingston/http-dns/pull/4) and add it to this bug? You can then put the regular review?johannh flag on it.

Thank you!

Comment 6

[Luke Crouch [:groovecoder]]

You can use the add-on repo if you like:

https://github.com/mozilla/blurts-addon

If you just want to make some general comments, you could file an issue. If you want to make some line-by-line comments, you could click to the last commit that touched a file and leave the comments on the specific commits.

Comment 7

["Saptarshi Guha[:joy]"]

(In reply to Peter Dolanjski [:pdol] from comment #2)
> Saptarshi, can you look at the data being collected in this study to
> validate permissibility of collected data fields?

The data looks great, the email address will be sent to the service and not to us?
That is all we collect is 

did the user click through for a scan?
did the user enter an email address to see if they've been in a breach?
did the user subscribe for future notifications

Note, in the above, we don't collect the email address. If this is correct, then r+ from me.

Comment 8

[Luke Crouch [:groovecoder]]

We WILL receive the email address. We will hash it and send only the 6-character prefix of the hash to the service. We may or may not store the email address. (Though it will be stored in our operational log data regardless)

Comment 9

["Saptarshi Guha[:joy]"]

(In reply to Luke Crouch [:groovecoder] from comment #8)
> We WILL receive the email address. We will hash it and send only the
> 6-character prefix of the hash to the service. We may or may not store the
> email address. (Though it will be stored in our operational log data
> regardless)

Following off from here: https://wiki.mozilla.org/Firefox/Data_Collection
assuming the following holds : " provided there is (i) advance user notice (ii) consent and (iii) an opt-out."
i guess its okay. Though how does the user opt out when the email has been sent?

I believe we're giving a lot of opportunity for user consent and advanced user notice (given the hangars demonstrated
) so I'm good with this. Also, though the experiment is presumably opt out, submitting the email address is very clearly opt in/ user choice driven, so again, i believe this data collection is okay.

Comment 10

[Peter Dolanjski [:pdol]]

(In reply to "Saptarshi Guha[:joy]" from comment #9)
> Following off from here: https://wiki.mozilla.org/Firefox/Data_Collection
> assuming the following holds : " provided there is (i) advance user notice
> (ii) consent and (iii) an opt-out."
> i guess its okay. Though how does the user opt out when the email has been
> sent?

In this experiment, we are not keeping the email address beyond use for the scan itself.

Comment 11

[Peter Dolanjski [:pdol]]

Note: There have been some updates to the PHD including the target date: June 18th, locale=en-US, platform=Windows.

Comment 12

[Peter Dolanjski [:pdol]]

Note: the recruitment size for each branch has increased to 50,000.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Do you have a PHD link handy for this study?

Comment 14

[Peter Dolanjski [:pdol]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #13)
> Do you have a PHD link handy for this study?

Yes, sorry, should have linked to it in the original submission:
https://docs.google.com/document/d/1pCBZqux0J_49xEof4wsIJ0gSldKvvMBY5Ift00FrV00/edit?pli=1

Comment 15

[Liz Henry (:lizzard) (relman/hg->git project)]

Treating this as a feature for 62.

Comment 16

[Ilana]

Science Review: R+

Comment 17

[Johann Hofmann [:johannh]]

Attachment: Shield Study Code Review

Shield study code as a PR here:

https://github.com/mozilla/blurts-addon/pull/66

Comment 18

[Johann Hofmann [:johannh]]

Nihanth, can you please do a simple try run with the add-on included in the tree?

An example is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2cee45c3edc7e47a4cba53ad5f898ebe3142ff65

Comment 19

[Johann Hofmann [:johannh]]

Comment on attachment 8988430 [details] [diff] [review]
Shield Study Code Review

Review of attachment 8988430 [details] [diff] [review]:
-----------------------------------------------------------------

::: src/background.js
@@ +62,5 @@
> +    ],
> +    endings: {
> +      "user-disable": {
> +        baseUrls: [
> +          "https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=user-disable",

(Repeating this in Bugzilla to be sure)

Please remember to update these before the study goes live.

Comment 20

[Johann Hofmann [:johannh]]

Comment on attachment 8988430 [details] [diff] [review]
Shield Study Code Review

Review of attachment 8988430 [details] [diff] [review]:
-----------------------------------------------------------------

r=me for this as a Shield study for a test audience.

As mentioned in previous conversations, this has a bunch of polish/usability issues (flickering, no rediscoverability, missing icon for the doorhanger) that in my view do not qualify this for landing in front of a larger audience than a test sample of our user population.

Thank you!

Comment 21

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: firefox_monitor-1.0.zip

This is a build of the latest add-on, for signing.

Comment 22

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 23

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: firefox_monitor-1.0.zip

Here's a new build, with a QA issue fixed, for signing. Thanks!

Comment 24

[Michael Cooper [:mythmon]]

I would strongly recommend bumping the version number, to make this distinct from the previous signed version.

Comment 25

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: firefox_monitor-1.1.zip

Version number bumped (in  manifest.json as well) to 1.1. Thanks!

Comment 26

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 27

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: firefox_monitor-1.2.zip

Michael, sorry for the repeated requests, but I got some feedback about the telemetry event naming and did some refactoring to better support the people who will be looking at the data later. This should be the last change. I've bumped the version number to 1.2.

Thanks!

Comment 28

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 29

[Carmen Fat [:cfat] - Ecosystem QA]

Breach Alerts - Firefox Monitor
Targeted: Firefox Release 61.0

We have finished testing the Breach Alerts - Firefox Monitor experiment. All major and blocker issues have been fixed and verified. We consider that all other remaining issues are not blockers for the experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:
- "Disable/Remove" issue (https://github.com/mozilla/blurts-addon/issues/34) was fixed and verified and the PBM metrics gathering concern has been addressed. Therefore, we consider that the experiment is good to be launched. We also spent more time doing regression testing and we didn't find new issues.

Testing Summary:
- Full Functional test suite: TestRail - https://tinyurl.com/ycppfnxf
- Verified that the Telemetry pings are correctly sent;
- Performed regression testing after reported issues were fixed.

Tested Platforms:
- Windows 10 x64

Tested Firefox versions:
- Firefox Release 61.0

Comment 30

[MarniePW [:marnie]]

Alright RyanVM, looks like you're the final NI for this experiment, from a Shield perspective.

Comment 31

[Ryan VanderMeulen [:RyanVM]]

Looks good to me, pending executive signoff per the PHD. Signing off for RelMan.

Comment 32

[Matt Grimes [:Matt_G]]

I'm signing off as well and flagging Nick for VP approval. Thanks everyone for all the hard work here.

Comment 33

[Nick Nguyen [:osunick]]

Approved.

Comment 34

[Josh Gaunt [:jgaunt]]

Monitor is live.

Comment 35

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Five variants packaged separately for signing

Michael, I've made separate builds for each of the five UI variations to support our user-testing effort that will be run parallel to the shield study. Could you please sign these? I've put all five builds in one zip file for conveniently attaching here.

Comment 36

[Michael Cooper [:mythmon]]

Each of the five add-ons here has the same add-on ID and version. That will make them harder to work with in Normandy, and harder to distinguish come analysis time. Different variations should have different versions or add-on IDs.

Additionally, it is generally a lot easier if the add-ons are attached directly to the bug, since the signing tool can work directly with Bugzilla. I can work with a zip bundle, but it will take longer.

Comment 37

[(inactive) Nihanth Subramanya [:nhnt11]]

(In reply to Michael Cooper [:mythmon] from comment #36)
> Each of the five add-ons here has the same add-on ID and version. That will
> make them harder to work with in Normandy, and harder to distinguish come
> analysis time. Different variations should have different versions or add-on
> IDs.
> 
> Additionally, it is generally a lot easier if the add-ons are attached
> directly to the bug, since the signing tool can work directly with Bugzilla.
> I can work with a zip bundle, but it will take longer.

These are not for shield. They do not interact with shield. background.js is only one line.

These are for a separate user-testing study that will involve instructing users to install the addon in realtime. I'm only requesting for them to be signed so that users won't have to install an unbranded build of Firefox. I can upload 5 attachments, no big deal, but I don't want to cause confusion on this bug. I'll file another bug and ni? you there.

Comment 38

[Matt Grimes [:Matt_G]]

We've ended version 1 of the study. Let's open a new bug if we decided to ship a v2. I'll leave this open until we have some analysis to share.

Comment 39

[Ryan VanderMeulen [:RyanVM]]

Matt, do you happen to know what the status on this analysis is?

Comment 40

[Matt Grimes [:Matt_G]]

Analysis is complete. I'll let Cindy recap the results and Product decisions made as a result. Please close this bug once that is done. Thanks!

Comment 41

[chsiang]

The experiment was concluded and the Monitor team will ship the site specific doorhanger with detailed breach information and modified two things. 
1. Took out the search box based on the feedback from the security review
2. UI treatment to increase the trustworthiness of the doorhanger - to address user concerns found in survey and usertesting analysis