1377563 - [Shield Opt-in] Privacy prefs breakage study

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Comment 1

[Luke Crouch [:groovecoder]]

pdolanjski: According to the study doc [1], you need to assign this bug to yourself.
glind: this is the privacy breakage study I showed in SF; we would like to ship ASAP
rweiss: The metrics doc is in the repo [2]


> Basic description of experiment/add-on and link to code.

The add-on puts users into a number of pref variations and allows the user to report broken web pages.
https://github.com/mozilla/shield-study-privacy

> What are the branches of the study? You can have multiple branches, but every study must include a control.

All branches see the breakage-reporting UI
* Control: No pref changes
* network.cookies.thirdparty.sessionOnly: true
* network.cookie.cookieBehavior: 1 (no 3rd-party cookies)
* network.cookie.cookieBehavior: 3 (3rd-party cookies only from visited)
* privacy.trackingprotection.enabled: true
* network.http.referer.trimmingPolicy: 2 (only send origins [not paths] to 3rd parties in the referer header)
* privacy.resistFingerprinting: true

> What percentage of users do you want in each branch? We can do branches of uneven proportions. Example: 20% overall with 5% in control and 15% in treatment 1. 

event split between branches: 14.28% each

> What Channels/locales do you intend to ship to?

Release en-US

> What is your intended go live date and how long will the study run? All experiments have expiration dates. 

Would like to go live July 17

> Are there specific criteria for participants? We can include/exclude based on anything available in telemetry.

The study eligibility code requires that the users have the default values for all the prefs under study

> What is the main effect you are looking for?  

The preferences we're studying can improve users' privacy, but they may break sites. We want to learn how often users see breakage, which sites are broken, and how they are broken, so we can decide if we can make some prefs defaults, or at least make them easily accessible in Firefox. We may also learn how to mitigate breakage.

> Who is the owner of the data analysis for this study?

Luke Crouch is working with Ryan Harter on ETL, and will work with Ilana Segall on the (SQL) analysis.

> Do you plan on surveying users at the end of the study? If so, please link to the survey.

Yes. Will add link in a future comment.

> Link to any relevant google docs / Drive files that describe the project.

> Paste a real telemetry packet generated by your addon for the Data Steward to review

>{
>  "branch": "thirdPartyCookiesOnlyFromVisited",
>  "event": "notes",
>  "originDomain": "redditmedia.com",
>  "breakage": "images",
>  "notes": "comment",
>}

>Links to prior art if it exists:
>If there is prior art (e.g. testpilot, usertesting.com, field research, etc.), review it in the bug (and link as necessary)
>If there are previous results (particularly if they allow for the removal of experimental branches), review them here.

[1] https://docs.google.com/document/d/1j_wUoPbDT8b56-rQAirsBev8LMem9KvnF1odDVqPGo4/edit#heading=h.jwelof12r1w5
[2] https://github.com/mozilla/shield-study-privacy/blob/master/docs/metrics.md

Comment 2

[Luke Crouch [:groovecoder]]

Attachment: shield-study-privacy.xpi

Initial .xpi for signing & deploying via Shield

May need to update this with more updates if we decide to add another variation.

Comment 3

[François Marier [:francois]]

(In reply to Luke Crouch [:groovecoder] from comment #1)
> * network.http.referer.trimmingPolicy: 2 (only send origins [not paths] to
> 3rd parties in the referer header)

That's the wrong pref. The correct one is network.http.referer.XOriginTrimmingPolicy

Sorry I should have caught this earlier.

Also, are you going to include "privacy.firstparty.isolate: true" like we talked about in SF? There seems to be a lot of interest in this Tor feature and it would be great to be able to compare it to the other variations.

Comment 4

[Luke Crouch [:groovecoder]]

Nice catch, thanks.

I'll add first-party isolation ...

glind - do we have ways of filling up our variations with more Test Subjects™ if our variations are not filling up?

Comment 5

[Luke Crouch [:groovecoder]]

Attachment: shield-study-privacy.xpi

Updated .xpi with fixed origin trimming policy pref, and with 2 variations to study first-party isolation.

Comment 6

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Answering all questions.

tl;dr,  SCIENCE: Good To Go.  
Claim:  DATA perspective is OKAY, but cc: rweiss or [steward] to confirm.

1.  NOT doing a technical review.
2.  We are NOT ABLE to over-recruit for specific branches, but CAN open the over recruitment wider.
3.  


DATA COLLECTION (@rweiss) - Opt-in with Type 3 send unencrypted to Mozilla via UT.
1.  OPT IN shield study.
2.  "Broken website" Metrics format at: https://github.com/mozilla/shield-study-privacy/blob/master/docs/metrics.md
3.  Consent page mentions that high-level domain is collected.

```
  <li>We collect given information regarding the breakage of reported webpages, including if there is a problem, what is the problem, and any additional comments.</li>
<li>Along with given information, we also collect the high-level domain name of each page reported.</li>
```

(https://github.com/mozilla/addons-server/blob/8252e99d35dd1a3b709480ff46419e4328e16473/src/olympia/pages/templates/pages/shield_study_14.html)

4.  AFTER USER ACTION (click on an interface elements),  eTLD's are reported via UT (Mozilla storage).



ALL LOOKS GOOD.

Comment 7

[[e-mail]]

r+

Comment 8

[Kamil Jozwiak [:kjozwiak]]

As per bug#1382270comment#7, please ensure that the Shield/Normandy server adds all the appropriate rules that will only recruit the correct users/configurations as the XPI doesn't do these checks on install.

General Test Cases:
===================

* ensured that whenever a user was placed into a branch that changed a pref, the "Orientation" tab appeared correctly
* ensured that whenever a user was placed into a branch that changed a pref, the "report an issue" icon was placed under the toolbar
* ensured that disabling/removing the shield study via about:addons correctly reverts the changed pref and removes the "report an issue" icon
* ensured that disabling/removing the shield study via the "report an issue" icon correctly reverts the changed pref and removes the icon
* ensured that once the shield study expires after 28 days, the prefs are reverted and the "report an issue" icon is removed
* ensured that once the shield study is removed/expired, it's removed from about:addons -> Experiments
* ensured that once the shield study is expired/removed, she survey is opened within a new window
* ensured that once the shield study is expired/removed, the "Thank You" page is opened within a new tab
* ensured that selecting "Make Firefox your Default Browser" under the "Thank You" page loads the correct SUMO article
* ensured that you can report issues using the "report an issue" icon under the toolbar

Branches Tested:

* ensured that "branch": "originOnlyRefererToThirdParties" correctly changed network.http.referer.XOriginTrimmingPolicy;2
** [e-mail];originOnlyRefererToThirdParties
* ensured that "branch": "sessionOnlyThirdPartyCookies" correctly changed network.cookies.thirdparty.sessionOnly;true
** [e-mail];sessionOnlyThirdPartyCookies
* ensured that "branch": "resistFingerprinting" correctly changed privacy.resistFingerprinting;true
** [e-mail];resistFingerprinting
* ensured that "branch": "thirdPartyCookiesOnlyFromVisited" correctly changed network.cookie.cookieBehavior;3
** [e-mail];thirdPartyCookiesOnlyFromVisited
* ensured that "branch": "firstPartyIsolation" correctly changed privacy.firstparty.isolate;true
** [e-mail];firstPartyIsolation
* ensured that "branch": "trackingProtection" correctly changed privacy.trackingprotection.enabled;true
** [e-mail];trackingProtection
* ensured that "branch": "firstPartyIsolationOpenerAccess" correctly changed both privacy.firstparty.isolate;true and privacy.firstparty.isolate.restrict_opener_access;false
** [e-mail];firstPartyIsolationOpenerAccess
* ensured that "branch": "noThirdPartyCookies" correctly changed network.cookie.cookieBehavior;1
** [e-mail];noThirdPartyCookies
* ensured that "branch": "control" didn't change any of the prefs under about:config
** [e-mail];control

Critical Issues:
================

The following three prefs are not being reverted once the shield study as been removed. I'm assuming it's because these were added/edited after the original implementation and might not have been included/changed in the cleanup code.

* network.http.referer.XOriginTrimmingPolicy;2 not being reverted
* privacy.firstparty.isolate;true not being reverted
* privacy.firstparty.isolate.restrict_opener_access;false not being reverted

Possible Issue:
===============

The "branch": "sessionOnlyThirdPartyCookies" currently changes network.cookies.thirdparty.sessionOnly;true but I've noticed that there's also network.cookie.thirdparty.sessionOnly under about:config. Was this a typo? Should it change the pref that includes "cookie" rather than creating a new pref that uses "cookies" instead? Or is this the intended behaviour?

Suggestions/Improvements:
=========================

* Clicking on "This Page Works Well" doesn't seem like it's doing anything. Maybe we can add some type of user confirmation that an action/submission was completed?
* When the user is put into "branch": "control, the "report an issue" button still appears under the toolbar. Should we be adding this button even though there hasn't been any changes to the prefs? It might confuse some users. This also includes the survey and "Thank You" page. Should we be showing those if users have been placed into the control branch?

Comment 9

[Luke Crouch [:groovecoder]]

Thanks for the thorough review! I fixed the critical and possible issues here:

https://github.com/mozilla/shield-study-privacy/pull/55

And I made a new release and got it signed here:

https://github.com/mozilla/shield-study-privacy/releases/tag/0.0.3

Can you QA that version of the .xpi to check those critical issues again?

The other suggestions are good, but let's go ahead and ship without them.

(In the 2nd case, we *do* want to show the "report an issue" button so we can measure how many users report issues even though we haven't changed anything! Ideally, we can subtract that amount of breakage out of the other variations to eliminate the "noise". I.e., some users will report problems just because we gave them a button!)

Comment 10

[Kamil Jozwiak [:kjozwiak]]

The following prefs are now correctly being removed once the shield study has either expired or has been removed:

* privacy.firstparty.isolate;true - PASSED
* privacy.firstparty.isolate.restrict_opener_access;false - PASSED

However, the following pref is still not being removed/reverted once the shield has been removed/expired:

* network.http.referer.XOriginTrimmingPolicy;2 - FAILED (Reproduced on Win/MacOS)

I also ensured that network.cookies.thirdparty.sessionOnly was removed and network.cookie.thirdparty.sessionOnly;true was being used instead.

Comment 11

[Luke Crouch [:groovecoder]]

Argh! Forgot that one. Made and signed a 0.0.4 .xpi here:

https://github.com/mozilla/shield-study-privacy/releases/tag/0.0.4

(Hopefully) one last run-thru?

Comment 12

[Kamil Jozwiak [:kjozwiak]]

It looks like we're good! I went through the same test cases as mentioned in comment#8 without any issues. Results:

* privacy.firstparty.isolate;true - PASSED
* privacy.firstparty.isolate.restrict_opener_access;false - PASSED
* network.http.referer.XOriginTrimmingPolicy;2 - PASSED

I also ensured that network.cookies.thirdparty.sessionOnly was renamed to network.cookie.thirdparty.sessionOnly.

Comment 13

[Ioana Chiorean]

[Tracking Requested - why for this release]:

[Tracking Requested - why for this release]:

Is this something we can test atm or only after the go live moment (like Bug 1393628)?

Comment 14

[François Marier [:francois]]

I'm pretty sure this study is all done.

Luke, is there anything else that needs to be done (e.g. are we going to make the results public?) or can we close this bug?

Comment 15

[Luke Crouch [:groovecoder]]

Yes, we've presented the results and the deck is public. We could write a blog post about them too, but that can be a follow-up bug.

Comment 16

[Luke Crouch [:groovecoder]]

Deck is here: https://docs.google.com/presentation/d/1OVtXAnyeBLX2N1yyZoTMP9AV_6HnI3mnXwIFlOL7yOA/edit

Comment 17

[Peter Dolanjski [:pdol]]

For posterity, here is the redash dashboard: https://sql.telemetry.mozilla.org/dashboard/shield-study-improve-privacy-settings