1559430 - [Shield] Add-On: JESTr Relaunch: Identifying Web Browsing Patterns in the Pioneer Population

Status: RESOLVED FIXED

(Shield :: Shield Study, task, P3)

Description

[Experimenter Bot]

JESTr Relaunch: Identifying Web Browsing Patterns in the Pioneer Population

This study aims to recreate the metrics in "Why Johnny Can’t Browse in Peace: On the Uniqueness of WebBrowsing History Patterns" (https://petsymposium.org/2012/papers/hotpets12-4-johnny.pdf) with the Pioneer population.

More information: https://experimenter.services.mozilla.com/experiments/jestr-relaunch-identifying-web-browsing-patterns-in-the-pioneer-population/

Comment 1

[fwollsen]

Attachment: jestr_pioneer_shield_study-1.2.0.zip

Attached v1.2.0 for test-signing

Comment 2

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 3

[fwollsen]

Attachment: jestr_pioneer_shield_study-1.2.1.zip

Attached v1.2.1 for signing

Comment 4

[fwollsen]

Attachment: jestr_pioneer_shield_study-1.2.2.zip

Attached v1.2.2 for signing

Comment 5

[Rehan Dalal (Inactive)]

Attachment: [e-mail]

Comment 6

[fwollsen]

Attachment: jestr_pioneer_shield_study-1.2.3.zip

Attached v1.2.3 for signing

Comment 7

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 8

[Cosmin Muntean [:cmuntean], Ecosystem QA]

We have finished testing the JESTr Relaunch: Identifying Web Browsing Patterns in the Pioneer Population experiment on Firefox 67.0.3 unbranded release build and Firefox 68.0b11 unbranded beta build.

QA’s recommendation: YELLOW - SHIP IT, CONDITIONALLY

Reasoning:

• All issues found are verified-fixed and no new issues were uncovered while testing the latest add-on version (1.2.3).
• Unfortunately, we haven't managed to test the enrollment part of the experiment add-on on Normandy stage server. This due to a Normandy issue #https://github.com/mozilla/delivery-console/issues/938), where we can no longer create recipes for add-on experiments. However, we’ve managed to verify this a few days ago, but with the 1.2.0 version of the experiment. Considering this, we cannot guarantee that the enrollment works as expected using the latest add-on version of the experiment, or that the Normandy production server is not affected by the same issue (#938).

Testing Summary:

• Test Suite: https://testrail.stage.mozaws.net/index.php?/plans/view/18278;
• Test Report of final run: https://testrail.stage.mozaws.net/index.php?/reports/view/1764;

Tested Platforms:

• Windows 10 x64;
• Windows 7 x64;
• macOS 10.14;
• Arch Linux 4.14.3 x64;
• Ubuntu 16.04 x64;

Tested Firefox versions:

• Unbranded Firefox 67.0.3 (EN-US);
• Dev Edition Firefox 67.0b19 (DE);
• Dev Edition Firefox 67.0b19 (FR);
• Unbranded Firefox 68.0b11 (En-US);
• Dev Edition Firefox 68.0b11 (ES-ES);

Comment 9

[Ilana]

Attachment: jestr_review.md

needed by: 07-09-2019

Comment 10

[Ilana]

Comment on attachment 9074855 [details]
jestr_review.md

[removed]

Comment 11

[Chris H-C :chutten]

Comment on attachment 9074855 [details]
jestr_review.md

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Unclear. FOLLOW UP: Please provide a link to public-facing documentation on the data that's being collected.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences. Also, the user may opt-out of the Pioneer cohort.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire after 2019-08-12.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 3, Web Activity.

    Is the data collection request for default-on or default-off?

Default on for all channels, for users within the Pioneer cohort alone. The mitigation that allows this Category 3 data to be collected default-on in release is that it is only being collected from the Pioneer cohort.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection will expire.

---
Result: datareview+ once we have public-facing data documentation.

Comment 12

[Ilana]

Public documentation available at https://github.com/mozilla/jestr-pioneer-shield-study/blob/master/docs/TELEMETRY.md
Does this provide everything you need?

Comment 13

[Chris H-C :chutten]

The section shield-study-addon pings, specific to THIS study. is sparse on details of what is being collected and why. (It reads as technical documentation instead of user-facing documentation).

It does meet the criteria "available publicly, complete and accurate", though so collection may proceed. I'm not 100% familiar with the onboarding done for Pioneer participants, but I'm guessing these collections wouldn't surprise participants, which is what the underlying Data Principle is for that requirement.

Comment 14

[Experimenter Bot]

    Experiment Type: Opt-Out experiment

    What are the branches of the experiment:

- Treatment pioneer 100%:

this study is observational, so no branches required
        

    What version and channel do you intend to ship to?

100% of Release Firefox 67.0

    Are there specific criteria for participants?

100% of the pioneer population restricted to versions 67 and 68, release channel.

    What is your intended go live date and how long will the experiment run?

Jul 15, 2019 - Aug 12, 2019 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

This study will gather the domains visited by pioneers over a period of 7 days. Using various heuristics, we will test the uniqueness of profiles in the sample (for example: restricting the set using top Alexa sites, top sites in the sample, and categories of visited sites). After a "slumber" period of 7 days, the study will collect 7 additional days of data to see if profiles can be correlated to the earlier examination period. If we see that this methodology does allow us to identify users, there are serious implications for private browsing users and those who clear history for anonymity. At that point, we would work on designing follow-up experiments that would aim to lower this identifiability (via url spoofing, wiping cookies midway through a session, etc). 

Using this data, we can also validate the accuracy of the head-list derived from the previous JESTr study, and thus have more insight into the stability of this data over time.

Because this is an observational study, there is no success metric that we are setting. In the case that follow-up studies as described above are not warranted, the findings are useful for baselining purposes, particularly with ETP rolling out in the near future.



Do you plan on surveying users at the end of the experiment? No

    Who is the owner of the data analysis for this experiment?

Dave Zeber, Sarah Bird, Ilana Segall

    Will this experiment require uplift?

False

    QA Status of your code:

https://github.com/mozilla/jestr-pioneer-shield-study/blob/master/docs/TESTPLAN.md

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/jestr-relaunch-identifying-web-browsing-patterns-in-the-pioneer-population/

Comment 15

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 16

[Remus Dranca [rdranca], Ecosystem QA]

We have finished testing the JESTr Relaunch: Identifying Web Browsing Patterns in the Pioneer Population experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• No new issues were found while re-running the test cases for the enrollment part of the experiment, on Normandy stage server. The user is successfully enrolled in the experiment and the telemetry pings are sent. Also, the experiment is displayed in the “about:studies” page and can be removed.
• However, we came upon an issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1564818) where opt-out experiments are displayed as active in "about:studies" page after they are removed, until the page is refreshed. We are not sure of the source of this issue but it seems to be affecting only new opt-out experiments. However, the issue does not affect the functionality or the end user in any way.

Testing Summary:

• Verified that the created recipe (https://delivery-console.stage.mozaws.net/recipe/740/) for the experiment works as expected on the Normandy stage server.

Tested Platforms:

• Windows 10 x64
• macOS 10.14
• Ubuntu 16.04 x64

Tested Firefox versions:

• Firefox Release 67.0.4
• Firefox Release 68