1426161 - Consider adding app.shield.optoutstudies.enabled pref to TelemetryEnvironment

Status: RESOLVED FIXED

(Firefox :: Normandy Client, enhancement, P3)

Description

[Chris H-C :chutten]

Near as I can tell we don't track whether the Study pref is enabled/disabled on a user's Firefox. We could add it to TelemetryEnvironment.jsm and then we'll get a session split whenever it changes.

This can allow us to measure how many users opt out of studies.

Comment 1

[Michael Cooper [:mythmon]]

Are there an privacy concerns that we need to work through for this? How are these prefs normally chosen?

Comment 2

[Chris H-C :chutten]

Data Collection Review[1] will cover these sorts of questions in more detail after there's a patch written and reviewed. But, in general, since it's a bool pref it shouldn't be too big of a deal.

These prefs[2] are generally chosen to be those that are infrequently changed and, when they are changed from their default, have some effect on how Firefox behaves. I think app.shield.optoutstudies.enabled counts.

[1]: https://wiki.mozilla.org/Firefox/Data_Collection
[2]: https://searchfox.org/mozilla-central/rev/97cb0aa64ae51adcabff76fb3b5eb18368f5f8ab/toolkit/components/telemetry/TelemetryEnvironment.jsm#185

Comment 3

[Michael Cooper [:mythmon]]

Attachment: Bug 1426161 - Add app.shield.optoutstudies.enabled pref to TelemetryEnvironment

Review commit: https://reviewboard.mozilla.org/r/216744/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/216744/

Comment 4

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 8946865 [details]
Bug 1426161 - Add app.shield.optoutstudies.enabled pref to TelemetryEnvironment

https://reviewboard.mozilla.org/r/216744/#review222574

This looks good to me.
This requires data collection review, make sure to get review by a peer:
https://wiki.mozilla.org/Firefox/Data_Collection

Comment 5

[Michael Cooper [:mythmon]]

Comment on attachment 8946865 [details]
Bug 1426161 - Add app.shield.optoutstudies.enabled pref to TelemetryEnvironment

# Request for data collection review form

1) What questions will you answer with this data?

What kind of users are turning off studies?

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

Can be used to determine the health of our study population. If a large portion of users opt-out of this feature, we have non-representative studies, or other issues.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

None. This is a very simple measure that can only be done one way.

4) Can current instrumentation answer these questions?

No.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the found on the Mozilla wiki.

Opt-out for shield add-on studies: level 1.

6) How long will this data be collected?  Choose one of the following:

I want to permanently monitor this data.

7) What populations will you measure?

All populations that use telemetry.

8) If this data collection is default on, what is the opt-out mechanism for users?

Turning off telemetry entirely.

9) Please provide a general description of how you will analyze this data.

Reading it from the telemetry main ping when relevant.

10) Where do you intend to share the results of your analysis?

It will be a part of the general telemetry pipeline.

Comment 6

[Robert (rrayborn, he/him)]

Comment on attachment 8946865 [details]
Bug 1426161 - Add app.shield.optoutstudies.enabled pref to TelemetryEnvironment

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, basic Telemetry Environment change

2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, Telemetry preferences

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Mythmon - documented in bug/commit references bug.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1

5) Is the data collection request for default-on or default-off?

default-on, with Telemetry

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No

7) Is the data collection covered by the existing Firefox privacy notice?

Yes, Telemetry policies

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

No, permanent

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bd31027a9006
Add app.shield.optoutstudies.enabled pref to TelemetryEnvironment r=gfritzsche

Comment 8

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/bd31027a9006

Comment 9

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Footnote: about 0.6% of DAU with telemetry enabled are opted out of experiments. https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/149830/command/149835