1567581 - Add fission pref in the telemetry environment

Status: RESOLVED FIXED

(Toolkit :: Telemetry, task, P1)

Description

[Neha Kochar [:neha]]

We would like to add the fission.autostart pref to the telemetry environment so we can get this information from telemetry reports.

Comment 1

[Abdoulaye O. LY]

Attachment: Bug 1567581 - Add fission pref in the telemetry environment. r?NeilDeakin

Comment 2

[Jan-Erik Rediger [:janerik]]

Adding P1 as its actively being worked on (and indeed even has a patch already).

As another point this requires Data Collection Review before it lands.

Comment 3

[Abdoulaye O. LY]

Attachment: data-review-request.md

Comment 4

[Chris H-C :chutten]

Comment on attachment 9083466 [details]
data-review-request.md

The method in Q9 doesn't allow analysis of Telemetry Environment. Did you instead intend to collect this as a boolean Scalar? Or do you intend to perform some more sophisticated analysis either within your team or by requesting help from Data Science?

Comment 5

[Abdoulaye O. LY]

We would like to collect this as a boolean Scalar.

Comment 6

[Abdoulaye O. LY]

Comment on attachment 9083466 [details]
data-review-request.md

># Request for data collection review form
>
>**All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.**
>
>1) What questions will you answer with this data?
>Whether Fission is enabled or not.
>
>2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
>
>* Provide information essential for advancing Fission development.
>
>* Distinguish difference in performance and/or crashes or functionality issues with and without fission.
> 
>3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
>No alternative method was considered.
>
>4) Can current instrumentation answer these questions?
>No, as far as I know.
>
>5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.   
>
>**Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.**
>
><table>
>  <tr>
>    <td>Measurement Description</td>
>    <td>Data Collection Category</td>
>    <td>Tracking Bug #</td>
>  </tr>
>  <tr>
>    <td>fission.autostart pref status</td>
>    <td>Technical data</td>
>    <td>1567581</td>
>  </tr>
></table>
>
>
>6) How long will this data be collected?  Choose one of the following:
>
>* This is scoped to a time-limited experiment/project. End of experiment/project is unknown.
>
>7) What populations will you measure?
>
>* Which release channels?
>All release channels.
>
>* Which countries?
>All countries.
>
>* Which locales?
>All locales.
>
>* Any other filters?  Please describe in detail below.
>
>8) If this data collection is default on, what is the opt-out mechanism for users?
>Disabling Telemetry.
>
>9) Please provide a general description of how you will analyze this data.
>We intend to collect this as a boolean Scalar.
>
>10) Where do you intend to share the results of your analysis?
>It will be shared in telemetry.mozilla.org.
>
>11) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:
>No.
>
>* Are you using that on the Mozilla backend? Or going directly to the third-party?
>Not applicable.
>

Comment 7

[Chris H-C :chutten]

Will you be updating the patch to use a Scalar? Will it satisfy your data needs to have it only be present in the first "main" ping of a session?

You might wish to also add it as a Crash Annotation to identify crashes. See CrashAnnotations.yaml.

Comment 8

[Abdoulaye O. LY]

Comment on attachment 9083466 [details]
data-review-request.md

># Request for data collection review form
>
>**All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.**
>
>1) What questions will you answer with this data?
>Whether Fission is enabled or not.
>
>2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
>
>* Provide information essential for advancing Fission development.
>
>* Distinguish difference in performance and/or crashes or functionality issues with and without fission.
> 
>3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
>No alternative method was considered.
>
>4) Can current instrumentation answer these questions?
>No, as far as I know.
>
>5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.   
>
>**Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.**
>
><table>
>  <tr>
>    <td>Measurement Description</td>
>    <td>Data Collection Category</td>
>    <td>Tracking Bug #</td>
>  </tr>
>  <tr>
>    <td>fission.autostart pref status</td>
>    <td>Technical data</td>
>    <td>1567581</td>
>  </tr>
></table>
>
>
>6) How long will this data be collected?  Choose one of the following:
>
>* This is scoped to a time-limited experiment/project. End of experiment/project is unknown.
>
>7) What populations will you measure?
>
>* Which release channels?
>All release channels.
>
>* Which countries?
>All countries.
>
>* Which locales?
>All locales.
>
>* Any other filters?  Please describe in detail below.
>
>8) If this data collection is default on, what is the opt-out mechanism for users?
>Disabling Telemetry.
>
>9) Please provide a general description of how you will analyze this data.
>We intent to create custom analysises to compare with or without fission.
>
>10) Where do you intend to share the results of your analysis?
>It will be shared in telemetry.mozilla.org.
>
>11) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:
>No.
>
>* Are you using that on the Mozilla backend? Or going directly to the third-party?
>Not applicable.
>

Comment 9

[Abdoulaye O. LY]

After some discussion on #fission slack group, what we actually want is to "tag telemetry sessions/pings with whether or not fission is enabled so
That way we can do [sophisticated] analysis to see how certain metrics are different with or without Fission".

With bug 1560977, crash reports are already being tagged with fission when the pref is enabled.

The plan is to also talk to data science about this very soon.

Sorry for all the confusion here.

Comment 10

[Chris H-C :chutten]

Comment on attachment 9083466 [details]
data-review-request.md

Okay, if this needs to be part of the TelemetryEnvironment then it will need some manual documentation in `toolkit/components/telemetry/docs/telemetry/data/environment.rst` (( The first question I need to answer in the data review response is "where is the public documentation", and unfortunately prefs in the Environment don't have descriptions, owners, expiry mechanisms, or tools for turning them into docs that the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/) can publish.  ))

(( Sorry for the delay in getting back to you. I'll put you on the top of my list for when this comes back around. Everything else in the request looks great. ))

Comment 11

[Abdoulaye O. LY]

No problem. Please let me know if what I added to the documentation is all we need.

Comment 12

[Chris H-C :chutten]

(In reply to Abdoulaye O. LY from comment #11)

No problem. Please let me know if what I added to the documentation is all we need.

Looks good to me. I'll go ahead with the review now.

Comment 13

[Chris H-C :chutten]

Comment on attachment 9083466 [details]
data-review-request.md

PRELIMINARY NOTE:

The answer to 9 "How you will analyze this data" is incomplete. From discussion, I know the plan is to use this in custom analyses to mark pings from fission-enabled clients.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is part of the Telemetry Environment so is documented in the [Telemetry Environment documentation](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/environment.html).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :neha is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

ni?neha to confirm that you're willing to oversee this data collection over its lifetime.

Comment 14

[Neha Kochar [:neha]]

Yes, confirming. Thanks!

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:ablayelyfondou, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 16

[Abdoulaye O. LY]

Neil: can we land this now?

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/adae662e3c6c
Add fission pref in the telemetry environment. r=NeilDeakin

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/adae662e3c6c

Comment 19

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Using the RECORD_PREF_VALUE mechanism in TelemetryEnvironment.jsm means that a value for this pref will only be recorded in a ping if it is set on the user branch. This will do what we expect now (i.e. report "true" or be missing), since Fission is off by default—but, later, if this pref is enabled on the default branch, be aware that users on those builds with Fission enabled will not send a value for this field.