Add telemetry to improve the understanding of current "about:config" usage patterns

RESOLVED FIXED in Firefox 64

Status

()

enhancement
P1
normal
RESOLVED FIXED
11 months ago
10 months ago

People

(Reporter: Paolo, Assigned: lschwalf)

Tracking

(Blocks 1 bug)

Trunk
mozilla64
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox64 fixed)

Details

Attachments

(2 attachments)

We can add a few histograms to understand how users of Nightly, and later Beta, interact with the current "about:config" user interface. Examples include whether settings are sorted, or whether new settings are added. We will not collect any data about the actual setting names.
Attachment #9012393 - Flags: review?(chutten)
Comment on attachment 9012393 [details]
Data Review Request for Bug 1493445

>
># Request for data collection review form
>
>**All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.**
>
>1) What questions will you answer with this data?
>
>  Which features are used on the about:config page most frequently?
>
>2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
>
>* Determine which features to prioritize as a part of the update being made to about:config UI [Bug 1493439](https://bugzilla.mozilla.org/show_bug.cgi?id=1493439). This is to help us organize the updated UI in a user friendly manner.
>
>3) What alternative methods did you consider to answer these questions?     Why were they not sufficient?
>
>  I considered conducting user surveys; however, as a new contributor to Firefox I don't know how I would conduct them and they would not give as detailed information about which individual features on the about:config page are used.
>
>4) Can current instrumentation answer these questions?
>
>  It does not as far as I am aware.
>
>5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.   
>
>**Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.**
>
><table>
>  <tr>
>    <td>Measurement Description</td>
>    <td>Data Collection Category</td>
>    <td>Tracking Bug #</td>
>  </tr>
>  <tr>
>    <td>Search Bar Used</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Sorted by Preference Name</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Sorted by Status</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Sorted by Type</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Sorted by Value</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Context Menu Modify Value Clicked</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Context Menu Copy Clicked</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Context Menu Copy Name Clicked</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Context Menu Copy Value Clicked</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Context Menu New Preference Created</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
>  <tr>
>    <td>Context Menu Reset Preference</td>
>    <td>2</td>
>    <td>1493445</td>
>  </tr>
></table>
>
>
>6) How long will this data be collected?  Choose one of the following:
>
>* I want this data to be collected for 6 months initially (potentially renewable).
>
>7) What populations will you measure?
>
>* Which release channels? Nightly, and later Beta
>
>* Which countries? Global
>
>* Which locales? All locales
>
>* Any other filters?  Please describe in detail below.
>
>8) If this data collection is default on, what is the opt-out mechanism for users?
>
>  I'm not aware of what the opt-out mechanisms are for data collection in Firefox
>
>9) Please provide a general description of how you will analyze this data.
>
>  Data will be analyzed for frequency and counts compared. Higher counts will be prioritized more in the about:config redesign.
>
>10) Where do you intend to share the results of your analysis?
>
>  As a part of the about:config redesign. Besides that I'm not sure.
Comment on attachment 9012393 [details]
Data Review Request for Bug 1493445

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, standard Telemetry mechanisms apply.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, standard Telemetry mechanisms apply. (the Data Choices checkbox in Firefox Preferences)

    If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, expiring data collection.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, user interaction.

    Is the data collection request for default-on or default-off?

Default-on for pre-release channels only.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

    Is the data collection covered by the existing Firefox privacy notice? 

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :lschwalf will be responsible for removing or renewing the data collection before it expires (or nominating someone to do it in their stead).

---
Result: datareview+
Attachment #9012393 - Flags: review?(chutten) → review+
Consider asking a Telemetry Peer to review your data collection patch. They might have input on how to best collect the least amount of data to answer your questions.
For "Search Bar Used", I'd be interested in distinguishing between normal search and regular expressions search. For these kind of changes, do we need to submit a new data review after the final patch, or is a Telemetry peer review on the patch sufficient?
Flags: needinfo?(chutten)
If the original collection hasn't yet landed in-tree you can consider (since you asked) the data collection review response to include categorizing the "search bar used" from being a single count to being a categorical measure.

If it has already landed, please attach a fresh review form to the bug landing the collection and request data collection review for the changed data collection.
Flags: needinfo?(chutten)
The original collection code has not landed yet. I have not gotten around to implementing it in-tree yet.
:paolo here is the current WIP diff. There are a few items in the JSON object that I am not sure about: For instance "expires_in_version" right now is unknown, so I left in a value I found in other JSON objects.

diff --git a/toolkit/components/telemetry/Histograms.json b/toolkit/components/telemetry/Histograms.json
--- a/toolkit/components/telemetry/Histograms.json
+++ b/toolkit/components/telemetry/Histograms.json
@@ -53,6 +53,15 @@
     "high": 60000,
     "n_buckets": 50
   },
+   "ABOUT_CONFIG_FEATURES_USAGE_COUNT": {
+    "record_in_processes": ["main"],
+    "expires_in_version": "never",
+    "kind": "enumerated",
+    "bug_numbers": [1493445],
+    "n_values": 13,
+    "description": "Record number of times particular features on about:config are used",
+    "alert_emails": ["pamadini@mozilla.com"]
+  },
   "ADDON_MANAGER_UPGRADE_UI_SHOWN": {
     "record_in_processes": ["main", "content"],
     "expires_in_version": "53",
(In reply to Luke Schwalfenberg [:lschwalf] from comment #8)
> :paolo here is the current WIP diff. There are a few items in the JSON
> object that I am not sure about: For instance "expires_in_version" right now
> is unknown, so I left in a value I found in other JSON objects.

Thank you for the update! For the expiration, I think we can do 67. According to the release calendar (<https://wiki.mozilla.org/Release_Management/Calendar>), this should give us data for Nightly (mozilla-central) until January 28, and in Beta until March 18.

> +    "kind": "enumerated",

We may be able to use the "categorical" type here, so the labels are self-documenting. An existing example is here:

https://searchfox.org/mozilla-central/rev/80ac71c1c54af788b32e851192dfd2de2ec18e18/browser/components/preferences/in-content/preferences.js#186-188

I'm not sure just by reading the documentation if we can add new labels afterwards if we find out we missed something useful to measure. Worst case, we can use a new histogram by adding a version number to the histogram name.

https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/histograms.html#categorical
Categorical histograms are actually the only type of histogram where you _can_ add new labels afterwards. Anything else and you'd have to give your probe a new name :)

( The relevant piece of the documentation for changing histograms is here: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/histograms.html#changing-a-histogram )
Thanks! I guess the "Note" in the categorical histograms documentation refers only to cases where you need to change "n_values".
Pushed by paolo.mozmail@amadzone.org:
https://hg.mozilla.org/integration/mozilla-inbound/rev/200b836d3aa5
Add telemetry to improve the understanding of current "about:config" usage patterns. r=paolo
https://hg.mozilla.org/mozilla-central/rev/200b836d3aa5
Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
The probe landed recently in Nightly 64 and should be in Beta 64 when the first version is published after the merge.

Chris, how long do we have to wait before we can verify that everything is working correctly in the pipeline? I tested "about:telemetry" locally and I see the probe on the Telemetry portal in the dictionary:

https://telemetry.mozilla.org/probe-dictionary/?search=about&searchtype=in_name&channel=nightly

However, for the moment I don't see any data.
Flags: needinfo?(chutten)
Ack, looks like we may have a data outage on the aggregates dataset. There's no data after Oct 11 at present. bug 1495453 for details.

Let me deploy the error message to the Measurement Dashboards so that it's clear.
Flags: needinfo?(chutten)
You need to log in before you can comment on or make changes to this bug.