Create telemetry probes to track which preference categories are opened

VERIFIED FIXED in Firefox 51

Status

()

Firefox
Preferences
VERIFIED FIXED
8 months ago
5 months ago

People

(Reporter: jaws, Assigned: jaws)

Tracking

(Blocks: 2 bugs)

Trunk
Firefox 53
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox51 verified, firefox52 verified, firefox53 verified)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

Comment hidden (obsolete)
This is being requested as part of a preferences redesign and reorganization.
Comment hidden (mozreview-request)
(Assignee)

Updated

8 months ago
Blocks: 1324168
Comment hidden (mozreview-request)

Comment 4

8 months ago
Feels like this would be more readable as a scalar. Do we have infrastructure to monitor that easily (like tmo's histogram tool) yet, Georg?
Flags: needinfo?(gfritzsche)

Comment 5

8 months ago
mozreview-review
Comment on attachment 8819508 [details]
Bug 1324167 - Create telemetry probes to track which preference categories are opened.  privacy-review=bsmedberg

https://reviewboard.mozilla.org/r/99246/#review99590

r=me, but I think scalars would be better here if there's a reasonable way of reading the values afterwards. See below for some other comments.

::: browser/components/preferences/in-content/preferences.js:127
(Diff revision 2)
> +  switch (category) {
> +    case "general":

An alternative way to implement this would be to have an array and use `indexOf`. That would be more concise. Up to you if you think that reduces readability/flexibility too much.

::: browser/components/preferences/in-content/preferences.js:158
(Diff revision 2)
> +        case "encryptionTab":
> +          return 11;
> +      }
> +      // fall-through for default case.
> +    default:
> +      throw new Error(`Unexpected category in telemetryBucketForCategory: ${category}`);

Shouldn't this just put it in a fallback bucket (for which bucket 0 might be the most appropriate, actually) ?
Attachment #8819508 - Flags: review?(gijskruitbosch+bugs) → review+
(In reply to :Gijs Kruitbosch from comment #4)
> Feels like this would be more readable as a scalar. Do we have
> infrastructure to monitor that easily (like tmo's histogram tool) yet, Georg?

Scalars are already available in the longitudinal dataset.
They will be added to t.m.o in the near future (bug 1297063).

Note that support for recording scalars from content processes is only being added (bug 1278556).
Flags: needinfo?(gfritzsche)

Comment 7

8 months ago
mozreview-review
Comment on attachment 8819508 [details]
Bug 1324167 - Create telemetry probes to track which preference categories are opened.  privacy-review=bsmedberg

https://reviewboard.mozilla.org/r/99246/#review99754

::: toolkit/components/telemetry/Histograms.json:5264
(Diff revision 2)
>    },
> +  "FX_PREFERENCES_CATEGORY_OPENED": {
> +    "bug_numbers": [1324167],
> +    "alert_emails": ["jaws@mozilla.com"],
> +    "expires_in_version": "56",
> +    "kind": "enumerated",

We have "kind":"categorical" now, which allows explicitly specifying labels.
See [choosing a histogram type](https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type), `TELEMETRY_TEST_CATEGORICAL` is an example.
Comment hidden (mozreview-request)
Hey Benjamin, can you do the telemetry/privacy review for this?

We would like to add an opt-out telemetry probe to track which preference categories users view. This will help us to know what categories are most popular, or at least to know what categories users try to visit when they have something in mind that they want to change.

These are set to expire in version 56.

We want an opt-out probe because we understand that users on Release channel use Firefox very differently than users on Nightly, especially when it comes to Preferences due to their "advanced" nature.

We may add some more probes in the future, which might track which preferences were changed and the time it takes from opening the preferences to the first preference changed, though that is not part of this bug or patch. We hope to use this data to help influence as well as measure the successfulness of reorganizing the preference categories as outlined in the specs on bug 1324168.
Flags: needinfo?(benjamin)
Comment hidden (mozreview-request)

Comment 11

8 months ago
data-review=me
Flags: needinfo?(benjamin)
Comment hidden (mozreview-request)

Comment 13

8 months ago
Pushed by jwein@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/551c356821b4
Create telemetry probes to track which preference categories are opened. r=Gijs privacy-review=bsmedberg

Comment 14

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/551c356821b4
Status: ASSIGNED → RESOLVED
Last Resolved: 8 months ago
status-firefox53: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 53
Created attachment 8821191 [details] [diff] [review]
Patch for Aurora52 and Beta51

Approval Request Comment
[Feature/Bug causing the regression]: telemetry probe to gather data about preference usage prior to reorg/redesign being tracked by bug 1324168
[User impact if declined]: none
[Is this code covered by automated tests?]: no, manual testing of telemetry probe was done locally
[Has the fix been verified in Nightly?]: not yet
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: adds a telemetry value when entering the preferences and another when changing categories
[String changes made/needed]: none
Attachment #8821191 - Flags: approval-mozilla-beta?
Attachment #8821191 - Flags: approval-mozilla-aurora?
Comment on attachment 8821191 [details] [diff] [review]
Patch for Aurora52 and Beta51

OK for aurora uplift. 
jaws, is there anything you can suggest for testing this over the next few days before we build the next beta?
Flags: needinfo?(jaws)
Attachment #8821191 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Andrei or Florin, it might be good to just check that enabling and disabling preferences works OK in aurora once this lands. I'm trying to think back on past issues with changing and adding new telemetry probes - usually performance problems or crashes.
This can still make it into the beta 11 build on Jan. 2.
Flags: needinfo?(florin.mezei)
Flags: needinfo?(andrei.vaida)

Comment 18

8 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-aurora/rev/4ed8e6215ade
status-firefox52: --- → fixed
As an update, we are currently tracking an issue with this probe on telemetry.mozilla.org, bug 1326414. This doesn't appear to affect people using the preferences, and may be a sign of a larger issue with new telemetry probes.
(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #19)
> As an update, we are currently tracking an issue with this probe on
> telemetry.mozilla.org, bug 1326414. This doesn't appear to affect people
> using the preferences, and may be a sign of a larger issue with new
> telemetry probes.

The issue is not related to this probe but to the type of probe that this is using. This does not affect data collection or functionality of Firefox. It only affects viewing the data that was collected and will not require any changes to the probe.
Flags: needinfo?(jaws)
Comment on attachment 8821191 [details] [diff] [review]
Patch for Aurora52 and Beta51

Thanks jaws. OK, this can make it into the beta 12 build on Thursday.
Attachment #8821191 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
https://hg.mozilla.org/releases/mozilla-beta/rev/f746607cc5966be9e56b33ff81eb280502423d69
status-firefox51: --- → fixed
I reproduced this issue using 53.0a1, build ID: 20161216030207, on Windows 10 x64.
I can confirm this issue is fixed, I verified using Fx 51.0b12, build ID: 20170105155013 and Fx 52.0a2, build ID: 20170106004019 and Fx53.0a1, build ID: 20170106030204, on Windows 10 x64, Mac OS X 10.11 and Ubuntu 14.04 LTS.

Cheers!
Status: RESOLVED → VERIFIED
status-firefox51: fixed → verified
status-firefox52: fixed → verified
status-firefox53: fixed → verified
Flags: needinfo?(florin.mezei)
Flags: needinfo?(andrei.vaida)
(Assignee)

Updated

7 months ago
Blocks: 1328561
No longer blocks: 1324168
Update from Frank Bertsch is that the category view on telemetry.mozilla.org is almost ready for viewing, but all data prior to Jan 17, 2017 was lost due to an issue with some processing code. I don't think that this data loss will cause a major change in our analysis, though we will want to wait a couple weeks longer before using the data to let more data collect. Frank says that we shouldn't expect any future data loss, this was a one-time issue and it was related to the new categorical type of probes.
Just want to point out that the data is *technically* still there, but histograms are all 0 (some other metadata isn't, so it truly messes up count information). I would recommend that you manually choose dates after 2017-01-17 in the distribution dashboard (under advanced options), and ignore all evolution data before that date as well (there's no way to limit the time range in the evolution view).

Updated

5 months ago
Blocks: 1330552
You need to log in before you can comment on or make changes to this bug.